Natural Born Coder

Making a Docker Container Use a VPN

—

by

doozer
in , ,

A while back I wrote a post on routing all traffic through a VPN under Linux. The solution discussed in that post is fine if you are only dealing with regular applications but when you are dealing with containers the world is a difference place. Docker networks are, or at least can be, complicated. By default when a single container is started (e.g. with Docker run) it goes into the default bridge network. When you start a number of services with Docker Compose it will, by default, create a new bridge network for you with a name based on the name of your project.

If you are running a firewall similar to the one I discussed earlier you might expect that the containers started with Docker would also be forced to use that VPN but you’d be wrong. Docker creates it’s own chains in iptables that bypass the rules set up by UFW and worse UFW can’t even show you that has happened. Docker effectively bypasses the firewall you’ve created.

As an experiment, if you set up a completely closed off firewall with UFW and then start up a container with a port mapped to the host (let’s say port 80 for nginx) you’ll be able to access it from your network even though the port isn’t open on the firewall according to UFW. If you change the docker run command to –net=host then you’ll find the firewall works, this is because you’ve placed the container on the host network and forced it to use the rules UFW created. Likewise, if that container connects to the outside world using the default bridge it won’t got out over the VPN but it will if it’s set to use the host network. Why is this a problem? Let’s say you’re using Deluge to download Linux ISO’s. You don’t want anyone to know you like Linux so you’ll want that to run over a VPN.

Solution

One solution I explored was to just place all the containers on the host network and continue using the UFW firewall I had already configured. This would be a quick solution as it would require only a minimal change to the compose file but it’s also not a great solution. Using the host network is really intended for applications that must have this type of access, for example when running a DHCP server that requires access to MAC addresses. A much better solution is to run a separate container that opens and maintains a VPN link and then make the other containers use the networking stack of that VPN container. This type of configuration is not well documented in Docker, the best I could find was this.

There are a number of images available that will create a VPN container that you can use. At the most basic end is an OpenVPN container but if you are using Nord or PIA there are specific containers for those VPN providers (and others). All these containers work in the same way and are configured similarly and the configuration of the Nord service is shown below. This VPN container requires additional capabilities (cap_add) because it needs to manipulate the network in ways containers aren’t usually allowed to (see here). At the bottom of the configuration are the ports that will be exposed to the local network, note that this setting is different to the PORT environment variable which is for ports you want to expose over Nord.

nordvpn:
    image: bubuntux/nordvpn
    networks: 
      pirate_net:
      # ipv4_address: 172.25.0.2 # Optionally give this a fixed IP address    
    cap_add:
      - NET_ADMIN               
      - SYS_MODULE              
    sysctls:
      - net.ipv4.conf.all.rp_filter=2
    devices:
      - /dev/net/tun            
    environment:                
      - [email protected]     
      - PASS=secret_password         
      - CONNECT=France
      - TECHNOLOGY=NordLynx
      - NETWORK=192.168.1.0/24 
    ulimits:                    
      memlock:
        soft: -1
        hard: -1
    ports:
      - 7878:7878 # Radarr
      - 8112:8112 # Deluge
      - 9117:9117 # Jackett

This configuration differs slightly from the reference configuration given by the image maintainer. The main difference is that I have placed the VPN container in it’s own bridge network. This isn’t strictly necessary but I like to have a named network rather than just relying on the default. It also makes it easy to assign a fixed IP address if you need that for some reason. Importantly, the other containers that use this VPN container for internet access can use “localhost” to reach each other. This is because from a network perspective they all exist within the VPN container. What the other containers can’t do is reference each other by name as the Docker internal DNS system doesn’t work with this set up. In other words Radarr reaches Deluge using the localhost:8112 address.

The custom network configuration looks like this:

networks:
  pirate_net:
    driver: bridge
    # ipam: 
    #   driver: default
    #   config: 
    #     - subnet: 172.25.0.0/24

For other containers use a configuration like this:

jackett:
    image: ghcr.io/linuxserver/jackett
    container_name: jackett
    restart: unless-stopped
    network_mode: service:nordvpn
    environment:
      - PUID=${PUID}
      - PGID=${PGID}
      - TZ=Europe/London
      - AUTO_UPDATE=true
    volumes:
      - ${JACKETT_CONFIG_DIR}:/config
      - ${BLACKHOLE_DIR}:/downloads
    depends_on: 
      - nordvpn

Notice that this container depends on the VPN container, this is important as the VPN container must come up first. All containers in this compose should depend on the VPN container. Also notice that there are no ports defined here, this is because you can’t access this container directly. Access needs to be through the VPN container hence why they ports are mentioned there. Most importantly though notice that the network mode is set to “service:nordvpn”, this tells the container to use the VPN container for it’s network.

Notes

If the VPN container goes down (simulate with docker-compose stop nordvpn) then any container depending on it becomes unreachable and will need to be restarted if you want to connect to the internet again. The Nord image I’m using here seems to be stable though and will transparently restart the VPN if that goes down.

Health Checks

After running the VPN container for about 24 hours I noticed that my remote IP address had changed. I watched the service for a little while and I noticed the address changed a couple more times. I grep’ed the log files for the container and sure enough there were a few instances of the IP address changing. I’ve removed the actual IP addresses but the number at the end indicates a unique IP (note, I get double entries for most lines, the duplicates have been removed).

$ docker logs container_name | grep 'rule add to'
2021/02/18 23:18:16 [nordlynx] ip -4 rule add to xxx.xxx.xxx.xx1 lookup main priority 32765
2021/02/18 23:28:14 [nordlynx] ip -4 rule add to xxx.xxx.xxx.xx2 lookup main priority 32765
2021/02/19 09:44:13 [nordlynx] ip -4 rule add to xxx.xxx.xxx.xx3 lookup main priority 32765
2021/02/19 10:04:15 [nordlynx] ip -4 rule add to xxx.xxx.xxx.xx4 lookup main priority 32765
2021/02/19 10:09:17 [nordlynx] ip -4 rule add to xxx.xxx.xxx.xx5 lookup main priority 32765

I started the service at 11:18 and it swapped IP at 11:28 but then was stable for nine hours before swapping IP in quick succession. After the 10:09 it seemed to become stable again. From what I can tell this is being caused by the healthcheck in the Dockerfile for the container. It attempts to check if the IP address you are using is protected and if it fails to return true it disconnects and reconnects your VPN. I’ll try turning on debugging later but for now it seems to be working even if it’s not as stable as it could be.

Additional Reading

I read a lot of pages to come up with my understanding of this problem, this is a list of just the ones that really helped me.