Docker Swarm with encrypted node-to-node traffic

VSHNeer Elia set up a Docker Swarm Cluster with full traffic encryption inside the cluster (crosspost to his private blog):
I have set up a Docker Swarm cluster on the new Hetzner Cloud. First things first – the Hetzner Cloud is really amazing: Super simple, super cheap and performs as expected. It is not a bloated cloud provider that has 100x services and features that you can use for your servers, this keeps the costs and complexity down – I am really a big fan of it.
To the topic: Because the feature-set is simple, the Hetzner Cloud does not provide private networking (yet!). With only public IP addresses, we need to secure the overlay traffic between our docker containers!
 

The Problem

Per default, Docker Swarm encrypts the traffic between the managers, so we won’t have any issues there. However, this default setting is not set for container-to-container traffic. Any traffic that uses the overlay network is not encrypted by default because most of the time people do have private network setups with a floating IP as access point to the cluster. Docker assumes that the private network is secure and thus can spare some resources for other tasks (Which for example is not the case at DigitalOcean, so I recommend using overlay encryption anyway!).
Now, let’s assume we have the following stack:

version: '3'
services:
  db:
    networks:
      - internal
    image: mysql:5.7
    environment:
      MYSQL_DATABASE: wordpress
      MYSQL_USER: wordpress
      MYSQL_PASSWORD: securepw
  wordpress:
    networks:
      - traefik_public
      - internal
    depends_on:
      - db
    image: wordpress:latest
    environment:
      WORDPRESS_DB_HOST: db:3306
      WORDPRESS_DB_USER: wordpress
      WORDPRESS_DB_PASSWORD: securepw
    deploy:
      labels:
        - traefik.frontend.rule=Host:blog.example.com
        - traefik.docker.network=traefik_public
        - traefik.port=80
networks:
  traefik_public:
    external: true
  internal: 

This is a WordPress Stack that creates the WP site and a MySQL database. Those two networks are defined:

• internal
• traefik_public

The interal  overlay network is used for the communication between the WP container and the database. This network is not reachable by the outside. traefik_public  is the network used for the reverse proxy. It is only attached to the WP container as this is the only public facing side of this setup.
The problem here is: Without a secured private network, traffic running through the network internal will go out to another worker (Docker Node), and this fully plain visible. Any password/authentication/<SENSITIVE_DATA> is sent plain text between the docker containers, should they be on two different nodes.
Most of the docker images are not made for public access in their simple utilization and that’s why most keep it as simple as possible, no complicated encryption. You can of course build your own image to enable application side encryption.

The Solution

Docker has a solution for this issue. You can simply enable encryption of the overlay network. Sadly I really didn’t see much discussion about this hence why I thought a blog post about this particular issue might be useful.
The encryption of the network needs to happen during it’s creation, you cannot encrypt a network once it has been setup already. To enable encryption we need to add a flag to the network definition:

networks:
   traefik_public:
     external: true
   internal:
     driver_opts:
       encrypted: ""

The network traefik_public  is of course also encrypted as you don’t want the reverse proxying in plain text.
The option encrypted  will create a IPSec tunnel between all the workers where tasks are scheduled for a stack. This will fully encrypt all the traffic of the overlay network internal and thus allow sensitive data being shared between the database and WordPress.
You can read the official documentation about this here.
 

Final thoughts

The information in regards of encryption is very “hidden” and mostly ignored in my opinion. People want to simply deploy applications with docker without thinking about the infrastructure under it and thus running into the problem like plain text traffic on overlay networks.
I hope to make people more aware of encryption with this blog post.
If you have any questions, please let me know below!