Docker Swarm with encrypted node-to-node traffic
VSHNeer Elia set up a Docker Swarm Cluster with full traffic encryption inside the cluster (crosspost to his private blog):
I have set up a Docker Swarm cluster on the new Hetzner Cloud. First things first – the Hetzner Cloud is really amazing: Super simple, super cheap and performs as expected. It is not a bloated cloud provider that has 100x services and features that you can use for your servers, this keeps the costs and complexity down – I am really a big fan of it.
To the topic: Because the feature-set is simple, the Hetzner Cloud does not provide private networking (yet!). With only public IP addresses, we need to secure the overlay traffic between our docker containers!
The Problem
Per default, Docker Swarm encrypts the traffic between the managers, so we won’t have any issues there. However, this default setting is not set for container-to-container traffic. Any traffic that uses the overlay network is not encrypted by default because most of the time people do have private network setups with a floating IP as access point to the cluster. Docker assumes that the private network is secure and thus can spare some resources for other tasks (Which for example is not the case at DigitalOcean, so I recommend using overlay encryption anyway!).
Now, let’s assume we have the following stack:
version: '3'
services:
db:
networks:
- internal
image: mysql:5.7
environment:
MYSQL_DATABASE: wordpress
MYSQL_USER: wordpress
MYSQL_PASSWORD: securepw
wordpress:
networks:
- traefik_public
- internal
depends_on:
- db
image: wordpress:latest
environment:
WORDPRESS_DB_HOST: db:3306
WORDPRESS_DB_USER: wordpress
WORDPRESS_DB_PASSWORD: securepw
deploy:
labels:
- traefik.frontend.rule=Host:blog.example.com
- traefik.docker.network=traefik_public
- traefik.port=80
networks:
traefik_public:
external: true
internal:
This is a WordPress Stack that creates the WP site and a MySQL database. Those two networks are defined:
- internal
- traefik_public
The interal overlay network is used for the communication between the WP container and the database. This network is not reachable by the outside. traefik_public is the network used for the reverse proxy. It is only attached to the WP container as this is the only public facing side of this setup.
The problem here is: Without a secured private network, traffic running through the network internal will go out to another worker (Docker Node), and this fully plain visible. Any password/authentication/<SENSITIVE_DATA> is sent plain text between the docker containers, should they be on two different nodes.
Most of the docker images are not made for public access in their simple utilization and that’s why most keep it as simple as possible, no complicated encryption. You can of course build your own image to enable application side encryption.
The Solution
Docker has a solution for this issue. You can simply enable encryption of the overlay network. Sadly I really didn’t see much discussion about this hence why I thought a blog post about this particular issue might be useful.
The encryption of the network needs to happen during it’s creation, you cannot encrypt a network once it has been setup already. To enable encryption we need to add a flag to the network definition:
networks:
traefik_public:
external: true
internal:
driver_opts:
encrypted: ""
The network traefik_public is of course also encrypted as you don’t want the reverse proxying in plain text.
The option encrypted will create a IPSec tunnel between all the workers where tasks are scheduled for a stack. This will fully encrypt all the traffic of the overlay network internal and thus allow sensitive data being shared between the database and WordPress.
You can read the official documentation about this here.
Final thoughts
The information in regards of encryption is very “hidden” and mostly ignored in my opinion. People want to simply deploy applications with docker without thinking about the infrastructure under it and thus running into the problem like plain text traffic on overlay networks.
I hope to make people more aware of encryption with this blog post.
If you have any questions, please let me know below!