4 ms·

Yes exactly, I have an API and a couple of static sites exposed via nginx. None of the other services have exposed ports and all the networking is done through

Yes exactly, I have an API and a couple of static sites exposed via nginx. None of the other services have exposed ports and all the networking is done through docker.

justinclift 6y ago
Just to double check (as a safety measure), are you using an external firewall for the docker container to ensure nothing can access them, or are you using customised firewall rules on the docker hosts? Asking because from what I've seen (so far) it's only safe to use an external firewall - eg Digital Ocean (or whoever) firewall applied to hosts - rather than iptables on the hosts themselves. Saying that because when starting the Docker service on a machine, the ~first thing it does is screw with any existing firewall config to ensure it can pipe data around between things. As a side effect, it seems to open up ports to the whole world. So any carefully secured config beforehand becomes useless. :(
- kavalg 6y ago
  Had the same issue with swarm and iptables/ufw. The solution I found was to replace nginx with traefik and never expose service ports out of the swarm. All HTTP/TCP/UDP termination is done by Traefik (itself running in Swarm too). So far the experience has been good. However, you need to bind the traefik container to the host network. Otherwise you will get not be able to get the remote host IP properly (x-forwarded-for). I am mostly doing scale up (Hetzner can go up to 256GB RAM and dedicated CPUs). This saves me the hassle of managing distributed storage (I just use local docker volumes and schedule a snapshot of the whole drive as a local backup at Hetzner). Should you need to scale traefik horizontally (e.g. > ~30K req/s), then you would need something like e.g. Consul to manage the distributed configuration. Dealing with docker volumes on a distributed storage may be harder though. Would be glad if someone can share their experience doing that.
  - justinclift 6y ago
    Interesting. Sounds like a reasonable path go with when initially growing. eg go bigger (hosts) first, then figure out horizontal scaling later if/when needed :)

Yes exactly, I have an API and a couple of static sites exposed via nginx. None of the other services have exposed ports and all the networking is done through docker.

justinclift 6y ago

Just to double check (as a safety measure), are you using an external firewall for the docker container to ensure nothing can access them, or are you using customised firewall rules on the docker hosts? Asking because from what I've seen (so far) it's only safe to use an external firewall - eg Digital Ocean (or whoever) firewall applied to hosts - rather than iptables on the hosts themselves. Saying that because when starting the Docker service on a machine, the ~first thing it does is screw with any existing firewall config to ensure it can pipe data around between things. As a side effect, it seems to open up ports to the whole world. So any carefully secured config beforehand becomes useless. :(

kavalg 6y ago
Had the same issue with swarm and iptables/ufw. The solution I found was to replace nginx with traefik and never expose service ports out of the swarm. All HTTP/TCP/UDP termination is done by Traefik (itself running in Swarm too). So far the experience has been good. However, you need to bind the traefik container to the host network. Otherwise you will get not be able to get the remote host IP properly (x-forwarded-for). I am mostly doing scale up (Hetzner can go up to 256GB RAM and dedicated CPUs). This saves me the hassle of managing distributed storage (I just use local docker volumes and schedule a snapshot of the whole drive as a local backup at Hetzner). Should you need to scale traefik horizontally (e.g. > ~30K req/s), then you would need something like e.g. Consul to manage the distributed configuration. Dealing with docker volumes on a distributed storage may be harder though. Would be glad if someone can share their experience doing that.
- justinclift 6y ago
  Interesting. Sounds like a reasonable path go with when initially growing. eg go bigger (hosts) first, then figure out horizontal scaling later if/when needed :)