5 ms·

I've been using docker swarm on Hetzner for my fledgling business and am quite happy with it. The main feature I use is rolling upgrades that make it very easy

I've been using docker swarm on Hetzner for my fledgling business and am quite happy with it. The main feature I use is rolling upgrades that make it very easy to deploy changes. It's simple, and has all the features I need for the stage I'm at. Once I have a more demanding infrastructure I'll probably switch over to a managed kubernetes, but those platforms have a significantly higher unit cost.

maxdo 6y ago
I've been using Kubernetes on hetzner that I installed with one command.
- majewsky 6y ago
  So how do you handle k8s version upgrades every three months? And please don't tell me "who needs upgrades" on a security-critical component.
justinclift 6y ago
With your setup, are you using a static front end web server(s?) - eg nginx - reverse proxying to the applications managed with Docker Swarm?
- alephu5 6y ago
  Yes exactly, I have an API and a couple of static sites exposed via nginx. None of the other services have exposed ports and all the networking is done through docker.
  - justinclift 6y ago
    Just to double check (as a safety measure), are you using an external firewall for the docker container to ensure nothing can access them, or are you using customised firewall rules on the docker hosts? Asking because from what I've seen (so far) it's only safe to use an external firewall - eg Digital Ocean (or whoever) firewall applied to hosts - rather than iptables on the hosts themselves. Saying that because when starting the Docker service on a machine, the ~first thing it does is screw with any existing firewall config to ensure it can pipe data around between things. As a side effect, it seems to open up ports to the whole world. So any carefully secured config beforehand becomes useless. :(
    - kavalg 6y ago
      Had the same issue with swarm and iptables/ufw. The solution I found was to replace nginx with traefik and never expose service ports out of the swarm. All HTTP/TCP/UDP termination is done by Traefik (itself running in Swarm too). So far the experience has been good. However, you need to bind the traefik container to the host network. Otherwise you will get not be able to get the remote host IP properly (x-forwarded-for). I am mostly doing scale up (Hetzner can go up to 256GB RAM and dedicated CPUs). This saves me the hassle of managing distributed storage (I just use local docker volumes and schedule a snapshot of the whole drive as a local backup at Hetzner). Should you need to scale traefik horizontally (e.g. > ~30K req/s), then you would need something like e.g. Consul to manage the distributed configuration. Dealing with docker volumes on a distributed storage may be harder though. Would be glad if someone can share their experience doing that.
      - justinclift 6y ago
        Interesting. Sounds like a reasonable path go with when initially growing. eg go bigger (hosts) first, then figure out horizontal scaling later if/when needed :)

maxdo 6y ago

I've been using Kubernetes on hetzner that I installed with one command.

majewsky 6y ago
So how do you handle k8s version upgrades every three months? And please don't tell me "who needs upgrades" on a security-critical component.

justinclift 6y ago

With your setup, are you using a static front end web server(s?) - eg nginx - reverse proxying to the applications managed with Docker Swarm?

alephu5 6y ago
Yes exactly, I have an API and a couple of static sites exposed via nginx. None of the other services have exposed ports and all the networking is done through docker.
- justinclift 6y ago
  Just to double check (as a safety measure), are you using an external firewall for the docker container to ensure nothing can access them, or are you using customised firewall rules on the docker hosts? Asking because from what I've seen (so far) it's only safe to use an external firewall - eg Digital Ocean (or whoever) firewall applied to hosts - rather than iptables on the hosts themselves. Saying that because when starting the Docker service on a machine, the ~first thing it does is screw with any existing firewall config to ensure it can pipe data around between things. As a side effect, it seems to open up ports to the whole world. So any carefully secured config beforehand becomes useless. :(
  - kavalg 6y ago
    Had the same issue with swarm and iptables/ufw. The solution I found was to replace nginx with traefik and never expose service ports out of the swarm. All HTTP/TCP/UDP termination is done by Traefik (itself running in Swarm too). So far the experience has been good. However, you need to bind the traefik container to the host network. Otherwise you will get not be able to get the remote host IP properly (x-forwarded-for). I am mostly doing scale up (Hetzner can go up to 256GB RAM and dedicated CPUs). This saves me the hassle of managing distributed storage (I just use local docker volumes and schedule a snapshot of the whole drive as a local backup at Hetzner). Should you need to scale traefik horizontally (e.g. > ~30K req/s), then you would need something like e.g. Consul to manage the distributed configuration. Dealing with docker volumes on a distributed storage may be harder though. Would be glad if someone can share their experience doing that.
    - justinclift 6y ago
      Interesting. Sounds like a reasonable path go with when initially growing. eg go bigger (hosts) first, then figure out horizontal scaling later if/when needed :)