16 ms·
Show HN: Docker rollout – Zero Downtime Deployment for Docker-compose
docker-compose is great for single node docker deployments, but it doesn't have a feature that would allow zero downtime deployments. It's not possible to deploy often if your app goes down every time, and using Kubernetes/Nomad/Swarm on a single node is an overkill.
I created this Docker plugin to be a drop-in replacement for the restart command in usual docker-compose deployment scripts. It performs a simple rolling deployment of a single service.
- Cardinal7167 4y agoAwesome, I was looking for something just like this the other day. I’ll be checking this out!
- moglito 4y ago> Using container orchestration tools like Kubernetes or Nomad is usually an overkill for projects that will do fine with a single-server Docker Compose setup. Couldn't agree more! Very nice!
- ciconia 4y agoFWIW I have discovered a simple technique that solves this problem without any additional software: I create two or more services that are identical in all but name, so instead of a `backend` service I'll have `backend1`, `backend2` etc. When I need to restart the service, I do it in a staggered fashion, first stop and restart `backend1`, wait a few seconds, then stop and restart `backend2` etc. I put this in a script, works without any problem.
- m00dy 4y agoThis one has downtime right ?
- pankarol 4y agoIf you restart them one at a time, you always have at least one instance running.
- keeperofdakeys 4y agoIt would depend on the program. If you've got a load balancer in front, or they're receiving requests through a database or message queue, then there won't be an outage. Arguably anything where uptime is important should be setup this way. Upgrades can fail, and that will cause downtime anyway.
- whazor 4y agoCall them 'blue' and 'green', and you have blue-green deployments.
- hu3 4y agoSimple and ingenious. I like it! Do they use separate ports? Is there a load balancer to point to both services?
- comprev 4y agoThat's known as a blue-green deployment strategy and has been around since deployments began
- bobbiechen 4y agoUnlike the sibling comments, I would call this a rolling deployment: you are removing an old host and adding a new one at the same time when you restart. In a blue green deployment I would expect to see you add backend3 and backend4 with the updated code, and then rename services (or redirect traffic in some other way) to them. The difference is, if something goes wrong, backend1 and backend2 are hot standbys and we can revert the change by redirecting traffic back to them.
- intelVISA 4y agoIsn't that functionality already buried somewhere between Docker Stack and Docker Swarm?
- skor 4y agoyeah, and docker swarm on a single node is not overkill at all. plus, don’t need to install anything. Sure you have just one manager but with one node you don’t need to worry about quorum, if it goes down then that’s that.
- arynda 4y agoany chance you have a link to a good readme on setting up single-node docker swarm?
- skibbityboop 4y ago1. Install regular old Docker 2. Type 'docker swarm init' 3. There is no 3, you're literally finished and now have a full-on Swarm node w/ all features.
- dijksterhuis 4y agothe docker docs are actually pretty good. general intro https://docs.docker.com/engine/swarm/ https://docs.docker.com/engine/swarm/ step by step tutorial https://docs.docker.com/engine/swarm/swarm-tutorial/ https://docs.docker.com/engine/swarm/swarm-tutorial/ ^ takes you through a 3 server setup and deploying services via the CLI rather than a docker "stack" file, which is basically a compose file with the ability to set additional deployment specific properties. just set up the one manager and skip anything about additional worker nodes or draining nodes. for docker stack files -- see the docker compose v3 reference but take heed of any `docker stack deploy` caveats in the compose documentation. anything about docker stack deploy` is "stack file" territory. https://docs.docker.com/compose/compose-file/compose-file-v3/ https://docs.docker.com/compose/compose-file/compose-file-v3... compose file v3 is being depreciated this year, with a new specification aiming to unify the two. but it gives you an idea of the historical differences between old `docker-compose up` and `docker stack deploy` (swarm).
- ThePhysicist 4y agoDocker compose is great, I use it a lot for local development and testing of distributed systems. With a few tweaks you can simulate almost anything in containers including systemd and low-level networking stuff, which e.g. makes simulating an entire Ansible based setup trivial. Too bad Docker doesn't seem to push this much, with a bit of extra work this probably could be the deployment platform for 95 % of all software systems.
- fellowniusmonk 4y agoI don't understand. Is this common usage or some unique setup you've discovered?
- bluehatbrit 4y agoI think the parent comment is saying that it's a shame Docker aren't pushing docker-compose more in general, rather than a specific use case. Assuming they are, I have to agree. Docker-compose is excellent for so many use cases where swarm, nomad, or k8s would be too much but straight docker isn't enough. Seeing more investment in compose would be fantastic.
- francislavoie 4y agoWell, the good news is compose v2 is a huge improvement and they've been releasing at a much higher cadence than v1.
- andix 4y agoDocker compose is also fine for production environments, if you are okay with a few minutes downtime during updates and your application is just running on one or two machines. Even 90% of the enterprise systems I worked with so far were okay with those restrictions. Sooo much simpler than k8s.
- nickcox 4y agoFWIW ECS supports rolling updated for docker compose deployments: https://docs.docker.com/cloud/ecs-integration/#rolling-update https://docs.docker.com/cloud/ecs-integration/#rolling-updat...
- francislavoie 4y agoAwesome! I might use this. I think this approach should work fine with https://github.com/lucaslorentz/caddy-docker-proxy https://github.com/lucaslorentz/caddy-docker-proxy as well.
- m00dy 4y ago>> as it's not possible to run multiple containers with the same name or port mapping. I think you can :)
- bobleeswagger 4y agoelaborate
- nijave 4y agoI'm not sure what parent means but there's SO_REUSEPORT. I don't know if you can set that with Docker. There's also other tricks like not forwarding ports with -p and manually adding a firewall rule to rewrite or forward to the correct socket (you can still access containers using their container IP) Not sure what is meant by service names
- esotericimpl 4y ago[dead]
- jcuenod 4y agohttps://dockerswarm.rocks/ https://dockerswarm.rocks/
- luckylion 4y ago> Whenever you read here "Docker Swarm" we are actually talking about "Docker Swarm mode". > Not the deprecated product called "Docker Swarm". "Hey, Bob, what can we do to confuse people?" - "I don't know... hey, did you hear about that Perl 6 thing? That worked pretty well!" I like it though, looks much more hands-on than the documentation which often feels just short of being useful.
- jve 4y agoIs docker swarm alive and well? I always get the feeling it is something abandoned. But I love the simplicity.
- raphinou 4y agoDocker Swarm is getting some more attention after having been neglected, with repo activity going up[1], and Mirantis even publishing a sponsored article on The Next Web[2]. Things look brighter for Swarm than one year ago. Interest in Docker Swarm also persist I think. My post on starting Docker Swarm in 2022 [3] on my very low profile blog gets very regular visits. [1]: https://github.com/moby/swarmkit/commits/master https://github.com/moby/swarmkit/commits/master [2]: https://thenewstack.io/docker-swarm-a-user-friendly-alternative-to-kubernetes/ https://thenewstack.io/docker-swarm-a-user-friendly-alternat... [3]: https://www.yvesdennels.com/posts/docker-swarm-in-2022/ https://www.yvesdennels.com/posts/docker-swarm-in-2022/
- josegonzalez 4y agoDokku Maintainer here. This is pretty neat. One of my gripes about docker-compose - and a major why I've been hesitant to add support for it - is that the updates are not zero-downtime. That makes it much more annoying to use for app deploys as you either have to rewrite the compose command as a docker command (defeating part of the purpose of a compose file) or accept the downtime during a deploy. I'll definitely be including this tool (or something like it) with Dokku once I actually add compose support. Combining this with either Caddy Docker Proxy[1] or Traefik[2] could be quite nice for a very simple app deployment system. Would be super awesome for this functionality to land in the official `compose` plugin, but for now this is a great way to dip your toes into app deployments without too much overhead. There is a small island of productivity tools around docker-compose that would be super nice to build, and it's nice to see something like this land :) - https://github.com/lucaslorentz/caddy-docker-proxy - https://doc.traefik.io/traefik/providers/docker/
- jskrablin 4y agoWhat exactly is the problem with short downtime during updates/deployments? A lot of people don't care if their deployment requires a few seconds of downtime. And most users of $randomwebsite don't really care if they need to hit reload button every now and then.
- bionsystem 4y agoI guess if you have great ambitions about "web scale" and client retention, a few seconds downtime on your website might mean a loss on a significant amount of potentially new customer. As for me, I actually like downtimes. We have data migrations to do anyway on most major updates (they are automatic when the new container boots but the app is unavailable), which can sometimes take hours, and that leaves us notifying our customers for updates and gives us time to do some sanity checks before handing them over control. Compose is a great tool for single (and probably few) VM(s) deployments. I used to love swarm for slightly bigger ones as well since it's a lot easier to manage than kube and can easily be tailored to ones need, still sad it's being deprecated.
- goodpoint 4y agoUrgh
- polyrand 4y agoI did something similar using git hooks (for Heroku-like deployments) and curl for health-checking the app [0]. In my case, instead of using replicas, I created multiple services in the docker-compose.yaml file. The services are the same but with different names (e.g: app1 and app2). Then, during deployment I can update only app1, run the health checks, then update app2 if everything is ok. [0]: https://ricardoanderegg.com/posts/git-push-deployments-docker-tags/#creating-our-hook https://ricardoanderegg.com/posts/git-push-deployments-docke...
- linuxftw 4y agoIs a single node k3s overkill? I don't think so. Once you outgrow it, you can port your deployments to a real k8s cluster.
- bongobingo1 4y agoFrom what I have ever seen, even k3s has a reasonably heavy memory requirement for cheap vps instances, vs docker-compose. I think they target different markets. e: I went to re-check, and running the default get.k3s.com install instructions had k3s using 30-35% of 1GB of memory, without anything running. Thats pretty meaningful on low resource machines like a RPI, NAS, $5/vps etc. Also FWIW, Hashicorp Nomad uses ~3.5% on the same machine (but provides less than k3s, more than compose).
- linuxftw 4y agoI mean, if you're in the realm of 'no-downtime upgrades' because it will impact your application, maybe spending an extra $5/mo makes sense rather than relying on bespoke tooling. There's also k0s, which aims to be even lighter than k3s.
- xyst 4y agoso a blue-green deployment for docker-compose, neat
- eckza 4y agoWoah, alright! This is what I've been waiting for. Thanks for building this... I can't wait to test it out!
- crummy 4y agoThis is great to see - fills the biggest hole in Docker Compose. I built something similar, it was a bit fragile though. What I ended up using instead was a lot simpler with only a little downside: Caddy for a reverse proxy will buffer requests when the downstream is down, unlike Nginx (which we used before). So during a deploy nobody gets any errors, just a brief 2s delay while the new service boots up. Seamless enough for me.
- correct-opinion 4y agoAFAIK nginx-proxy does not enable true zero-downtime. Thus using this tool with nginx-proxy does not enable zero-downtime. Example. Deployment goes like this: 1) 2 app versions are deployed - blue (vCurrent) and green (vNext); both are up and ready to handle connections 2) We are about to shut down blue and replace it with green 3) Blue is handling long-running http request; receives shutdown signal; keeps handling long-running request, but does not accept new connections from now on 4) because it is still active (nginx-proxy wont remove it), blue will still receive connections from nginx-proxy, and all of them will fail because of reasons stated in (3) Connections routed to green will succeed; This is where zero-downtime fails. 5) Once long running request is handled, and blue removed from docker, only then nginx-proxy refreshes its configuration and routes all traffic to green. If i am correct, this tool solves issue where non-working container are configured to receive traffic by nginx-proxy, but does not solve issue where partially shutdown old container still receices traffic and drops it. I dont know about caddy-docker-proxy.
- shafyy 4y agoInterestingly, it looks like DHH is working on something similar (for Rails): https://github.com/mrsked/mrsk https://github.com/mrsked/mrsk > MRSK deploys web apps in containers to servers running Docker with zero downtime. It uses the dynamic reverse-proxy Traefik to hold requests while the new application container is started and the old one is stopped. It works seamlessly across multiple hosts, using SSHKit to execute commands.
- rsolva 4y agoI just did a quick search for how to achieve a basic no-downtime deploy with Podman. It turns out it is not that hard [0]. Not sure how to do this if pods or containers are managed as systemd services, though. [0] https://github.com/evolutics/zero-downtime-deployments-with-podman https://github.com/evolutics/zero-downtime-deployments-with-...
- miked85 4y agoIf you are only running on a single node, it doesn't sound like downtime is a big concern regardless?
- rahoulb 4y agoThere's a difference between a hardware or other catastrophic failure giving you serious downtime and wanting to deploy a new version several times per week.
- miked85 4y agoDefinitely true, but having zero redundancy doesn't seem like uptime is a high concern.
- nijave 4y agoThere's other automation like running the singular machine in a scaling group with health checks. I think on all the major cloud providers scaling sets are free so it offers a way to run automatically healing singular machines.
- simplotek 4y agoWhat's wrong with Docker swarm mode? It already supports stacks and blue/green deployments. Doesn't this represent reinventing the wheel?
- fullofdev 4y agothanks , have been looking a simple solution for this
- robius 4y agoDocker has some serious flaws with respect to app health and doing something about it when it's unhealthy. Just look at this PR that is 3years old and counting: https://github.com/moby/moby/issues/28400 https://github.com/moby/moby/issues/28400
- robius 4y agoAs you can see Swarm is not a solution to having a proper action to a health check event on a container. If a fly lands on you, we don't need to shoot you and awaken the new clone of you.