15 ms·

Best practices around creating production ready web apps with Docker Compose

514 points by nickjj 5y ago

znpy 5y ago
Pro-tip: nowadays podman can pretend to be a kubernetes cluster and launch pod And deployment definitions (in yaml, of course). That of course is way close to the "production" in "production ready".
- ecliptik 5y ago
  That sounds really cool. Do you happen to have a good link for this?
  - buildbot 5y ago
    Not GP, but Perhaps https://www.redhat.com/sysadmin/compose-kubernetes-podman https://www.redhat.com/sysadmin/compose-kubernetes-podman?
- buildbot 5y ago
  Without podman-compose too! I didn't know this was the case now: https://www.redhat.com/sysadmin/compose-kubernetes-podman https://www.redhat.com/sysadmin/compose-kubernetes-podman
- 2OEH8eoCRo0 5y ago
  Also daemonless and rootless
  - znpy 5y ago
    This is one of the coolest features! Different users run different containers, and no user can see other user's containers.
  - deckard1 5y ago
    docker has rootless now. Tried to set it up the other day on bog standard Debian. That's a deep ass rabbit hole with a whole lot of gotchas and I'm not entirely convinced the whole thing won't topple over at the slightest breeze. It took me an embarrassingly long time to figure out how to install the correct version of slirp4netns, which is not in the stable repo. Reassuring.
    - emerongi 5y ago
      Similar experience. I used Docker rootless for a while 1-2 years ago. There were hiccups and upgrading was a pain, so I stopped using it. It's possible it has improved, though.
    - znpy 5y ago
      It works out of the box on rhel 8.3, and I've used succession Ubuntu 20.04 installed by the PPA repos.
- 0xbadcafebee 5y ago
  ....but then when you run it in production, your app fails, because it was designed against assumptions around the fake cluster and pod, or a different config file. I'm not saying there's a solution to this, but we should remind people that "we can pretend X is Y" is a reproducibility red flag.
  - znpy 5y ago
    There are step between development laptop and production you know, smarty pants? Like, a staging cluster? A pre-prod cluster?
    - 0xbadcafebee 5y ago
      Which then creates its own special snowflake environment which isn't either dev or prod, and duplicates resources, and can only test one change at a time (unless you have real ephemeral environments, and most people don't). It's better to have blue/green, or something like blue green, and use that as staging, than have an actual dedicated "staging cluster". But like I say, dev still is far from close to that. So you end up doing a lot of development in dev, and then fixing a ton of things in staging because you weren't developing against staging. And your tests aren't fantastic, so some bugs make it through both staging and prod, and it turns out they're because your design was making assumptions based on dev. Hence, make dev and prod as close as possible, and then you don't need to have "staging", because you have confidence that what worked in dev will also work in prod.
- messo 5y ago
  Podman is great! I was surprised to learn that the latest version is compatible with docker-compose (minus the Docker Swarm part). I have been playing around with it on my workstation (Arch/Podman/docker-compose) and it works really well, rootless and without any daemon.
barbazoo 5y ago
> web: > <<: default-app > ... > worker: > <<: default-app This is always somewhat of a code smell to me. This makes me want to dig deeper and see if the two use cases shouldn't have been separated into separate Docker files or even separate apps.
- TameAntelope 5y ago
  I tend to start with monolith and split things out when I get frustrated with how coupled they are.
- pentium166 5y ago
  It seems reasonable to me (for a compose-in-prod setup). The standard setup (at least for Rails, in my experience) is that the web server and background job processor will have the same dependencies and nearly identical configuration. Until the app has gone off the rails and I really do need to have the server and worker diverge, this approach seems like less to worry about than maintaining multiple images and duplicated but 98% identical container definitions.
  - Ensorceled 5y ago
    Yeah, same with Django ... really try to keep these environments all in sync so you don't have different environments for no real gain. What do I gain when my worker doesn't have django-allauth installed other than possible development environment complexity.
wiktorcie 5y ago
just use dokku instead, it's much easier. Why bother with docker compose?
- clintonb 5y ago
  Dokku has a lot of overhead to run.
mdoms 5y ago
I don't understand his "Avoiding 2 Compose Files for Dev and Prod with an Override File" point. He just says it, but never justifies it.
- randompwd 5y ago
  It's a terrible idea that's why. It comes with far more(&more dangerous) downsides than upsides. It's a rookie mistake for simple applications which most people learn to not do.
- reportt 5y ago
  The idea is that you have a core structure that is used in both local/dev environment and the prod environment, with just the override applying changes. For example, your local override might have the "build" configuration, whereas the dev environment or production environment docker-compose should not have that configuration (assuming you are pulling from a private registry). That single core docker-compose.yml can be copied to any environment it needs and it should just run the app, but if you really, really need to override something for a specific reason, it's not hard to do it without modifying the core docker-compose.yml.
- trinovantes 5y ago
  Probably so you don't have duplicate settings in docker-compose.prod.yml and docker-compose.dev.yml when the only difference is probably just 1 line specifying NODE_ENV etc.
- nickjj 5y ago
  Hi, It's mentioned on video and briefly in the blog post. It's so you don't have to duplicate a bunch of services between 2 files while you slightly change a bit of configuring for those services depending on the environment you're in, or even running different containers all together (which is the problem the override file solves).
stevebmark 5y ago
I haven't heard of using docker compose for production development, only local development. In production I normally see people building production images and spinning up them up in their cloud with Terraform and whatever other infrastructure they need, especially for databases where your local MySQL container isn't going to be configured or provisioned like an AWS RDS instance. What do you hand a docker-compose file off to in production? Swarm? When would you choose this path over a separate system to provision your production resources (like Terraform)?
- namdnay 5y ago
  Same here, docker compose is a great little tool for local development. You can get your apps up and running with their respective elasticsearches and localstacks etc, then attach to process if needed to debug For anything more serious, Terraform makes much more sense IMHO
- ensignavenger 5y ago
  Docker Swarm is one option, but Docker Desktop has the ability to deploy from a Compose file directly to AWS. There are additional tools for other platforms. See https://aws.amazon.com/blogs/containers/deploy-applications-on-amazon-ecs-using-docker-compose/ https://aws.amazon.com/blogs/containers/deploy-applications-... for example.
- cs02rm0 5y ago
  I don't think you would in prod, unless you were happy with it all running on one host. Shame really, I find compose simple and neat and k8s+terraform an unsettling mess.
  - 411111111111111 5y ago
    Docker swarm deploys to multiple hosts and there is also kompose for k8s. I haven't used either of them in production though
  - walleeee 5y ago
    The single host model is really pretty common Personal or small/medium business sites, research software, free web utilities What's considered lean in startupland can get pretty beefy compared to the shoestring a lot of projects are compelled by necessity to get by on Compose can be great if you just need a web server and maybe a database and background worker and you don't have anybody suing you or losing millions for the occasional maintenance hour
nonameiguess 5y ago
I had remembered docker's own documentation in the past recommending against using compose at all in production, but that is apparently no longer the case: https://docs.docker.com/compose/production/ https://docs.docker.com/compose/production/ I'm fairly surprised by this. You can deploy a fairly similar compose file into a Docker swarm cluster, but if you're doing a single-server deployment of everything, that is awfully brittle for production, to say nothing of considerations brought up in other comments where you may be using managed services from a cloud provider for things like database and caching in production but you're probably just spinning up a local postgres server on your laptop for development.
- reportt 5y ago
  Using the docker-compose.override.yml helps with that. In local you can just have those services/configurations added, and then the production environment doesn't even need to know they exist.
- throwaway2139 5y ago
  https://github.com/docker/compose-on-kubernetes https://github.com/docker/compose-on-kubernetes looks dead now, maybe for misguided political reasons while they remain in denial that swarm might compete? But it's too bad, there's no reason some k8s api/operator thing can't make this frictionless. Still, someone already mentioned docker-desktop having some tricks for deploying straight to aws and [there is definitely a way to do this with ecs](https://aws.amazon.com/blogs/containers/deploy-applications-on-amazon-ecs-using-docker-compose/ https://aws.amazon.com/blogs/containers/deploy-applications-...). There's also tools like kompose which translate configs automatically and are suitable for use in a pipeline, so the only thing you need is a compose file in version control. So yes, you can still basically use compose in production, even on a non-swarm cluster, and it's fine. A lot of people that push back against this are perhaps just invested in their own mad kubectl'ing and endless wrangling with esoteric templates and want to push all this on their teammates. From what I've seen, that's often to the detriment of local development experience and budgeting, because if even your dev environment always requires EKS, that gets expensive. (Using external/managed databases is always a good idea, but beside the main point here I think. You can do that or not with or without docker-compose or helm packages or whatever, and you can even do that for otherwise totally local development on a shared db cluster if you design things for multi-tenant) At this point I'll face facts that the simple myth here (docker-compose is merely a toy) is winning out over the reality (docker-compose is partly a tool but isn't a platform, and it's just a format/description language). But consider.. pure k8s, k8s-helm, ECS cloudformation, k8s-terraforming over EKS, and docker-compose all have pretty stable schemas that require almost all the same data and any one of them could pretty reasonably be considered as a lingua-franca that you could build the other specs from (even programmatically). From this point of view there's an argument that for lots of simple yet serious projects, docker-compose should win by default because it is at the bottom of the complexity ladder. It's almost exactly the minimal usable subset of the abstract description language we're working with, and one that's easy to onboard with and requires the least dependencies to actually run. For example: even without kompose it's trivial to automate pulling data out of the canonical docker-compose yaml and injecting that into terraform as part of CD pipeline for your containers on EKS; then you keep options for local-developer experience open and you're maintaining a central source of truth for common config so that your config/platform is not diverging more than it has to. I'm an architect who works closely with ops, and in many ways not a huge fan of docker-compose. But I like self-service and you-ship-it-you-run it kinds of things that are essential for scaling orgs. So for simple stuff I'd rather just use compose as the main single-source-of-truth than answer endless bootstrappy questions about k8s or ECS if I'm working with others who don't have my depth of knowledge. (Obviously compose has been popular for a reason, which is that kubernetes really is still too complicated for a lot of people and use-cases.) Don't like these ready-made options for compose-on-ECS, or compose-to-k8s via kompose? Ok, just give me your working docker-compose and I'll find a way to deploy it to any other new weird platform, and if I need some pull-values/place-values/render-template song and dance with one more weird DSL for one more weird target deployment platform, then so be it. I've often found the alternative here is a lot of junior devs deciding that deployment/dev-bootstrap is just too confusing, their team doesn't help them and pushes them to an external cloud-engineering team who doesn't want to explain this again because there's docs and 10 examples that went unfollowed, so then junior devs just code without testing it all until they have to when QA is broken. Sometimes the whole org is junior devs in the sense that they have zero existing familiarity with docker, much less kubernetes! Keep things as simple as possible, no simpler. Seen this argument a million times, and no doubt platform choices are important but even pivoting on platforms is surprisingly easy these days. When you consider that compose is not itself a platform but just basically a subset of a wider description language, this all starts to seem a bit like a json vs yaml debate. If you need comments and anchors, then you want yaml. If you need serious packaging/dependencies of a bunch of related microservices, and a bunch of nontrivial JIT value lookup/rendering, then you want helm. But beyond org/situation specific considerations like this, the difference doesn't matter much. My main take-away lately is that leadership needs to actually decide/enforce where the org will stand on topics like "local development workflows"; it's crazy to have a team divided where half is saying "we develop on laptops with docker-compose" and half is saying "we expect to deploy to EKS in the dev environment". In that circumstance you just double your footprint of junk to support and because everyone wants to be perfectly pleased, everyone is annoyed.
peterthehacker 5y ago
Running docker compose in production is not a good practice from what I understand. The persistence layers, like MySQL or Redis, shouldn’t be managed with local volumes. If you want to do something like this then kubernetes with persistent volume claims would be better, but running databases in k8s can be pretty tricky. Most of the time RDS is the best practice for your DB.
- zmmmmm 5y ago
  Swapping out the persistence layer in prod vs dev is one of the topics covered by the article, for the reasons you mention.
  - peterthehacker 5y ago
    There’s no mention of kubernetes persistent volume claims.. it does mention that you can swap out a DATABASE_URL env var here: > Or perhaps you want to use a managed PostgreSQL database in production but run PostgreSQL locally in a container for development. It’s mentioned in passing with no discussion of why you should not run your database in docker compose - a very common use case - so the reasons I mentioned were not explicitly discussed.
oliwary 5y ago
I have really enjoyed working with docker compose for development. For production, there is a great guide by the creator of FastAPI on how to spin up a docker swarm cluster to use almost the same files here: https://dockerswarm.rocks/ https://dockerswarm.rocks/
layoric 5y ago
Not for production, but used it for deployments to remote servers via SSH using GitHub Actions. Wrote a guide and video tutorial of how to as well, it uses nginx proxy and surprisingly updates were zero down time since the nginx proxy didn’t change and handles switching app ports seamlessly. For low traffic app could be used for production but I wouldn’t since it is a single server deployment setup. Good for dev/test environment, I use this pattern for 20+ live demo apps on a single server. Tutorial: https://docs.servicestack.net/do-github-action-mix-deployment https://docs.servicestack.net/do-github-action-mix-deploymen... Video: https://youtu.be/0PvzcnxlBvc https://youtu.be/0PvzcnxlBvc
0xbadcafebee 5y ago
Docker Compose now works natively with AWS ECS, so you could conceivably "scale" Docker Compose. Plus there's AWS's ecs-cli which can work, though IIRC you basically have to maintain two different configs, one for local development, one for remote. And that's the general flaw in using Compose outside of local development. It's simply not designed as a general purpose (and thus large-scale) deployment orchestrator. If you get it working locally, you probably need a different setup for remote. That affects things like testing, which means it's not really the same in dev as in prod. And even if it were, Compose isn't persistent, meaning it's just a script to start jobs on some other thing. Every time I've worked on a team that tried to do Compose in production (or even just for CI) it ended up a weird snowflake or with duplicate processes, and we replaced it with something more standard. My standard advice now is to model both production and local development to be as close as possible and then pick the simplest method to support that use case. In general this means going from "Compose on one host" to Swarm/Nomad/ECS on multiple hosts to K8s. And of course, try to make your CI use the same exact methods you use for dev and prod, so that using CI doesn't give you different results than in dev or prod.
- _asummers 5y ago
  I use Docker Compose for integration testing against DBs in CI, and even for that, have to go through gymnastics working around not having k8s things.
maxrev17 5y ago
Maybe I'm showing my length of tooth here, but what's wrong with systemd? I can run a decently sized API on an AWS nano instance that gracefully deploys/releases when creating a release on github. I've just backed off the docker-compose version of this that I was deploying left right and centre to save cost and reduce complexity - it's a breath of fresh air! Edit: RDS is great for the DB in this setup!
- rzimmerman 5y ago
  I suppose one benefit to docker compose over a strictly systemd setup (even one that runs units in containers) is on the dev side. If your dev team is working on macOS or windows the docker compose tooling will let them spin up the whole system just like on the AWS instance.
  - maxrev17 5y ago
    That's a fair point I did omit the fact that for local database - it's a docker compose file, with docker volumes. That way I can spin up a db per project super easily as can other devs.
- ta988 5y ago
  Do you mind describing a bit more your setup here?
  - maxrev17 5y ago
    I'm super busy (sorry for dropping in and out!) I'll try to do a blog post shortly - and remember to come back here!
    - ta988 5y ago
      Thanks!
rzimmerman 5y ago
It’s widely accepted “best practice” to not run processes in docker containers as root but I think it does a disservice to not explain why. The top benefit in my mind is to be able to mark interpreted code (server side JS/python/ruby) as read only in case someone gains shell access in the container. It’s great to set up another user for execution, but you should also make sure to copy in code and build artifacts as read-only. I’m curious if people can name other direct benefits to a non-root user other than “I don’t trust docker/Linux process isolation”.
- easton 5y ago
  Because giving your app root even to the container could let it blow something up that’s important inside the container, and it means you don’t necessarily know what permissions it actually requires. Then when you get to a client site and they say “Docker isn’t on our approved software list” you won’t know what permissions it needs to install. There’s a low chance of this scenario happening, but lord knows there’s a Unix greybeard somewhere that doesn’t want Docker on his precious pets, and will like to know what permissions your app actually needs.
- Spivak 5y ago
  If you’re using a userns then power to you it’s all the same since you’re not really root. If not then you’re playing a dangerous game because now your container is running as root on the host system which is almost certainly not what you want. The fundamental security boundary on Linux is users and containers are not a sandboxing solution. You can use seccomp, SELinux, gvisor, or honestly virtualization to get real sandboxing and docker can use those tools to create a sandbox but Linux namespaces alone “containers” aren’t that. You can use these tools without any namespacing and vice versa. It’s not a question of “not trusting” process isolation but defense in depth. If you wouldn’t be comfortable running every process on your host system as root and put full trust into SELinux, AppArmor, and seccomp then you shouldn’t start just because you added a high-tech chroot.
dboreham 5y ago
For me, docker-compose (I suppose I should say docker compose these days) has the right balance of moving parts vs functionality to keep me inside the sanity bubble. I've worked with k8s in the past, but...that bubble sure feels good.
- KurtMueller 5y ago
  Have you deployed via docker compose up? If so, how have you found the experience? https://docs.docker.com/cloud/aci-compose-features/ https://docs.docker.com/cloud/aci-compose-features/ https://docs.docker.com/cloud/ecs-compose-features/ https://docs.docker.com/cloud/ecs-compose-features/
  - ngrilly 5y ago
    Using docker compose up to deploy a GraphQL API with ECS at Northvolt. Good experience so far, except for having to rely quite a lot on x-aws-cloudformation overlays to work around the tool limitations. In other words, it helps a lot, but you still need to know about AWS in general and CloudFormation in particular.
pizza 5y ago
Have a question about the safety of using environment variables to store sensitive information. e.g. suppose you have your postgres db url in an environment variable - with the password in the string - isn't that a big no-no? If someone found an exploit in your web app container then they would also in theory get access to valuable secret info - the db data is by far most valuable part of your app imo. Is there a more managed way to handle this kind of secret?
- jffry 5y ago
  If your web app container can query the database, and your web app container is compromised, then yes the attacker could query the database. This is an issue independent of how you conveyed that access to the web app container. If you want to limit the blast radius, you will need to find ways to limit what parts of the database the web app has access to.
- reportt 5y ago
  > If someone found an exploit in your web app container [...] A good pattern here is to reverse proxy all requests to the application through something like nginx. My applications tend to have a back-end application that is not accessible to the internet, with an nginx instance that proxies all API requests itself. Only port 80 is public facing. If someone can get console access to an nginx container and then use that to springboard to another container and get root access there (again, where the only open ports are ports 80, maybe 8000?) to get envvars, they should get access to it all. If you are worried about secrets, check out the Docker Compose 3.9 documentation: https://docs.docker.com/compose/compose-file/compose-file-v3/#secrets https://docs.docker.com/compose/compose-file/compose-file-v3...
- deckard1 5y ago
  if the exploit is in your web app, then you're already too late. Security is about layering. Which is why postgres/mysql have users and permissions. One thing I typically do is prevent DELETE statements from the web app db user. You can also prevent access to tables or even columns. Or you can go in the opposite direction and YOLO-it by putting SQLite right in your web app process, following the recent craze. Environment variables are typically how secrets are used. Anything else is going to add further complexity and have another set of caveats. Env vars are simple and all devs (and programming languages) know how they work.
gassius 5y ago
I run my personal services (Web, db, mail, nextcloud, traefick) off a cheap hetzner vps using docker-compose and its a pleasure how simple is to manage it. Before that I ran a vps with manually configured services and, specially for mail, it was very hard to deploy and manage. Also if p0wned a service, the whole box was as good as gone. Now, there are other alternatives with less downtime, more scalability and a lot more or complexity? Sure there are... but for my personal box I won't be touching any of that, thanks Edit/Add: ... and also VPN, which I added later on in like, 10m or so. Adding OpenVPN in my previous box was not as trivial, I tell you that.
- jtsiskin 5y ago
  Check out https://caprover.com/ https://caprover.com/ - nothing cooler being able to deploy postgres from your phone
  - speedgoose 5y ago
    On the list of cool things, deploying postgresql from a phone is most likely pretty down the list. I think my partner would find more cool that I clean the kitchen for example.
- zmmmmm 5y ago
  yeah .... compose sits at a really sweet spot on the complexity-power tradeoff spectrum. Yes it doesn't do a bunch of stuff, but what you get for how simple it is to do that is really compelling.
- rualca 5y ago
  > I run my personal services (Web, db, mail, nextcloud, traefick) off a cheap hetzner vps using docker-compose and its a pleasure how simple is to manage it. Me too. I have about half a dozen personal projects running in Docker Swarm on Hetzner. I have an Ansible script that gets a freshly created barebones VM to serve live traffic in less than a minute, and I am free to just delete all VMs and reinstall from scratch without any problem at all. I even split the Ansible playbooks into stack-specific playbooks so that I can redeploy specific stacks if I feel like it. I firmly believe that Docker Swarm is a killer application that is not given the attention it deserves because everyone is stuck in a mindset where they purposely want to make their own lives harder by doing CV-oriented development while running after the devops version of doing enterprise versions of hello world.
sneak 5y ago
Chaining app deps in a healthcheck (db, redis) is a great way to trigger a web node autoscaling catastrophe when your redis or db goes down for a few minutes for maintenance. The health check should only include the health of the service it resides within, not its dependencies.
SrslyJosh 5y ago
Lots of overlap with the twelve-factor app: https://12factor.net https://12factor.net
sandGorgon 5y ago
FYI, Compose is now an independent standard - https://www.compose-spec.io/ https://www.compose-spec.io/ Kubernetes using Compose would be a killer product. People would pay for it. Its funny why nobody built it. Interestingly, now there is Cue lang + Buildkit - https://docker.events.cube365.net/dockercon-live/2021/content/Videos/TamRA3F2zT8fycvq5 https://docker.events.cube365.net/dockercon-live/2021/conten...
aitchnyu 5y ago
I have a Django+Vue app with Docker Compose and I mounted the whole project as a volume for both python and js containers and the venv (python packages are installed here) and node_modules are in the volume. My IDE (Pycharm) uses those dirs for code completion. If I move them within container, my IDE can't do code completion and I will have to rebuild image each time I install packages. Or is there a solution?
- nickjj 5y ago
  I'm not sure how PyCharm deals with Docker but VSCode has a "remote containers" feature. You can run your app in a container and volume mount your code so you don't need to rebuild it, but the remote container feature lets you get full editor support (code complete for everything, etc.) with only defining a few lines of JSON. Maybe check around to see if PyCharm has a similar feature?
jiehong 5y ago
I personally use minikube[0] to have local deployments and production deployments to be very similar. Minikube is your personal kubernetes, and it's very to use, and is very close to the production kubernetes, but on a single node. I don't see enough praise for minikube on HN, but I think it's deserved. Albeit, you cannot do a "docker-compose up --build", but instead "docker-compose build && kub apply -f <template.yaml>", which is equally fast IMO. [0]: https://minikube.sigs.k8s.io/docs/ https://minikube.sigs.k8s.io/docs/
- husarcik 5y ago
  Ah, interesting! Any downsides to this you've run into?
  - driton 5y ago
    It's been a few years since I tried using it, so take this with a grain of salt, but my biggest gripe with it was that it would grind my computer (a beefy Dell) to a halt. Of course, this depends on what sort of environment you're trying to replicate. If it's 2 to 3 microservices and a database, I would imagine it's fine, but anything more, which is what I had to deal with at the time, it was a no-go. Especially so since this meant that running minikube and having a RAM-thirsty IDE, like Intellij IDEA, was practically impossible.
    - EricLeer 5y ago
      Ram usage is a lot better when using docker as a driver instead of a vm, which immediately reserves all memory you give it. see https://minikube.sigs.k8s.io/docs/drivers/ https://minikube.sigs.k8s.io/docs/drivers/ for more info
Vobletop 5y ago
I have experience with Docker Compose just a couple of years and I found this article to be very helpful. It contains concentrated know how accumulated over the years. Thank you for sharing this with us. It would be helpful if sub-topics have dedicated anchors. This way I can add direct links into source code or git commits if I want to point out more detailed explanation why this change was good.
Ambix 5y ago
Second this. I really like to simplify all things, so sometime it reasonable to use systemd and Makefile with start, stop and update commands. Works in production like a charm.

514 points by nickjj 5y ago

znpy 5y ago

Pro-tip: nowadays podman can pretend to be a kubernetes cluster and launch pod And deployment definitions (in yaml, of course). That of course is way close to the "production" in "production ready".

ecliptik 5y ago
That sounds really cool. Do you happen to have a good link for this?
- buildbot 5y ago
  Not GP, but Perhaps https://www.redhat.com/sysadmin/compose-kubernetes-podman https://www.redhat.com/sysadmin/compose-kubernetes-podman?
buildbot 5y ago
Without podman-compose too! I didn't know this was the case now: https://www.redhat.com/sysadmin/compose-kubernetes-podman https://www.redhat.com/sysadmin/compose-kubernetes-podman
2OEH8eoCRo0 5y ago
Also daemonless and rootless
- znpy 5y ago
  This is one of the coolest features! Different users run different containers, and no user can see other user's containers.
- deckard1 5y ago
  docker has rootless now. Tried to set it up the other day on bog standard Debian. That's a deep ass rabbit hole with a whole lot of gotchas and I'm not entirely convinced the whole thing won't topple over at the slightest breeze. It took me an embarrassingly long time to figure out how to install the correct version of slirp4netns, which is not in the stable repo. Reassuring.
  - emerongi 5y ago
    Similar experience. I used Docker rootless for a while 1-2 years ago. There were hiccups and upgrading was a pain, so I stopped using it. It's possible it has improved, though.
  - znpy 5y ago
    It works out of the box on rhel 8.3, and I've used succession Ubuntu 20.04 installed by the PPA repos.
0xbadcafebee 5y ago
....but then when you run it in production, your app fails, because it was designed against assumptions around the fake cluster and pod, or a different config file. I'm not saying there's a solution to this, but we should remind people that "we can pretend X is Y" is a reproducibility red flag.
- znpy 5y ago
  There are step between development laptop and production you know, smarty pants? Like, a staging cluster? A pre-prod cluster?
  - 0xbadcafebee 5y ago
    Which then creates its own special snowflake environment which isn't either dev or prod, and duplicates resources, and can only test one change at a time (unless you have real ephemeral environments, and most people don't). It's better to have blue/green, or something like blue green, and use that as staging, than have an actual dedicated "staging cluster". But like I say, dev still is far from close to that. So you end up doing a lot of development in dev, and then fixing a ton of things in staging because you weren't developing against staging. And your tests aren't fantastic, so some bugs make it through both staging and prod, and it turns out they're because your design was making assumptions based on dev. Hence, make dev and prod as close as possible, and then you don't need to have "staging", because you have confidence that what worked in dev will also work in prod.
messo 5y ago
Podman is great! I was surprised to learn that the latest version is compatible with docker-compose (minus the Docker Swarm part). I have been playing around with it on my workstation (Arch/Podman/docker-compose) and it works really well, rootless and without any daemon.

barbazoo 5y ago

> web: > <<: default-app > ... > worker: > <<: default-app This is always somewhat of a code smell to me. This makes me want to dig deeper and see if the two use cases shouldn't have been separated into separate Docker files or even separate apps.

TameAntelope 5y ago
I tend to start with monolith and split things out when I get frustrated with how coupled they are.
pentium166 5y ago
It seems reasonable to me (for a compose-in-prod setup). The standard setup (at least for Rails, in my experience) is that the web server and background job processor will have the same dependencies and nearly identical configuration. Until the app has gone off the rails and I really do need to have the server and worker diverge, this approach seems like less to worry about than maintaining multiple images and duplicated but 98% identical container definitions.
- Ensorceled 5y ago
  Yeah, same with Django ... really try to keep these environments all in sync so you don't have different environments for no real gain. What do I gain when my worker doesn't have django-allauth installed other than possible development environment complexity.

wiktorcie 5y ago

just use dokku instead, it's much easier. Why bother with docker compose?

clintonb 5y ago
Dokku has a lot of overhead to run.

mdoms 5y ago

I don't understand his "Avoiding 2 Compose Files for Dev and Prod with an Override File" point. He just says it, but never justifies it.

randompwd 5y ago
It's a terrible idea that's why. It comes with far more(&more dangerous) downsides than upsides. It's a rookie mistake for simple applications which most people learn to not do.
reportt 5y ago
The idea is that you have a core structure that is used in both local/dev environment and the prod environment, with just the override applying changes. For example, your local override might have the "build" configuration, whereas the dev environment or production environment docker-compose should not have that configuration (assuming you are pulling from a private registry). That single core docker-compose.yml can be copied to any environment it needs and it should just run the app, but if you really, really need to override something for a specific reason, it's not hard to do it without modifying the core docker-compose.yml.
trinovantes 5y ago
Probably so you don't have duplicate settings in docker-compose.prod.yml and docker-compose.dev.yml when the only difference is probably just 1 line specifying NODE_ENV etc.
nickjj 5y ago
Hi, It's mentioned on video and briefly in the blog post. It's so you don't have to duplicate a bunch of services between 2 files while you slightly change a bit of configuring for those services depending on the environment you're in, or even running different containers all together (which is the problem the override file solves).

stevebmark 5y ago

I haven't heard of using docker compose for production development, only local development. In production I normally see people building production images and spinning up them up in their cloud with Terraform and whatever other infrastructure they need, especially for databases where your local MySQL container isn't going to be configured or provisioned like an AWS RDS instance. What do you hand a docker-compose file off to in production? Swarm? When would you choose this path over a separate system to provision your production resources (like Terraform)?

namdnay 5y ago
Same here, docker compose is a great little tool for local development. You can get your apps up and running with their respective elasticsearches and localstacks etc, then attach to process if needed to debug For anything more serious, Terraform makes much more sense IMHO
ensignavenger 5y ago
Docker Swarm is one option, but Docker Desktop has the ability to deploy from a Compose file directly to AWS. There are additional tools for other platforms. See https://aws.amazon.com/blogs/containers/deploy-applications-on-amazon-ecs-using-docker-compose/ https://aws.amazon.com/blogs/containers/deploy-applications-... for example.
cs02rm0 5y ago
I don't think you would in prod, unless you were happy with it all running on one host. Shame really, I find compose simple and neat and k8s+terraform an unsettling mess.
- 411111111111111 5y ago
  Docker swarm deploys to multiple hosts and there is also kompose for k8s. I haven't used either of them in production though
- walleeee 5y ago
  The single host model is really pretty common Personal or small/medium business sites, research software, free web utilities What's considered lean in startupland can get pretty beefy compared to the shoestring a lot of projects are compelled by necessity to get by on Compose can be great if you just need a web server and maybe a database and background worker and you don't have anybody suing you or losing millions for the occasional maintenance hour

nonameiguess 5y ago

I had remembered docker's own documentation in the past recommending against using compose at all in production, but that is apparently no longer the case: https://docs.docker.com/compose/production/ https://docs.docker.com/compose/production/ I'm fairly surprised by this. You can deploy a fairly similar compose file into a Docker swarm cluster, but if you're doing a single-server deployment of everything, that is awfully brittle for production, to say nothing of considerations brought up in other comments where you may be using managed services from a cloud provider for things like database and caching in production but you're probably just spinning up a local postgres server on your laptop for development.

reportt 5y ago
Using the docker-compose.override.yml helps with that. In local you can just have those services/configurations added, and then the production environment doesn't even need to know they exist.
throwaway2139 5y ago
https://github.com/docker/compose-on-kubernetes https://github.com/docker/compose-on-kubernetes looks dead now, maybe for misguided political reasons while they remain in denial that swarm might compete? But it's too bad, there's no reason some k8s api/operator thing can't make this frictionless. Still, someone already mentioned docker-desktop having some tricks for deploying straight to aws and [there is definitely a way to do this with ecs](https://aws.amazon.com/blogs/containers/deploy-applications-on-amazon-ecs-using-docker-compose/ https://aws.amazon.com/blogs/containers/deploy-applications-...). There's also tools like kompose which translate configs automatically and are suitable for use in a pipeline, so the only thing you need is a compose file in version control. So yes, you can still basically use compose in production, even on a non-swarm cluster, and it's fine. A lot of people that push back against this are perhaps just invested in their own mad kubectl'ing and endless wrangling with esoteric templates and want to push all this on their teammates. From what I've seen, that's often to the detriment of local development experience and budgeting, because if even your dev environment always requires EKS, that gets expensive. (Using external/managed databases is always a good idea, but beside the main point here I think. You can do that or not with or without docker-compose or helm packages or whatever, and you can even do that for otherwise totally local development on a shared db cluster if you design things for multi-tenant) At this point I'll face facts that the simple myth here (docker-compose is merely a toy) is winning out over the reality (docker-compose is partly a tool but isn't a platform, and it's just a format/description language). But consider.. pure k8s, k8s-helm, ECS cloudformation, k8s-terraforming over EKS, and docker-compose all have pretty stable schemas that require almost all the same data and any one of them could pretty reasonably be considered as a lingua-franca that you could build the other specs from (even programmatically). From this point of view there's an argument that for lots of simple yet serious projects, docker-compose should win by default because it is at the bottom of the complexity ladder. It's almost exactly the minimal usable subset of the abstract description language we're working with, and one that's easy to onboard with and requires the least dependencies to actually run. For example: even without kompose it's trivial to automate pulling data out of the canonical docker-compose yaml and injecting that into terraform as part of CD pipeline for your containers on EKS; then you keep options for local-developer experience open and you're maintaining a central source of truth for common config so that your config/platform is not diverging more than it has to. I'm an architect who works closely with ops, and in many ways not a huge fan of docker-compose. But I like self-service and you-ship-it-you-run it kinds of things that are essential for scaling orgs. So for simple stuff I'd rather just use compose as the main single-source-of-truth than answer endless bootstrappy questions about k8s or ECS if I'm working with others who don't have my depth of knowledge. (Obviously compose has been popular for a reason, which is that kubernetes really is still too complicated for a lot of people and use-cases.) Don't like these ready-made options for compose-on-ECS, or compose-to-k8s via kompose? Ok, just give me your working docker-compose and I'll find a way to deploy it to any other new weird platform, and if I need some pull-values/place-values/render-template song and dance with one more weird DSL for one more weird target deployment platform, then so be it. I've often found the alternative here is a lot of junior devs deciding that deployment/dev-bootstrap is just too confusing, their team doesn't help them and pushes them to an external cloud-engineering team who doesn't want to explain this again because there's docs and 10 examples that went unfollowed, so then junior devs just code without testing it all until they have to when QA is broken. Sometimes the whole org is junior devs in the sense that they have zero existing familiarity with docker, much less kubernetes! Keep things as simple as possible, no simpler. Seen this argument a million times, and no doubt platform choices are important but even pivoting on platforms is surprisingly easy these days. When you consider that compose is not itself a platform but just basically a subset of a wider description language, this all starts to seem a bit like a json vs yaml debate. If you need comments and anchors, then you want yaml. If you need serious packaging/dependencies of a bunch of related microservices, and a bunch of nontrivial JIT value lookup/rendering, then you want helm. But beyond org/situation specific considerations like this, the difference doesn't matter much. My main take-away lately is that leadership needs to actually decide/enforce where the org will stand on topics like "local development workflows"; it's crazy to have a team divided where half is saying "we develop on laptops with docker-compose" and half is saying "we expect to deploy to EKS in the dev environment". In that circumstance you just double your footprint of junk to support and because everyone wants to be perfectly pleased, everyone is annoyed.

peterthehacker 5y ago

Running docker compose in production is not a good practice from what I understand. The persistence layers, like MySQL or Redis, shouldn’t be managed with local volumes. If you want to do something like this then kubernetes with persistent volume claims would be better, but running databases in k8s can be pretty tricky. Most of the time RDS is the best practice for your DB.

zmmmmm 5y ago
Swapping out the persistence layer in prod vs dev is one of the topics covered by the article, for the reasons you mention.
- peterthehacker 5y ago
  There’s no mention of kubernetes persistent volume claims.. it does mention that you can swap out a DATABASE_URL env var here: > Or perhaps you want to use a managed PostgreSQL database in production but run PostgreSQL locally in a container for development. It’s mentioned in passing with no discussion of why you should not run your database in docker compose - a very common use case - so the reasons I mentioned were not explicitly discussed.

oliwary 5y ago

I have really enjoyed working with docker compose for development. For production, there is a great guide by the creator of FastAPI on how to spin up a docker swarm cluster to use almost the same files here: https://dockerswarm.rocks/ https://dockerswarm.rocks/

layoric 5y ago

Not for production, but used it for deployments to remote servers via SSH using GitHub Actions. Wrote a guide and video tutorial of how to as well, it uses nginx proxy and surprisingly updates were zero down time since the nginx proxy didn’t change and handles switching app ports seamlessly. For low traffic app could be used for production but I wouldn’t since it is a single server deployment setup. Good for dev/test environment, I use this pattern for 20+ live demo apps on a single server. Tutorial: https://docs.servicestack.net/do-github-action-mix-deployment https://docs.servicestack.net/do-github-action-mix-deploymen... Video: https://youtu.be/0PvzcnxlBvc https://youtu.be/0PvzcnxlBvc

0xbadcafebee 5y ago

Docker Compose now works natively with AWS ECS, so you could conceivably "scale" Docker Compose. Plus there's AWS's ecs-cli which can work, though IIRC you basically have to maintain two different configs, one for local development, one for remote. And that's the general flaw in using Compose outside of local development. It's simply not designed as a general purpose (and thus large-scale) deployment orchestrator. If you get it working locally, you probably need a different setup for remote. That affects things like testing, which means it's not really the same in dev as in prod. And even if it were, Compose isn't persistent, meaning it's just a script to start jobs on some other thing. Every time I've worked on a team that tried to do Compose in production (or even just for CI) it ended up a weird snowflake or with duplicate processes, and we replaced it with something more standard. My standard advice now is to model both production and local development to be as close as possible and then pick the simplest method to support that use case. In general this means going from "Compose on one host" to Swarm/Nomad/ECS on multiple hosts to K8s. And of course, try to make your CI use the same exact methods you use for dev and prod, so that using CI doesn't give you different results than in dev or prod.

_asummers 5y ago
I use Docker Compose for integration testing against DBs in CI, and even for that, have to go through gymnastics working around not having k8s things.

maxrev17 5y ago

Maybe I'm showing my length of tooth here, but what's wrong with systemd? I can run a decently sized API on an AWS nano instance that gracefully deploys/releases when creating a release on github. I've just backed off the docker-compose version of this that I was deploying left right and centre to save cost and reduce complexity - it's a breath of fresh air! Edit: RDS is great for the DB in this setup!

rzimmerman 5y ago
I suppose one benefit to docker compose over a strictly systemd setup (even one that runs units in containers) is on the dev side. If your dev team is working on macOS or windows the docker compose tooling will let them spin up the whole system just like on the AWS instance.
- maxrev17 5y ago
  That's a fair point I did omit the fact that for local database - it's a docker compose file, with docker volumes. That way I can spin up a db per project super easily as can other devs.
ta988 5y ago
Do you mind describing a bit more your setup here?
- maxrev17 5y ago
  I'm super busy (sorry for dropping in and out!) I'll try to do a blog post shortly - and remember to come back here!
  - ta988 5y ago
    Thanks!

rzimmerman 5y ago

It’s widely accepted “best practice” to not run processes in docker containers as root but I think it does a disservice to not explain why. The top benefit in my mind is to be able to mark interpreted code (server side JS/python/ruby) as read only in case someone gains shell access in the container. It’s great to set up another user for execution, but you should also make sure to copy in code and build artifacts as read-only. I’m curious if people can name other direct benefits to a non-root user other than “I don’t trust docker/Linux process isolation”.

easton 5y ago
Because giving your app root even to the container could let it blow something up that’s important inside the container, and it means you don’t necessarily know what permissions it actually requires. Then when you get to a client site and they say “Docker isn’t on our approved software list” you won’t know what permissions it needs to install. There’s a low chance of this scenario happening, but lord knows there’s a Unix greybeard somewhere that doesn’t want Docker on his precious pets, and will like to know what permissions your app actually needs.
Spivak 5y ago
If you’re using a userns then power to you it’s all the same since you’re not really root. If not then you’re playing a dangerous game because now your container is running as root on the host system which is almost certainly not what you want. The fundamental security boundary on Linux is users and containers are not a sandboxing solution. You can use seccomp, SELinux, gvisor, or honestly virtualization to get real sandboxing and docker can use those tools to create a sandbox but Linux namespaces alone “containers” aren’t that. You can use these tools without any namespacing and vice versa. It’s not a question of “not trusting” process isolation but defense in depth. If you wouldn’t be comfortable running every process on your host system as root and put full trust into SELinux, AppArmor, and seccomp then you shouldn’t start just because you added a high-tech chroot.

dboreham 5y ago

For me, docker-compose (I suppose I should say docker compose these days) has the right balance of moving parts vs functionality to keep me inside the sanity bubble. I've worked with k8s in the past, but...that bubble sure feels good.

KurtMueller 5y ago
Have you deployed via docker compose up? If so, how have you found the experience? https://docs.docker.com/cloud/aci-compose-features/ https://docs.docker.com/cloud/aci-compose-features/ https://docs.docker.com/cloud/ecs-compose-features/ https://docs.docker.com/cloud/ecs-compose-features/
- ngrilly 5y ago
  Using docker compose up to deploy a GraphQL API with ECS at Northvolt. Good experience so far, except for having to rely quite a lot on x-aws-cloudformation overlays to work around the tool limitations. In other words, it helps a lot, but you still need to know about AWS in general and CloudFormation in particular.

pizza 5y ago

Have a question about the safety of using environment variables to store sensitive information. e.g. suppose you have your postgres db url in an environment variable - with the password in the string - isn't that a big no-no? If someone found an exploit in your web app container then they would also in theory get access to valuable secret info - the db data is by far most valuable part of your app imo. Is there a more managed way to handle this kind of secret?

jffry 5y ago
If your web app container can query the database, and your web app container is compromised, then yes the attacker could query the database. This is an issue independent of how you conveyed that access to the web app container. If you want to limit the blast radius, you will need to find ways to limit what parts of the database the web app has access to.
reportt 5y ago
> If someone found an exploit in your web app container [...] A good pattern here is to reverse proxy all requests to the application through something like nginx. My applications tend to have a back-end application that is not accessible to the internet, with an nginx instance that proxies all API requests itself. Only port 80 is public facing. If someone can get console access to an nginx container and then use that to springboard to another container and get root access there (again, where the only open ports are ports 80, maybe 8000?) to get envvars, they should get access to it all. If you are worried about secrets, check out the Docker Compose 3.9 documentation: https://docs.docker.com/compose/compose-file/compose-file-v3/#secrets https://docs.docker.com/compose/compose-file/compose-file-v3...
deckard1 5y ago
if the exploit is in your web app, then you're already too late. Security is about layering. Which is why postgres/mysql have users and permissions. One thing I typically do is prevent DELETE statements from the web app db user. You can also prevent access to tables or even columns. Or you can go in the opposite direction and YOLO-it by putting SQLite right in your web app process, following the recent craze. Environment variables are typically how secrets are used. Anything else is going to add further complexity and have another set of caveats. Env vars are simple and all devs (and programming languages) know how they work.

gassius 5y ago

I run my personal services (Web, db, mail, nextcloud, traefick) off a cheap hetzner vps using docker-compose and its a pleasure how simple is to manage it. Before that I ran a vps with manually configured services and, specially for mail, it was very hard to deploy and manage. Also if p0wned a service, the whole box was as good as gone. Now, there are other alternatives with less downtime, more scalability and a lot more or complexity? Sure there are... but for my personal box I won't be touching any of that, thanks Edit/Add: ... and also VPN, which I added later on in like, 10m or so. Adding OpenVPN in my previous box was not as trivial, I tell you that.

jtsiskin 5y ago
Check out https://caprover.com/ https://caprover.com/ - nothing cooler being able to deploy postgres from your phone
- speedgoose 5y ago
  On the list of cool things, deploying postgresql from a phone is most likely pretty down the list. I think my partner would find more cool that I clean the kitchen for example.
zmmmmm 5y ago
yeah .... compose sits at a really sweet spot on the complexity-power tradeoff spectrum. Yes it doesn't do a bunch of stuff, but what you get for how simple it is to do that is really compelling.
rualca 5y ago
> I run my personal services (Web, db, mail, nextcloud, traefick) off a cheap hetzner vps using docker-compose and its a pleasure how simple is to manage it. Me too. I have about half a dozen personal projects running in Docker Swarm on Hetzner. I have an Ansible script that gets a freshly created barebones VM to serve live traffic in less than a minute, and I am free to just delete all VMs and reinstall from scratch without any problem at all. I even split the Ansible playbooks into stack-specific playbooks so that I can redeploy specific stacks if I feel like it. I firmly believe that Docker Swarm is a killer application that is not given the attention it deserves because everyone is stuck in a mindset where they purposely want to make their own lives harder by doing CV-oriented development while running after the devops version of doing enterprise versions of hello world.

sneak 5y ago

Chaining app deps in a healthcheck (db, redis) is a great way to trigger a web node autoscaling catastrophe when your redis or db goes down for a few minutes for maintenance. The health check should only include the health of the service it resides within, not its dependencies.

SrslyJosh 5y ago

Lots of overlap with the twelve-factor app: https://12factor.net https://12factor.net

sandGorgon 5y ago

FYI, Compose is now an independent standard - https://www.compose-spec.io/ https://www.compose-spec.io/ Kubernetes using Compose would be a killer product. People would pay for it. Its funny why nobody built it. Interestingly, now there is Cue lang + Buildkit - https://docker.events.cube365.net/dockercon-live/2021/content/Videos/TamRA3F2zT8fycvq5 https://docker.events.cube365.net/dockercon-live/2021/conten...

aitchnyu 5y ago

I have a Django+Vue app with Docker Compose and I mounted the whole project as a volume for both python and js containers and the venv (python packages are installed here) and node_modules are in the volume. My IDE (Pycharm) uses those dirs for code completion. If I move them within container, my IDE can't do code completion and I will have to rebuild image each time I install packages. Or is there a solution?

nickjj 5y ago
I'm not sure how PyCharm deals with Docker but VSCode has a "remote containers" feature. You can run your app in a container and volume mount your code so you don't need to rebuild it, but the remote container feature lets you get full editor support (code complete for everything, etc.) with only defining a few lines of JSON. Maybe check around to see if PyCharm has a similar feature?

jiehong 5y ago

I personally use minikube[0] to have local deployments and production deployments to be very similar. Minikube is your personal kubernetes, and it's very to use, and is very close to the production kubernetes, but on a single node. I don't see enough praise for minikube on HN, but I think it's deserved. Albeit, you cannot do a "docker-compose up --build", but instead "docker-compose build && kub apply -f <template.yaml>", which is equally fast IMO. [0]: https://minikube.sigs.k8s.io/docs/ https://minikube.sigs.k8s.io/docs/

husarcik 5y ago
Ah, interesting! Any downsides to this you've run into?
- driton 5y ago
  It's been a few years since I tried using it, so take this with a grain of salt, but my biggest gripe with it was that it would grind my computer (a beefy Dell) to a halt. Of course, this depends on what sort of environment you're trying to replicate. If it's 2 to 3 microservices and a database, I would imagine it's fine, but anything more, which is what I had to deal with at the time, it was a no-go. Especially so since this meant that running minikube and having a RAM-thirsty IDE, like Intellij IDEA, was practically impossible.
  - EricLeer 5y ago
    Ram usage is a lot better when using docker as a driver instead of a vm, which immediately reserves all memory you give it. see https://minikube.sigs.k8s.io/docs/drivers/ https://minikube.sigs.k8s.io/docs/drivers/ for more info

Vobletop 5y ago

I have experience with Docker Compose just a couple of years and I found this article to be very helpful. It contains concentrated know how accumulated over the years. Thank you for sharing this with us. It would be helpful if sub-topics have dedicated anchors. This way I can add direct links into source code or git commits if I want to point out more detailed explanation why this change was good.

Ambix 5y ago

Second this. I really like to simplify all things, so sometime it reasonable to use systemd and Makefile with start, stop and update commands. Works in production like a charm.