12 ms·

The Docker Compose Specification

251 points by alexellisuk 7y ago

justincormack 7y ago
Previously Compose did not have a specification, only the implementation, and it was tied to Docker's products. The aim here is to open it up, and let the specification evolve in new directions, such as better supporting deployment on Kubernetes, as an example. It is supported with some other implementations but we want to bring these closer together. Compose has a lot of users, and hundreds of thousands of Compose files on GitHub, so we wanted to open up development as much as possible. (I work at Docker)
- deleted 7y ago
  [deleted]
- liveoneggs 7y ago
  Does this spec carry forward the v2 stuff into v3 that would otherwise require --compatibility to function without swarm?
  - justincormack 7y ago
    So far we have not made changes to the spec, just carried it forward but we definitely want to start cleaning up the v2 v3 mess, and make a single spec work for all platforms. See https://github.com/compose-spec/compose-spec/issues/12 https://github.com/compose-spec/compose-spec/issues/12 and also https://github.com/compose-spec/compose-spec/issues/13 https://github.com/compose-spec/compose-spec/issues/13 for making it feature not version number driven.
    - znpy 7y ago
      <insert python joke here>
    - liveoneggs 7y ago
      I would be surprised if there is the political will at docker to do this when it wasn't a priority for compose itself
      - fapjacks 7y ago
        Docker (Inc) has new priorities and different influences. I'm inclined to feel some optimism that this shifting focus will have beneficial repercussions in places like this.
  - geerlingguy 7y ago
    This. My current usage is a muddy mess of v2 and v3 specific features an some require the compatibility flag while others don't.
- e12e 7y ago
  > better supporting deployment Is there any "supported deployment" for docker? I had the impression that swarm has been sidelined, and docker-compose was discouraged. Is there a path to deployment on kubernetes that uses only compose? (I have a hard time seeing how that'll work with ingress, and "compose" well with other kubernetes deployments?) We have some on-prem stuff that would work on single-node swarm - but I've felt that compose has been in a bit of limbo this past year or so... Is some kind of production style deployment comming back?
  - hannibalhorn 7y ago
    "Compose on Kubernetes"[1] seems to be what they're advocating at the moment, and it's included in Docker Desktop. It's probably an OK first step for simpler deployments, though I suspect in order to do any troubleshooting, one would have to learn k8s anyway. 1. https://github.com/docker/compose-on-kubernetes https://github.com/docker/compose-on-kubernetes
    - mfer 7y ago
      To debug an application running on linux one does not need to learn machine code or assembly. There are tools that make it all possible. Higher level abstractions can work and need to start somewhere. It really depends what tools are developed.
      - e12e 7y ago
        Have you tried moving from on-premise docker-compose to managed kubernetes in the cloud? If you've been diligent in striving for stateless containers that will help you - but pretty much everything thing else in your compose file is useless. Docker-compose gets you the "pod"-level in k8s, but doesn't really help with ingress/services etc. So no one can reach your app, certainly not via a load balancer that terminate ssl for you. This is no longer a game of "beating the nginx container until it gives up and does what you need it to".
        mfer 7y ago
- amerine 7y ago
  Hi Justin, I work at Heroku for Salesforce. I'm unsure if you've worked with anyone here in the past (like on our older app.yml, heroku.yml and constellations, etc stuff). But I just wanted to say thanks for establishing a standard for this and helping the rest of us in this space continue to evolve.
- mcny 7y ago
  Hi, I would like to use docker-compose without using docker. Specifically, I’d like to run devil box (php development environment) on fedora using pod man and docker-compose. Is that possible now?
- girvo 7y ago
  I'm super excited for this. Back when Docker Compose was still called Fig, I got annoyed at how it handled volume-only containers for databases, and wrote Dup [1]. Nowadays Compose is way WAY more useful, but there's a number of projects that still rely on it -- the spec means I can write a converter for the config files (from Dup to Compose) and be sure that it will work :) [1] https://github.com/girvo/dup https://github.com/girvo/dup
L_226 7y ago
Sweet, looks good. I reckon docker and docker-compose have been probably some of the greatest enablers for my development process, and how I think about desiging software systems.
- daveguy 7y ago
  Agreed. I was worried docker-compose might be abandoned for swarm-only at the kubernetes level. Compose is a much easier level to develop and work with. Doesn't do all the balancing and HA, but for most projects you don't need that, just good organization of the pieces.
  - kalev 7y ago
    This is exactly why we still use compose as well. Everybody is talking about kubernetes, but I honestly don’t see the point in adding a huge dependency only for balancing and restarting failed containers.. I just wish there was some SaaS platform which allows you to upload a docker-compose file and run the containers for you. That way, you don’t have to touch the underlying OS.
    - larrybud 7y ago
      Azure App service lets you do this. https://docs.microsoft.com/en-us/azure/app-service/containers/quickstart-multi-container https://docs.microsoft.com/en-us/azure/app-service/container...
    - tilolebo 7y ago
      The AWS ECS cli has this kind of feature (I never used it) : https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-compose.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... v2 even allows to create the underlying networking resources (VPC), provision an application load balancer, etc.
  - jswny 7y ago
    Aren't Compose and Swarm/Stack very similar, but Stack has support for distribution between multiple nodes? I know their actual configurations aren't equivalent, but they are similar (Stack supports config options for distribution, etc.).
clSTophEjUdRanu 7y ago
This is cool- hopefully this allows standardized podman-compose.
- jabl 7y ago
  There is https://github.com/abalage/pods-compose https://github.com/abalage/pods-compose
earthboundkid 7y ago
This is good. My main annoyance with Docker in the past was that I would make a decent Compose file for local dev but then have to rewrite everything in a proprietary YAML format for use in CI and prod.
- peterwwillis 7y ago
  You'll still be doing that with this new spec. It's still just a "how would you like me to run multiple containers" spec, it's not defining the relationships of arbitrary infrastructure and services on arbitrary networks/providers. What's going to happen is, you (or the dev) will fill this thing out to the best of your ability. Then someone's going to run it in one env; it won't work, but Ops will modify it to make it work. So then you'll commit that as your one Compose file. Then someone will try to run it in a different env, and it won't work; they'll fix it to make it run. Now you have two Compose files to maintain, with 3 different use cases. So to make it easier to manage, you go back to one for dev, and one for each env. All this because the different components could not be defined independently and loosely coupled, because the spec (and tools) don't support that. I like Compose for what it's designed for: let a dev mock something up locally. But it's going to become a pain in any environment that scales and will need a different solution.
crb 7y ago
> Docker is working with Amazon Web Services (AWS), Microsoft and others in the open source community Google?
move-on-by 7y ago
This is great news. Docker really screwed up v2->v3 and have continually suppressed Compose’s potential because it wasn’t inline with swarm. Looking forward to Compose getting some new life put into it.
- cpuguy83 7y ago
  Suppressed is not quite the right word. Too many inputs (from the community) and not enough help managing and maintaining, and curating these inputs is really what this is. That said, I don't personally believe there exists a single specification that can work (optimally) for both local-dev and production deployments. Disclaimer: ex-Docker engineer, still an active maintainer on the main docker project.
- hinkley 7y ago
  You may recall that compose used to be called fig, and they merged into Docker. I wouldn't be surprised if there was some Us vs Them politics going on there.
andybak 7y ago
I've been mostly avoiding Docker, Docker-compose etc. (We got stuck at the "Ansible > Digital Ocean works fine for us" stage. Docker was too new at the time and never really had a reason to re-examine things) But just this past week after a ton of pain previously trying to get a working Geodjango stack running on my Mac locally for experimentation I discovered a docker-compose script that gave me everything I wanted in about 5 minutes. A sample app. Django, nginx, redis, PostGIS and all the complex dependencies - all working together. I literally typed "docker-compose up" and I had a working web GIS-enabled web app running on localhost. And the fact that the deployment to staging is hopefully just as simple makes me smile. Of course - production is a different story as I then have to worry that all the magic pieces aren't full of backdoors and security holes but I guess that's a job that needs doing anyway.
- 0x138d5 7y ago
  There are some things that have to be made clear to the team when moving to containers (don't use joerandom/alpine as a baseimage, build via digest instead of tag etc) but the speed at which you can onboard people is amazing. I couldn't imagine deploying a Python application without Docker nowadays.
  - baq 7y ago
    i've been doing python for 15+ years now and docker is one of the best things to happen to it, possibly it's saving grace in server side software. deployment and dev env setup used to be brutal.
  - jwbwater 7y ago
    How is this better than ansible, salt, chef, puppet, etc.? Once it's automated it's automated, no?
    - 0x138d5 7y ago
      I wouldn't use the term "better" - I still rely heavily on puppet and ansible. But the containerization aspect gives the option to pull up and destroy services without worrying about polluting the host you're in.
crucialfelix 7y ago
Good news! Everybody wants to just write a docker compose file and use that in production. Deploy on ECS or k8s or heroku, it would be awesome. That's the simplicity most people want for small projects.
lxe 7y ago
Just diving into Docker, and I'm already a bit confused... There's a docker stack (or swarm) that allows me to define containers (services?) and their networking... but then there's also docker compose, which does the same thing? Even the yaml config specs are similar.
- Eikon 7y ago
  Swarm is an orchestrator while compose is designed to describe services to be run on a single machine. Basically, you may want to use swarm (or k8s or anything else) in order to orchestrate your containers across a multi-machine cluster and compose when defining a set of services running on a single host.
  - hinkley 7y ago
    Does compose really have to be single machine? Yes, it focuses a lot on shared resources, but the negative space defined by where resources are not shared certainly informs any human-defined networking topology. I'm not so sure a tool couldn't make a pretty good stab at doing the same thing.
- peterwwillis 7y ago
  Docker is a tool that includes a daemon that runs in the background and a client app that communicates with the daemon. The daemon runs a packaged-up program and its dependencies ("image") as basically a chroot environment using an overlay filesystem and isolated network, in a custom cgroup and namespace (basically, process space/virtual filesystem paths different from your normal user process space/filesystem). Swarm is a (sadly) soon-to-be-abandoned way to spin up a cluster of multiple hosts with Docker daemons; don't waste your time on it. The comparable solutions are AWS ECS or [to a lesser extent] HashiCorp Nomad. Docker Compose is a console app that takes a config file and starts multiple Docker containers, creates volumes, and configures Docker networks - on a single Docker instance, so it's only really useful for testing stuff locally that requires multiple containers running at once.
  - cholmon 7y ago
mfer 7y ago
This is pretty exiting for 2 reasons... 1. Writing all the YAML to deploy an application into Kubernetes is a lot of work. If one does everything fully filled out it's pretty normal to have over a thousand lines of YAML... or much more. With lots of repeat yourself. Tools built from a simpler spec that can "compile" to that would be so great. Like writing in a higher level language instead of machine code or assembly. 2. There are a variety of platforms now. I first thought of Kubernetes vs Nomad. But, Kubernetes using Istio is a different platform from Kubernetes where applications use Ingress. There is more diversity showing up. Being able to define an application in one spec and "compile" for different situations would be useful. Just my 2 cents. As a Kubernetes user I'm happy to see the higher level tool bar moving. It's desperately needed.
- znpy 7y ago
  > But, Kubernetes using Istio is a different platform from Kubernetes where applications use Ingress This. After almost a year working with kubernetes I came to the same conclusion. I am/was studying for the kubernetes certification (cka) but in some way it's useless. I mean, it would prove basic proficiency, but the whole ecosystem is so varied that wouldn't necessarily mean actual and practical proficiency. It would probably help passing through vogon recruiters but a technical interviewer that knows his way around kubernetes wouldn't necessarily buy that.
- strangecasts 7y ago
  > Tools built from a simpler spec that can "compile" to that would be so great. Getting into Kubernetes, having Kompose (https://kompose.io/ https://kompose.io/) generate Kubernetes specifications (and Helm charts) from Docker Compose files was a godsend.
pmichaud 7y ago
I'm not a devops person at all, but this pile of abstractions seems like it's going the wrong way. Like, this seems almost like a parody? You have this problem where you want to set up infrastructure in an abstract way, so you come up with a way to write some infrastructure requirements and now you have a program that does all the actions necessary to take your setup from wherever it is, to what you specified. Then you have multiple of those systems, and it gets confusing. So you come up with a generic way to specify the abstractions that can be sort of compiled into any of the flavors, so that you can set up infrastructure using any of the sub-abstractions available, instead of only a specific one. This has the smell of a FactoryFactoryFactory sort of thing. Like, if this whole cottage industry were my architecture project I would be saying to myself "This is too much, a rickety, foot-gun-laden tower, destined to crumble. I've obviously chosen a poor abstraction, lemme back way the hell up." Am I just totally wrong here? I feel like I'm taking crazy pills.
- radus 7y ago
  Maybe you have a point but from my perspective you're overthinking it. Lots of developers use Docker Compose and will continue to do so because it's a great product. The news here is that there is now an open specification that others can contribute to.
- carlisle_ 7y ago
  I think you generally have the right sentiment, but some of these solutions are born out of necessity. Using containers at scale is really hard when you're trying to solve for optimal resource usage and resilience. These abstractions may seem crazy but other solutions are probably much crazier. The other part of this problem that isn't applicable generally to other things, is this is a fairly unique problem to cgroups. By design cgroups don't have a singular implementation or solution, so there are a lot of "competing" solutions. I think some amount of unix philosphy is at play here, to do one thing (or really, fewer) and that's why it seems insane how many systems you have to hook up together to make the thing work the way it should. I'm fairly devops centric, but I'd still take this with a grain of salt b/c containers aren't my area of expertise.
- jayd16 7y ago
wwarner 7y ago
To read the spec: https://github.com/compose-spec/compose-spec/blob/master/spec.md https://github.com/compose-spec/compose-spec/blob/master/spe...
oehpr 7y ago
ok, so... what? The notion here is to take the compose file, which is basically saying "this is what I want the state of docker to look like", and turn it into a generic application specification that multiple orchestration engines can contribute to and use... And... How would this work? Are we gonna have this smorgasbord of definitions in compose now with a subset of each working on specific orchestration systems, or something? Because that would be completely useless. So unless there genuinely is a commitment to being able to have each orchestration engine just unreservedly guaranteed take any definition possible in docker-compose and deploy it, then there is no value to be gained here. None. Without that guarantee, then for every system you attempt to deploy a compose file to, you must DEEP DIVE into its internals to understand its limitations and costs and tradeoffs. This gigantic massive timesink of a process is something I have always had to do. And I always discover all kinds of things that would have absolutely blocked practical deployments. This is the real thing that's bothering me, not that k8s and azure and docker all can sorta read some subset of a configuration. I don't care at all about that. It's not useful.
- hinkley 7y ago
  I'm not saying this will happen, but one way I might like to see this play out is a bit like the following. I have a bunch of apps. Some of them talk to each other, some do not. In general, if I deploy 5 copies of something, you should assume I mean "never on the same box" by default. (I'm looking at you, Kubernetes. WITAF.) You could, as a vendor, do a lot of sophisticated work on the spanning tree to reduce the amount of network traffic versus loopback traffic. You might try to spread out unshared volumes across as many motherboards as possible. You could differentiate on how to most efficiently and/or stably migrate from previous config to new config, or vice versa. You could do a bunch of knapsack work, including (and this is a pet peeve of mine) pre-scaling for cyclical traffic patterns. If you've ever looked at the Nest thermostat, one of several defining features was that it figures out the thermal lag in your heating and cooling system and it can make the house be the correct temperature at a particular time, rather than waiting until the appointed time to do anything. If a hockey puck on my wall can do shit like this then why doesn't every cloud provider do this day 1? Tack onto this some capacity planning infographics, and a system to schedule bulk, low priority operations around resource contention, and I could probably help get you a meeting with my last boss, my current boss, and probably at least the one after that.
rgovostes 7y ago
Tangentially, it is a little irksome how the docker-compose tool is released completely independently from Docker itself. There are binaries published on GitHub for amd64, but if you want it on an ARM platform, you need to install with pip. But pip can't install system packages, so you also need to know the incantation to install those (python3 python3-pip libffi-dev libssl-dev). I'm sure there's a historical reason for it not being `docker compose ...`, but it sounds like it's a good time to promote it.
- bdcravens 7y ago
  It was an external project acquired by Docker (originally called "fig") https://www.docker.com/blog/welcoming-the-orchard-and-fig-team/ https://www.docker.com/blog/welcoming-the-orchard-and-fig-te... Of course that was six years ago, so it is a bit surprising they haven't done a deeper integration since then.
  - fapjacks 7y ago
    Remember the outcry when Swarm was shipped as part of the core Docker binary? I wouldn't be surprised if this caused some kind of cooldown with respect to the binary assimilating features.
- justincormack 7y ago
  Hi, from Docker here, yes that should have been done before, but the previous work was incomplete, we have started working on it again.
  - ollien 7y ago
    How will this differ from `docker stack`? Just that `docker stack` only reads old compose files?
- sdan 7y ago
  Not sure what you're saying, I can run ARM and regular AMD platform containers via docker compose without problems (other than I have to build ARM using buildx).
nicksantos 7y ago
Docker Compose has always been an awesome file format, built on top of an idiosyncratic orchestration engine that doesn't interoperate well with anything else. Writing a spec is a good first step! If work is underway to make this a shared multi-service dev environment that works across Kubernetes, ECS, etc, that would be even better. When I read the spec, there are ominous words about partial implementations. That makes me nervous. But I'm cautiously optimistic.
- justincormack 7y ago
  There are partial implementations because some of the things in the existing spec are just not a good match for ECS or Kubernetes, for example. The aim is to fix those things in the spec. There are conformance tests so we can hopefully start checking implementations soon.
  - nicksantos 7y ago
    Yay for conformance tests! What's the strategy for handling things that don't make sense for other platforms? Do you expect that they'll be removed from the spec eventually, or fit in as some kind of platform-specific extension mechanism?
    - justincormack 7y ago
      We are expecting to have to have some platform specific extension mechanism for some things eventually, but yes I think some things can be removed too.
tln 7y ago
My two wishes are: 1. Bring back extends 2. Specify services that are only intended to be run Maybe #2 is possible now and I just don't know it? I wrote a wrapper at previous job around docker-compose that made it easy to define aliases within the compose file for complete run commands... Eg to run migrate in your rails container.
stevebmark 7y ago
This is great. Docker sometimes feels like a tangle of hacks, for example docker-compose build and docker build diverge and don't use the same layers. More standardization is better
nixpulvis 7y ago
Maybe they'll address https://github.com/docker/compose/issues/745 https://github.com/docker/compose/issues/745 now...
markl42 7y ago
Nice! I wonder if there will be a mapping between compose and the cli maintained as part of this? (Disclaimer, I maintain https://composerize.com/ https://composerize.com/, hoping this helps us track parity)
AskTheRightQs 7y ago
This is great. I might use it in my web app http://www.trackmycouriers.com http://www.trackmycouriers.com

251 points by alexellisuk 7y ago

Previously Compose did not have a specification, only the implementation, and it was tied to Docker's products. The aim here is to open it up, and let the specification evolve in new directions, such as better supporting deployment on Kubernetes, as an example. It is supported with some other implementations but we want to bring these closer together. Compose has a lot of users, and hundreds of thousands of Compose files on GitHub, so we wanted to open up development as much as possible. (I work at Docker)

deleted 7y ago
[deleted]
liveoneggs 7y ago
Does this spec carry forward the v2 stuff into v3 that would otherwise require --compatibility to function without swarm?
- justincormack 7y ago
  So far we have not made changes to the spec, just carried it forward but we definitely want to start cleaning up the v2 v3 mess, and make a single spec work for all platforms. See https://github.com/compose-spec/compose-spec/issues/12 https://github.com/compose-spec/compose-spec/issues/12 and also https://github.com/compose-spec/compose-spec/issues/13 https://github.com/compose-spec/compose-spec/issues/13 for making it feature not version number driven.
  - znpy 7y ago
    <insert python joke here>
  - liveoneggs 7y ago
    I would be surprised if there is the political will at docker to do this when it wasn't a priority for compose itself
    - fapjacks 7y ago
      Docker (Inc) has new priorities and different influences. I'm inclined to feel some optimism that this shifting focus will have beneficial repercussions in places like this.
- geerlingguy 7y ago
  This. My current usage is a muddy mess of v2 and v3 specific features an some require the compatibility flag while others don't.
e12e 7y ago
> better supporting deployment Is there any "supported deployment" for docker? I had the impression that swarm has been sidelined, and docker-compose was discouraged. Is there a path to deployment on kubernetes that uses only compose? (I have a hard time seeing how that'll work with ingress, and "compose" well with other kubernetes deployments?) We have some on-prem stuff that would work on single-node swarm - but I've felt that compose has been in a bit of limbo this past year or so... Is some kind of production style deployment comming back?
- hannibalhorn 7y ago
  "Compose on Kubernetes"[1] seems to be what they're advocating at the moment, and it's included in Docker Desktop. It's probably an OK first step for simpler deployments, though I suspect in order to do any troubleshooting, one would have to learn k8s anyway. 1. https://github.com/docker/compose-on-kubernetes https://github.com/docker/compose-on-kubernetes
  - mfer 7y ago
    To debug an application running on linux one does not need to learn machine code or assembly. There are tools that make it all possible. Higher level abstractions can work and need to start somewhere. It really depends what tools are developed.
    - e12e 7y ago
      Have you tried moving from on-premise docker-compose to managed kubernetes in the cloud? If you've been diligent in striving for stateless containers that will help you - but pretty much everything thing else in your compose file is useless. Docker-compose gets you the "pod"-level in k8s, but doesn't really help with ingress/services etc. So no one can reach your app, certainly not via a load balancer that terminate ssl for you. This is no longer a game of "beating the nginx container until it gives up and does what you need it to".
      - mfer 7y ago
amerine 7y ago
Hi Justin, I work at Heroku for Salesforce. I'm unsure if you've worked with anyone here in the past (like on our older app.yml, heroku.yml and constellations, etc stuff). But I just wanted to say thanks for establishing a standard for this and helping the rest of us in this space continue to evolve.
mcny 7y ago
Hi, I would like to use docker-compose without using docker. Specifically, I’d like to run devil box (php development environment) on fedora using pod man and docker-compose. Is that possible now?
girvo 7y ago
I'm super excited for this. Back when Docker Compose was still called Fig, I got annoyed at how it handled volume-only containers for databases, and wrote Dup [1]. Nowadays Compose is way WAY more useful, but there's a number of projects that still rely on it -- the spec means I can write a converter for the config files (from Dup to Compose) and be sure that it will work :) [1] https://github.com/girvo/dup https://github.com/girvo/dup

L_226 7y ago

Sweet, looks good. I reckon docker and docker-compose have been probably some of the greatest enablers for my development process, and how I think about desiging software systems.

daveguy 7y ago
Agreed. I was worried docker-compose might be abandoned for swarm-only at the kubernetes level. Compose is a much easier level to develop and work with. Doesn't do all the balancing and HA, but for most projects you don't need that, just good organization of the pieces.
- kalev 7y ago
  This is exactly why we still use compose as well. Everybody is talking about kubernetes, but I honestly don’t see the point in adding a huge dependency only for balancing and restarting failed containers.. I just wish there was some SaaS platform which allows you to upload a docker-compose file and run the containers for you. That way, you don’t have to touch the underlying OS.
  - larrybud 7y ago
    Azure App service lets you do this. https://docs.microsoft.com/en-us/azure/app-service/containers/quickstart-multi-container https://docs.microsoft.com/en-us/azure/app-service/container...
  - tilolebo 7y ago
    The AWS ECS cli has this kind of feature (I never used it) : https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-compose.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... v2 even allows to create the underlying networking resources (VPC), provision an application load balancer, etc.
- jswny 7y ago
  Aren't Compose and Swarm/Stack very similar, but Stack has support for distribution between multiple nodes? I know their actual configurations aren't equivalent, but they are similar (Stack supports config options for distribution, etc.).

clSTophEjUdRanu 7y ago

This is cool- hopefully this allows standardized podman-compose.

jabl 7y ago
There is https://github.com/abalage/pods-compose https://github.com/abalage/pods-compose

earthboundkid 7y ago

This is good. My main annoyance with Docker in the past was that I would make a decent Compose file for local dev but then have to rewrite everything in a proprietary YAML format for use in CI and prod.

peterwwillis 7y ago
You'll still be doing that with this new spec. It's still just a "how would you like me to run multiple containers" spec, it's not defining the relationships of arbitrary infrastructure and services on arbitrary networks/providers. What's going to happen is, you (or the dev) will fill this thing out to the best of your ability. Then someone's going to run it in one env; it won't work, but Ops will modify it to make it work. So then you'll commit that as your one Compose file. Then someone will try to run it in a different env, and it won't work; they'll fix it to make it run. Now you have two Compose files to maintain, with 3 different use cases. So to make it easier to manage, you go back to one for dev, and one for each env. All this because the different components could not be defined independently and loosely coupled, because the spec (and tools) don't support that. I like Compose for what it's designed for: let a dev mock something up locally. But it's going to become a pain in any environment that scales and will need a different solution.

crb 7y ago

> Docker is working with Amazon Web Services (AWS), Microsoft and others in the open source community Google?

move-on-by 7y ago

This is great news. Docker really screwed up v2->v3 and have continually suppressed Compose’s potential because it wasn’t inline with swarm. Looking forward to Compose getting some new life put into it.

cpuguy83 7y ago
Suppressed is not quite the right word. Too many inputs (from the community) and not enough help managing and maintaining, and curating these inputs is really what this is. That said, I don't personally believe there exists a single specification that can work (optimally) for both local-dev and production deployments. Disclaimer: ex-Docker engineer, still an active maintainer on the main docker project.
hinkley 7y ago
You may recall that compose used to be called fig, and they merged into Docker. I wouldn't be surprised if there was some Us vs Them politics going on there.

andybak 7y ago

I've been mostly avoiding Docker, Docker-compose etc. (We got stuck at the "Ansible > Digital Ocean works fine for us" stage. Docker was too new at the time and never really had a reason to re-examine things) But just this past week after a ton of pain previously trying to get a working Geodjango stack running on my Mac locally for experimentation I discovered a docker-compose script that gave me everything I wanted in about 5 minutes. A sample app. Django, nginx, redis, PostGIS and all the complex dependencies - all working together. I literally typed "docker-compose up" and I had a working web GIS-enabled web app running on localhost. And the fact that the deployment to staging is hopefully just as simple makes me smile. Of course - production is a different story as I then have to worry that all the magic pieces aren't full of backdoors and security holes but I guess that's a job that needs doing anyway.

0x138d5 7y ago
There are some things that have to be made clear to the team when moving to containers (don't use joerandom/alpine as a baseimage, build via digest instead of tag etc) but the speed at which you can onboard people is amazing. I couldn't imagine deploying a Python application without Docker nowadays.
- baq 7y ago
  i've been doing python for 15+ years now and docker is one of the best things to happen to it, possibly it's saving grace in server side software. deployment and dev env setup used to be brutal.
- jwbwater 7y ago
  How is this better than ansible, salt, chef, puppet, etc.? Once it's automated it's automated, no?
  - 0x138d5 7y ago
    I wouldn't use the term "better" - I still rely heavily on puppet and ansible. But the containerization aspect gives the option to pull up and destroy services without worrying about polluting the host you're in.

crucialfelix 7y ago

Good news! Everybody wants to just write a docker compose file and use that in production. Deploy on ECS or k8s or heroku, it would be awesome. That's the simplicity most people want for small projects.

lxe 7y ago

Just diving into Docker, and I'm already a bit confused... There's a docker stack (or swarm) that allows me to define containers (services?) and their networking... but then there's also docker compose, which does the same thing? Even the yaml config specs are similar.

Eikon 7y ago
Swarm is an orchestrator while compose is designed to describe services to be run on a single machine. Basically, you may want to use swarm (or k8s or anything else) in order to orchestrate your containers across a multi-machine cluster and compose when defining a set of services running on a single host.
- hinkley 7y ago
  Does compose really have to be single machine? Yes, it focuses a lot on shared resources, but the negative space defined by where resources are not shared certainly informs any human-defined networking topology. I'm not so sure a tool couldn't make a pretty good stab at doing the same thing.
peterwwillis 7y ago
Docker is a tool that includes a daemon that runs in the background and a client app that communicates with the daemon. The daemon runs a packaged-up program and its dependencies ("image") as basically a chroot environment using an overlay filesystem and isolated network, in a custom cgroup and namespace (basically, process space/virtual filesystem paths different from your normal user process space/filesystem). Swarm is a (sadly) soon-to-be-abandoned way to spin up a cluster of multiple hosts with Docker daemons; don't waste your time on it. The comparable solutions are AWS ECS or [to a lesser extent] HashiCorp Nomad. Docker Compose is a console app that takes a config file and starts multiple Docker containers, creates volumes, and configures Docker networks - on a single Docker instance, so it's only really useful for testing stuff locally that requires multiple containers running at once.
- cholmon 7y ago

mfer 7y ago

This is pretty exiting for 2 reasons... 1. Writing all the YAML to deploy an application into Kubernetes is a lot of work. If one does everything fully filled out it's pretty normal to have over a thousand lines of YAML... or much more. With lots of repeat yourself. Tools built from a simpler spec that can "compile" to that would be so great. Like writing in a higher level language instead of machine code or assembly. 2. There are a variety of platforms now. I first thought of Kubernetes vs Nomad. But, Kubernetes using Istio is a different platform from Kubernetes where applications use Ingress. There is more diversity showing up. Being able to define an application in one spec and "compile" for different situations would be useful. Just my 2 cents. As a Kubernetes user I'm happy to see the higher level tool bar moving. It's desperately needed.

znpy 7y ago
> But, Kubernetes using Istio is a different platform from Kubernetes where applications use Ingress This. After almost a year working with kubernetes I came to the same conclusion. I am/was studying for the kubernetes certification (cka) but in some way it's useless. I mean, it would prove basic proficiency, but the whole ecosystem is so varied that wouldn't necessarily mean actual and practical proficiency. It would probably help passing through vogon recruiters but a technical interviewer that knows his way around kubernetes wouldn't necessarily buy that.
strangecasts 7y ago
> Tools built from a simpler spec that can "compile" to that would be so great. Getting into Kubernetes, having Kompose (https://kompose.io/ https://kompose.io/) generate Kubernetes specifications (and Helm charts) from Docker Compose files was a godsend.

pmichaud 7y ago

I'm not a devops person at all, but this pile of abstractions seems like it's going the wrong way. Like, this seems almost like a parody? You have this problem where you want to set up infrastructure in an abstract way, so you come up with a way to write some infrastructure requirements and now you have a program that does all the actions necessary to take your setup from wherever it is, to what you specified. Then you have multiple of those systems, and it gets confusing. So you come up with a generic way to specify the abstractions that can be sort of compiled into any of the flavors, so that you can set up infrastructure using any of the sub-abstractions available, instead of only a specific one. This has the smell of a FactoryFactoryFactory sort of thing. Like, if this whole cottage industry were my architecture project I would be saying to myself "This is too much, a rickety, foot-gun-laden tower, destined to crumble. I've obviously chosen a poor abstraction, lemme back way the hell up." Am I just totally wrong here? I feel like I'm taking crazy pills.

radus 7y ago
Maybe you have a point but from my perspective you're overthinking it. Lots of developers use Docker Compose and will continue to do so because it's a great product. The news here is that there is now an open specification that others can contribute to.
carlisle_ 7y ago
I think you generally have the right sentiment, but some of these solutions are born out of necessity. Using containers at scale is really hard when you're trying to solve for optimal resource usage and resilience. These abstractions may seem crazy but other solutions are probably much crazier. The other part of this problem that isn't applicable generally to other things, is this is a fairly unique problem to cgroups. By design cgroups don't have a singular implementation or solution, so there are a lot of "competing" solutions. I think some amount of unix philosphy is at play here, to do one thing (or really, fewer) and that's why it seems insane how many systems you have to hook up together to make the thing work the way it should. I'm fairly devops centric, but I'd still take this with a grain of salt b/c containers aren't my area of expertise.
jayd16 7y ago

wwarner 7y ago

To read the spec: https://github.com/compose-spec/compose-spec/blob/master/spec.md https://github.com/compose-spec/compose-spec/blob/master/spe...

oehpr 7y ago

ok, so... what? The notion here is to take the compose file, which is basically saying "this is what I want the state of docker to look like", and turn it into a generic application specification that multiple orchestration engines can contribute to and use... And... How would this work? Are we gonna have this smorgasbord of definitions in compose now with a subset of each working on specific orchestration systems, or something? Because that would be completely useless. So unless there genuinely is a commitment to being able to have each orchestration engine just unreservedly guaranteed take any definition possible in docker-compose and deploy it, then there is no value to be gained here. None. Without that guarantee, then for every system you attempt to deploy a compose file to, you must DEEP DIVE into its internals to understand its limitations and costs and tradeoffs. This gigantic massive timesink of a process is something I have always had to do. And I always discover all kinds of things that would have absolutely blocked practical deployments. This is the real thing that's bothering me, not that k8s and azure and docker all can sorta read some subset of a configuration. I don't care at all about that. It's not useful.

hinkley 7y ago
I'm not saying this will happen, but one way I might like to see this play out is a bit like the following. I have a bunch of apps. Some of them talk to each other, some do not. In general, if I deploy 5 copies of something, you should assume I mean "never on the same box" by default. (I'm looking at you, Kubernetes. WITAF.) You could, as a vendor, do a lot of sophisticated work on the spanning tree to reduce the amount of network traffic versus loopback traffic. You might try to spread out unshared volumes across as many motherboards as possible. You could differentiate on how to most efficiently and/or stably migrate from previous config to new config, or vice versa. You could do a bunch of knapsack work, including (and this is a pet peeve of mine) pre-scaling for cyclical traffic patterns. If you've ever looked at the Nest thermostat, one of several defining features was that it figures out the thermal lag in your heating and cooling system and it can make the house be the correct temperature at a particular time, rather than waiting until the appointed time to do anything. If a hockey puck on my wall can do shit like this then why doesn't every cloud provider do this day 1? Tack onto this some capacity planning infographics, and a system to schedule bulk, low priority operations around resource contention, and I could probably help get you a meeting with my last boss, my current boss, and probably at least the one after that.

rgovostes 7y ago

Tangentially, it is a little irksome how the docker-compose tool is released completely independently from Docker itself. There are binaries published on GitHub for amd64, but if you want it on an ARM platform, you need to install with pip. But pip can't install system packages, so you also need to know the incantation to install those (python3 python3-pip libffi-dev libssl-dev). I'm sure there's a historical reason for it not being `docker compose ...`, but it sounds like it's a good time to promote it.

bdcravens 7y ago
It was an external project acquired by Docker (originally called "fig") https://www.docker.com/blog/welcoming-the-orchard-and-fig-team/ https://www.docker.com/blog/welcoming-the-orchard-and-fig-te... Of course that was six years ago, so it is a bit surprising they haven't done a deeper integration since then.
- fapjacks 7y ago
  Remember the outcry when Swarm was shipped as part of the core Docker binary? I wouldn't be surprised if this caused some kind of cooldown with respect to the binary assimilating features.
justincormack 7y ago
Hi, from Docker here, yes that should have been done before, but the previous work was incomplete, we have started working on it again.
- ollien 7y ago
  How will this differ from `docker stack`? Just that `docker stack` only reads old compose files?
sdan 7y ago
Not sure what you're saying, I can run ARM and regular AMD platform containers via docker compose without problems (other than I have to build ARM using buildx).

nicksantos 7y ago

Docker Compose has always been an awesome file format, built on top of an idiosyncratic orchestration engine that doesn't interoperate well with anything else. Writing a spec is a good first step! If work is underway to make this a shared multi-service dev environment that works across Kubernetes, ECS, etc, that would be even better. When I read the spec, there are ominous words about partial implementations. That makes me nervous. But I'm cautiously optimistic.

justincormack 7y ago
There are partial implementations because some of the things in the existing spec are just not a good match for ECS or Kubernetes, for example. The aim is to fix those things in the spec. There are conformance tests so we can hopefully start checking implementations soon.
- nicksantos 7y ago
  Yay for conformance tests! What's the strategy for handling things that don't make sense for other platforms? Do you expect that they'll be removed from the spec eventually, or fit in as some kind of platform-specific extension mechanism?
  - justincormack 7y ago
    We are expecting to have to have some platform specific extension mechanism for some things eventually, but yes I think some things can be removed too.

tln 7y ago

My two wishes are: 1. Bring back extends 2. Specify services that are only intended to be run Maybe #2 is possible now and I just don't know it? I wrote a wrapper at previous job around docker-compose that made it easy to define aliases within the compose file for complete run commands... Eg to run migrate in your rails container.

stevebmark 7y ago

This is great. Docker sometimes feels like a tangle of hacks, for example docker-compose build and docker build diverge and don't use the same layers. More standardization is better

nixpulvis 7y ago

Maybe they'll address https://github.com/docker/compose/issues/745 https://github.com/docker/compose/issues/745 now...

markl42 7y ago

Nice! I wonder if there will be a mapping between compose and the cli maintained as part of this? (Disclaimer, I maintain https://composerize.com/ https://composerize.com/, hoping this helps us track parity)

AskTheRightQs 7y ago

This is great. I might use it in my web app http://www.trackmycouriers.com http://www.trackmycouriers.com