3 ms·

Docker founder here. I keep reading articles stating that "the Docker API changes with every release", but the assertion is never backed by any specific exampl

by shykes 10y ago

Docker founder here. I keep reading articles stating that "the Docker API changes with every release", but the assertion is never backed by any specific examples. Has anyone here encountered an actual breaking change? If so, I would appreciate you sharing the specifics so we can fix it. Docker is by no means perfect: - I remember that in 1.10 the switch to content-addressed registries meant that older clients could not pull by digest (but all other commands, and even non-pinned pull, still worked). This was not an accidental breaking change: it was the result of a difficult tradeoff. In the end we decided that the benefits of the new content-addressed model outweighed the inconvenience. To guide our decision we used data from Docker Hub to assess how many clients would be affected. I forget the exact number but it was a very small minority. - And in 1.12 we got bitten by a change in how Go 1.6 processes HTTP headers (it became more strict and thus rejected headers from older clients). That was quite simply a screwup on our part. So we've had our share of screw-ups, no question. But lately I've been reading the "breaks at every release" meme more and more. Based on the evidence I have, it seems incredibly disconnected from reality. What am I missing?

OhSoHumble 10y ago
Oh, hai there. I haven't gone 'whole hog' on Docker yet and subsequently tried to push for rolling it out in production for my team. I don't know how incorrect I am, but it seems like the Docker project is more interested in features rather than stability. Except, like you said, I don't have any specific examples to back that up. Maybe it's just community hive mind thought bleeding onto my decision making abilities? The most concrete failing I've experienced was the deprecation of the boot2docker project in favor of docker machine. When I started using docker machine, stuff just... broke... and it left a bad taste in my mouth. In fact, using docker on either a mac or windows is just awful for me. My anecdotal experience is that it breaks in small ways that then that can really disrupt my productivity. I'm in the process of picking up docker again and it led me to install Arch on my work laptop just so I can have native containers. Right now my use case is using containers to test Chef cookbooks and it works... alright. Not as well as virtual machines, but alright. How docker handles init bothers me a little bit but it's nothing the maintainers of the testing suites for Chef couldn't handle through yaml configuration options. Oh, I guess while you're here... one thing I've been wrestling over is the usage of dockerfiles. I have no problem with containers but to me, it looks like using a configuration management system to bake container images seems to grant so much more flexibility and testing capabilities that I can't really see dockerfiles as anything other than an intermediate step. Is this a valid viewpoint?
- gerdesj 10y ago
  "but it seems like the Docker project is more interested in features rather than stability." I think that is a fair dig but I think they are still exploring the limits of what it can do. They are following the path of release often and make changes and show your working really well - you get to see the code and can contribute yourself. The thing is that you have to take all that into account when you are using their product. You need to take an Engineering approach to "production". Use Docker in the way that works for you. You state that you ended up using Arch instead of Windows or Mac - so fine, you have constructed a tool chain that works for you and welcome aboard! You see, Windows and Mac also do odd things every now and then but you don't get to see the workings. You are clearly not daft - you use Arch. I run Arch (my wife's laptop), Gentoo (my laptop and a fair few mail proxies and Icinga instances and other things), a lot of Ubuntus and a slack handful of Kalis and a sprinkling of Debians, Centos and others. You need to use engineering principles when going to production. Also you have to look at the price you paid to use Docker - it's a purely social one but one that is extremely important and may not involve money but cost you a fortune anyway. You need to decide if you want to pay that cost. You are clearly someone who understands the open source methodology but I would suggest that you look a little deeper into the dev -> prod thing. There is no reason why dev = Docker => prod = Docker. That's something that only you can decide after careful testing. Cheers Jon
gerdesj 10y ago
Thanks for stopping by. I too would like to see something more than a "my toys broke - mwaaaah". I personally find Docker a phenomenal resource for development. For production however, you analyze your use case, the risks, the failure modes etc etc and then you decide whether Docker is a part of that. I'm by no means a massive Docker user but compared to many of the sub systems running on my laptop it has been very stable for me although I do run Gentoo. Actually my entire laptop is surprisingly stable TBH. I absolutely love the way I run a few short commands and a fully tooled up web app with all the supporting infrastructure simply appears. Funnily enough, only yesterday I had a look at this: https://nextcloud.com/collaboraonline/ https://nextcloud.com/collaboraonline/ which I found in an article somewhere (LWN?) Yep - love it (NC) so the next step will be to decide how to use it for my home production system (yes I have dev/production at home) For that I will use pretty much the same criteria that I will use for my office and finally my customer systems. In this case Docker for production probably makes sense but I need to analyze it a lot more. A CoreOS VM will probably be spun up very soon in my attic for stage 2 in home production. I really don't like to see rants along the lines of the other article, denigrating a product that a lot of people have worked hard on, that shows a lack of discipline and clarity of thought. I for one would like to thank you for your work and making it available for all. It has genuinely saved me time and effort and made my IT life a little bit better. To answer your final question: a thicker skin, mate 8) Cheers Jon
teabee89 10y ago
Maintainer here, After verifying, I have to rectify your second point: 1.12 DOES NOT BREAK older clients. To add more information, the issue was present for Docker clients with version 1.9 and older talking to THEN Docker master. Thankfully it was seen before 1.11 was released, and thus we decided to stay on Go 1.5 for Docker 1.11 for this reason. For 1.12, we added a workaround for this issue, allowing us to upgrade to Go 1.6. UPDATE: Docker 1.10 was built with Go 1.5 so this issue was not present with a Docker 1.9 client talking to a 1.10 daemon either.
web007 10y ago
You're missing the fact that docker clients complain about the API version if they're different from the server - regardless of actual compatibility problems. http://stackoverflow.com/questions/37617400/ http://stackoverflow.com/questions/37617400/ sums it up. There's a magic (afaik undocumented) env var DOCKER_APIVERSION you can set for compatibility, but nobody can find it. If you can't mix even minor versions then yes, it's a problem. I can't run X in prod and anything other than X in dev of I'm working with the same toolset.
- teabee89 10y ago
  The issue you are referring to happens only if the docker client version is newer than the docker daemon version. In the case where the client is older than the daemon, it has always worked. Thanks for bringing this issue up though, because we are currently working on a fix. Here is the PR: https://github.com/docker/docker/pull/27745 https://github.com/docker/docker/pull/27745 As for DOCKER_API_VERSION, it is possibly not well documented, but it is indeed present in the documentation: https://docs.docker.com/engine/reference/commandline/cli/ https://docs.docker.com/engine/reference/commandline/cli/ Hopefully this was helpful.
  - web007 10y ago
    The client will nearly always be newer, so that message will almost always show up. You'll update your laptop tools regularly, and only bump your infrastructure occasionally. How many versions has this been a problem? It's not thinking about things like this until (apparently) 11 days ago that give Docker haters such ammo. The UX of the entire ecosystem feels like an afterthought. Yes, it's been evolving rapidly, but that's no excuse for not having a good, unified user experience for what's there or what's coming next.
    - cpuguy83 10y ago
      That's a fair statement, especially with d4mac/d4win and auto-update. Posting a pull request and thinking about a problem are two very different things. The project has chosen to err on the side of safety (ie, user asked for some feature to be used but it is silently ignored since it's connecting to an older daemon) in this regard. -- this is not the only problem to solve
DrNemski 10y ago
Hi There, I created an account here so I could let you know I've clarified some points in my post to address your concerns.
cyphar 10y ago
> Has anyone here encountered an actual breaking change? The fact that if your local client is newer than the daemon, the client will fail to do anything is pretty annoying. You can hack around it, but that's not a solution to the problem. Also, while you shake off the 1.10 change as though it wasn't a huge deal, the migration tool that was written was not fit for purpose. First of all, it sometimes caused corruption after the migration (fixed after release but still an issue, and you couldn't practically un-bork the corruption). There were other issues with the tool, but the biggest issue was the there wasn't any sane transition period -- which made packaging very difficult. Because Docker couldn't handle restarting the daemon and keeping the containers around (and after restarting a post-1.10 daemon you'd have to wait for the full migration to finish before you could even interact with the daemon -- another issue with the transition), you'd have to publish a fake version bump of the Docker package to even alert users to the issue! A more sane way to handle it would've been how databases handle the issue. You lazily do the conversion (or you have your software be able to handle both formats so you can run your migration while still having your containers work). And yes, I'm aware that it'd be difficult to implement but the transition path implemented was really bad. I spent several weeks of my life (that I'll never get back) trying to make the migration path work. It didn't end up working because it wasn't designed to work nicely from the outset. So I ended up just telling users of our Docker packages about the issue and packaging the migration tool separately. Another issue that I've had recently is that Docker is making it increasingly difficult to use `runc` by another name. In openSUSE we package a system runC separately to Docker (and it lives in /usr/sbin/runc), but because of the fact that in 1.12 --runtime was intentionally broken (and containerd's defaults are no longer obeyed by Docker) now you can't effectively use --exec-opt native.cgroupdriver=systemd. Great. So now we're going to have to package docker-runc and docker-containerd separately. Don't get me started on the containerd split. I was not very happy the day that PR was merged, because it caused a bunch of headaches for us. It's also a bit amusing to note that containerd was tagged as "pre-alpha" until Docker bundled it (then it was considered stable). That all being said, Docker is a great project -- it's just that you have to admit that issues with Docker are not "incredibly disconnected from reality".

by shykes 10y ago

OhSoHumble 10y ago

Oh, hai there. I haven't gone 'whole hog' on Docker yet and subsequently tried to push for rolling it out in production for my team. I don't know how incorrect I am, but it seems like the Docker project is more interested in features rather than stability. Except, like you said, I don't have any specific examples to back that up. Maybe it's just community hive mind thought bleeding onto my decision making abilities? The most concrete failing I've experienced was the deprecation of the boot2docker project in favor of docker machine. When I started using docker machine, stuff just... broke... and it left a bad taste in my mouth. In fact, using docker on either a mac or windows is just awful for me. My anecdotal experience is that it breaks in small ways that then that can really disrupt my productivity. I'm in the process of picking up docker again and it led me to install Arch on my work laptop just so I can have native containers. Right now my use case is using containers to test Chef cookbooks and it works... alright. Not as well as virtual machines, but alright. How docker handles init bothers me a little bit but it's nothing the maintainers of the testing suites for Chef couldn't handle through yaml configuration options. Oh, I guess while you're here... one thing I've been wrestling over is the usage of dockerfiles. I have no problem with containers but to me, it looks like using a configuration management system to bake container images seems to grant so much more flexibility and testing capabilities that I can't really see dockerfiles as anything other than an intermediate step. Is this a valid viewpoint?

gerdesj 10y ago
"but it seems like the Docker project is more interested in features rather than stability." I think that is a fair dig but I think they are still exploring the limits of what it can do. They are following the path of release often and make changes and show your working really well - you get to see the code and can contribute yourself. The thing is that you have to take all that into account when you are using their product. You need to take an Engineering approach to "production". Use Docker in the way that works for you. You state that you ended up using Arch instead of Windows or Mac - so fine, you have constructed a tool chain that works for you and welcome aboard! You see, Windows and Mac also do odd things every now and then but you don't get to see the workings. You are clearly not daft - you use Arch. I run Arch (my wife's laptop), Gentoo (my laptop and a fair few mail proxies and Icinga instances and other things), a lot of Ubuntus and a slack handful of Kalis and a sprinkling of Debians, Centos and others. You need to use engineering principles when going to production. Also you have to look at the price you paid to use Docker - it's a purely social one but one that is extremely important and may not involve money but cost you a fortune anyway. You need to decide if you want to pay that cost. You are clearly someone who understands the open source methodology but I would suggest that you look a little deeper into the dev -> prod thing. There is no reason why dev = Docker => prod = Docker. That's something that only you can decide after careful testing. Cheers Jon

gerdesj 10y ago

Thanks for stopping by. I too would like to see something more than a "my toys broke - mwaaaah". I personally find Docker a phenomenal resource for development. For production however, you analyze your use case, the risks, the failure modes etc etc and then you decide whether Docker is a part of that. I'm by no means a massive Docker user but compared to many of the sub systems running on my laptop it has been very stable for me although I do run Gentoo. Actually my entire laptop is surprisingly stable TBH. I absolutely love the way I run a few short commands and a fully tooled up web app with all the supporting infrastructure simply appears. Funnily enough, only yesterday I had a look at this: https://nextcloud.com/collaboraonline/ https://nextcloud.com/collaboraonline/ which I found in an article somewhere (LWN?) Yep - love it (NC) so the next step will be to decide how to use it for my home production system (yes I have dev/production at home) For that I will use pretty much the same criteria that I will use for my office and finally my customer systems. In this case Docker for production probably makes sense but I need to analyze it a lot more. A CoreOS VM will probably be spun up very soon in my attic for stage 2 in home production. I really don't like to see rants along the lines of the other article, denigrating a product that a lot of people have worked hard on, that shows a lack of discipline and clarity of thought. I for one would like to thank you for your work and making it available for all. It has genuinely saved me time and effort and made my IT life a little bit better. To answer your final question: a thicker skin, mate 8) Cheers Jon

teabee89 10y ago

Maintainer here, After verifying, I have to rectify your second point: 1.12 DOES NOT BREAK older clients. To add more information, the issue was present for Docker clients with version 1.9 and older talking to THEN Docker master. Thankfully it was seen before 1.11 was released, and thus we decided to stay on Go 1.5 for Docker 1.11 for this reason. For 1.12, we added a workaround for this issue, allowing us to upgrade to Go 1.6. UPDATE: Docker 1.10 was built with Go 1.5 so this issue was not present with a Docker 1.9 client talking to a 1.10 daemon either.

web007 10y ago

You're missing the fact that docker clients complain about the API version if they're different from the server - regardless of actual compatibility problems. http://stackoverflow.com/questions/37617400/ http://stackoverflow.com/questions/37617400/ sums it up. There's a magic (afaik undocumented) env var DOCKER_APIVERSION you can set for compatibility, but nobody can find it. If you can't mix even minor versions then yes, it's a problem. I can't run X in prod and anything other than X in dev of I'm working with the same toolset.

teabee89 10y ago
The issue you are referring to happens only if the docker client version is newer than the docker daemon version. In the case where the client is older than the daemon, it has always worked. Thanks for bringing this issue up though, because we are currently working on a fix. Here is the PR: https://github.com/docker/docker/pull/27745 https://github.com/docker/docker/pull/27745 As for DOCKER_API_VERSION, it is possibly not well documented, but it is indeed present in the documentation: https://docs.docker.com/engine/reference/commandline/cli/ https://docs.docker.com/engine/reference/commandline/cli/ Hopefully this was helpful.
- web007 10y ago
  The client will nearly always be newer, so that message will almost always show up. You'll update your laptop tools regularly, and only bump your infrastructure occasionally. How many versions has this been a problem? It's not thinking about things like this until (apparently) 11 days ago that give Docker haters such ammo. The UX of the entire ecosystem feels like an afterthought. Yes, it's been evolving rapidly, but that's no excuse for not having a good, unified user experience for what's there or what's coming next.
  - cpuguy83 10y ago
    That's a fair statement, especially with d4mac/d4win and auto-update. Posting a pull request and thinking about a problem are two very different things. The project has chosen to err on the side of safety (ie, user asked for some feature to be used but it is silently ignored since it's connecting to an older daemon) in this regard. -- this is not the only problem to solve

DrNemski 10y ago

Hi There, I created an account here so I could let you know I've clarified some points in my post to address your concerns.

cyphar 10y ago

> Has anyone here encountered an actual breaking change? The fact that if your local client is newer than the daemon, the client will fail to do anything is pretty annoying. You can hack around it, but that's not a solution to the problem. Also, while you shake off the 1.10 change as though it wasn't a huge deal, the migration tool that was written was not fit for purpose. First of all, it sometimes caused corruption after the migration (fixed after release but still an issue, and you couldn't practically un-bork the corruption). There were other issues with the tool, but the biggest issue was the there wasn't any sane transition period -- which made packaging very difficult. Because Docker couldn't handle restarting the daemon and keeping the containers around (and after restarting a post-1.10 daemon you'd have to wait for the full migration to finish before you could even interact with the daemon -- another issue with the transition), you'd have to publish a fake version bump of the Docker package to even alert users to the issue! A more sane way to handle it would've been how databases handle the issue. You lazily do the conversion (or you have your software be able to handle both formats so you can run your migration while still having your containers work). And yes, I'm aware that it'd be difficult to implement but the transition path implemented was really bad. I spent several weeks of my life (that I'll never get back) trying to make the migration path work. It didn't end up working because it wasn't designed to work nicely from the outset. So I ended up just telling users of our Docker packages about the issue and packaging the migration tool separately. Another issue that I've had recently is that Docker is making it increasingly difficult to use `runc` by another name. In openSUSE we package a system runC separately to Docker (and it lives in /usr/sbin/runc), but because of the fact that in 1.12 --runtime was intentionally broken (and containerd's defaults are no longer obeyed by Docker) now you can't effectively use --exec-opt native.cgroupdriver=systemd. Great. So now we're going to have to package docker-runc and docker-containerd separately. Don't get me started on the containerd split. I was not very happy the day that PR was merged, because it caused a bunch of headaches for us. It's also a bit amusing to note that containerd was tagged as "pre-alpha" until Docker bundled it (then it was considered stable). That all being said, Docker is a great project -- it's just that you have to admit that issues with Docker are not "incredibly disconnected from reality".