7 ms·

Disclaimer: I work at AWS, but on a product which does not compete with Docker or its orchestration tools in any way shape or form. My opinions are my own. I w

Disclaimer: I work at AWS, but on a product which does not compete with Docker or its orchestration tools in any way shape or form. My opinions are my own. I wouldn't even limit this to just the swarm feature. We've been running Docker in production for a year, using it in dev environments a year before that, and we've had major problems nearly every release. We had to upgrade directly from Docker 1.7 to 1.11 because every release in between was too unstable or had glaring performance regressions. We ended up running a custom build and backporting features which were worth the risk. Speaking of 1.12, my heart sank when I saw the announcement. Native swarm adds a huge level of complexity to an already unstable piece of software. Dockercon this year was just a spectacle to shove these new tools down everyone's throats and really made it feel like they saw the container parts of Docker as "complete." One of the keynote slides literally read "No one cares about containers." I get the feeling we'll be running 1.11 for quite some time...

tinco 10y ago
To provide some weight the other way, we've been using Docker in production for about 3 years now, and have not had any big issues. Obviously, you guys probably have a bit more extreme use cases at AWS. Things that bug us are generally missing features, but those gradually get added in over the course of the years, though some get less love than others. For example for some reason it's still not possible to ADD something and change its permissions/ownership in one layer, resulting basically in a doubling of the size of such layers. I wouldn't go as far as saying it's in any kind of a 'sad' state though. It's a neat wrapper over some cool Linux kernel features, and it's been that way since before 1.0. I'm curious how you even get performance issues from Docker, what feature did cause performance issues for you?
- andrewguenther 10y ago
  Always fun to hear experiences from other production veterans. Glad to hear things are working well for you guys. Our use case involves rapid creation and destruction of containers. Granted, this use case was pretty unheard of when we first adopted Docker, but it is becoming much more common. Before Docker moved over to containerd, the docker daemon was riddled with locks which resulted in frequent dead-locking scenarios under load as well as poor performance. Thankfully, containerd now uses an event loop and is lock-free. This was a huge motivating factor for us to move forward to Docker 1.11. To me, the sad state has more to do with Docker the company pushing new features out as quickly as possible and leaving stabilization to contributors. There are some days where it really feels like Docker is open-source so that Docker Inc can get free QA. To most users things may not feel in a sad state, but it can really suck for contributors.
  - tinco 10y ago
    Alright that makes sense, most of our containers are long running, usually months. Only the containers that have our apps that are under active development will see multiple rollovers per day. Now that I think about it we did have one semi-serious bug in Docker, though that was also our own fault. Our containers would log a lot, and we hadn't configured our rsyslog very well so under some circumstances its buffers would fill up and the log writes would become blocking and be real slow. When this would happen some commands like `docker ps` would totally lock up, which messed with our (hand rolled) orchestration system. It wasn't until one of us noticed the logs would be minutes behind that we discovered killing rsyslog would make docker responsive again and thus found out what was happening. Since it didn't actually affect our service I didn't remember it as particularily bad, but I can imagine that if our service depended on having fast interactions with Docker that would have hurt bad. IIRC they did recognize the severity of the issue and quickly had a fix ready. I bet Docker Inc. has a tough mission, building out Docker services far enough to compete with the dozens of platforms that integrate Docker such as AWS or OpenStack so they can actually make money off the enterprise.
  - byroot 10y ago
    > the docker daemon was riddled with locks which resulted in frequent dead-locking scenarios under load as well as poor performance I second this. We use Docker in a similar scenario for a distributed CI. So we spawn between 70k and 90k containers every day. Up to very recently we were running 1.9 and got a staggering 9% of failures due to diverse Docker bugs. It's getting better though, since we upgraded to 1.12 a few days ago we're down to a more manageable 4%, but I'd still consider this very unreliable for an infrastructure tool. edit: my metrics were slightly flawed, we're down to 4% not 0.5%
    - andrewguenther 10y ago
      You were likely seeing the bug that kept us from deploying 1.9 which was related to corruption of the bit mask which managed IP address application. We saw failure rates very similar to yours with that issue.
    - nogox 10y ago
- piva00 10y ago
  I've been using Docker in production for the past 2 and a half years (in two different companies) and even with no extreme use cases we've had problems with: performance of volumes/devicemapper, random breaking bugs: daemon would restart without warning or errors in 1.4, randomly killing containers in 1.9, having to restart the daemon in 1.8 when it hung pulling images (consequently killing the containers in the process). I still like Docker and can see myself, team and company using it for a long time if nothing MUCH better show up (rkt is promising to take some of the complexity pain away but we are not diving into it yet) but I can't say I've not been bitten enough to completely avoid upgrading Docker if it isn't needed, we follow a rule to only upgrade to ".1" releases as most of our problems have been with ".0" ones.
  - SEJeff 10y ago
    My favorite was docker exec -it $container bash would cause a nil pointer deterrence in docker 1.6.0 and kill the docker daemon. We've seen gobs of bugs since, but that was the most wtf gnarly one
cyphar 10y ago
I'd recommend looking at using runC (which is the underlying runtime underneath Docker). Currently we're heading for a 1.0, and the Open Containers Initiative is working on specifications that will make container runtimes interoperable and eventually provide tooling that works with all OCI runtimes. If you have anything to contribute, I would hope that you can give us a hand. :D
- andrewguenther 10y ago
  I'm a huge fan of the work being done on runC and would love to give you guys a hand! You'll probably see me around soon :)
StreamBright 10y ago
Would rkt be a worth to try alternative?
- andrewguenther 10y ago
  I really haven't looked at rkt as much as I should, but we're more likely to invest in looking at lower level tools like runC moving forward.
  - StreamBright 10y ago
    Amazing thank you! I need to chose a containerization tech in the next month and I am pretty worried to go with Docker because I hear many stories about how it is not really production ready. Thanks for mentioning runC I will check it out.
    - colemickens 10y ago
      runC isn't an orchestration solution. It's a low-level component that can be (or is already) used by higher-level orchestration technologies.
      - StreamBright 10y ago
        Great, I do not need any orchestration solution at all. I need a container running thing that can encapsulate any software that we are developing (Java, C#, Node.JS, etc.). And now we are approaching the question what is my problem with Docker. I believe it is a misconception to compete with already existing tools like systemd. I especially do not want any mediocre orchestration solutions in my infrastructure that introduce big overhead and complexity that I do not need at all. One thing I learned along the way of managing large clusters (5K+ nodes) that Swarm like frameworks are extremely error prone. If you flip the problem and build a startup script the pulls down the container configuration from S3 for example and the container itself has code that attaches the instance to the right service (EKB, Haproxy, etc.) you can achieve the same without introducing services that sole purpose is to maintain a state that you do not need.
lacker 10y ago
If I were you I would add a disclaimer when criticizing Docker, mentioning that you work on AWS, since the products are competitive in some ways like EC2 Container Registry vs Docker Hub. It would be great for AWS if Docker simply focused on open source bug-fixing and let AWS provide the profitable services....
- andrewguenther 10y ago
  Added a disclaimer, however, the product I work on does not compete with Docker in any way. We actually rely on Docker quite heavily. No conspiracy here.
- paulddraper 10y ago
  "It'd be great if Docker wasn't profitable." I sort of agree, but that's not entirely realistic :)
  - lcarlson 10y ago
    Why the anti capitalistic sentiment? Is programmers want to get paid for our work, right?
    - brianwawok 10y ago
      No. Programmers want huge paychecks but everyone ELSE should be FOSS and code for us for free
      - cyphar 10y ago
        ahem Free software does not need to be gratis. There are several examples of companies which charge money for free software.
        softawre 10y ago
        ahem That's obviously not what is being discussed here. > It would be great for AWS if Docker simply focused on open source bug-fixing and let AWS provide the profitable services....
mtanski 10y ago
The same experiences we switched to using rkt, supervised by upstart (and now systemd). We have an "application" state template in our salt config and every docker update something would cause all of them to fail. Thankful the "application" state template abstracted running container enough were we switched from docker -> rkt under the covers without anybody noticing, except now we no longer fearing of container software updates.
- pescerosso 10y ago
  Hi mtanski, How did you replace docker with rkt? Do you have an howto that you can share?
  - flexd 10y ago
    I haven't replaced Docker with rkt on a big scale (or ran Docker on a big scale), but I recently changed over some Docker containers to rkt. First off, this and the rest of the rkt docs is a good starting point https://coreos.com/rkt/docs/latest/rkt-vs-other-projects.html https://coreos.com/rkt/docs/latest/rkt-vs-other-projects.htm... Second, rkt runs Docker images without modifications, so you can swap over really easily https://coreos.com/rkt/docs/latest/running-docker-images.html https://coreos.com/rkt/docs/latest/running-docker-images.htm... rkt uses acbuild (which is part of the application container specification, see https://github.com/appc/spec https://github.com/appc/spec) to build images, and I had a very tiny Docker image just running a single Go process. I just created a shell script that ran the required acbuild commands to get as similar image. A good place to get started is the getting started guide https://coreos.com/rkt/docs/latest/getting-started-guide.html https://coreos.com/rkt/docs/latest/getting-started-guide.htm... Docker runs as a daemon, and rkt doesn't (which is one of the benefits). I just start my rkt container using systemd, so I have a systemd file with 'ExecStart=/usr/bin/rkt run myimage:1.23.4', but you can start the containers with whatever you want. It's also possible to use rkt with Kubernetes, but I have not tried that yet. http://kubernetes.io/docs/getting-started-guides/rkt/ http://kubernetes.io/docs/getting-started-guides/rkt/
- mtanski 10y ago
  An example of changing behavior that broke us not to long ago: https://github.com/docker/distribution/issues/1662 https://github.com/docker/distribution/issues/1662 . By the time this happened we were already working on the transition, just more motivation.
kordless 10y ago
If you haven't heard of Giant Swarm, I encourage you to contact them. They have a scalable microservices provisioning solution that can use either Docker or Kubernetes. German company. Disclaimer: I worked for them last year. Holler if you need an intro.
cookiecaper 10y ago
We're in the process of converting our 100+ cloud nodes to Docker+k8s and I have a lot of the same reservations -- the space is very immature and the tooling has a lot of kinks to work out, not only functionally but also aesthetically. It's already been a nightmare and we're not even deployed to prod yet.
- justinsb 10y ago
  If "cloud" is AWS, you should join the kubernetes slack sig-aws channel. Lots of community people figuring out those kinks together.
- jacques_chester 10y ago
  It's obligatory for me to recommend Cloud Foundry here. We've already built the platform, there's no need to build and maintain your own. It just works. Really well, actually. Disclosure: I work for Pivotal, we donate the majority of engineering to Cloud Foundry.
RRRA 10y ago
Not to mention 1.11's restart timer never were reset to 0 even after the container ran well for more than 10 seconds. (ie: after a few restart, your container would be waiting hours to start!). This and I can attest to 1.12 problems listed in this article. Can't remember the specific with 1.10, but basically, nothing really ever works as promised which make people waste a lot of time trying to make something work when it can't and second, doesn't give much trust in the product's stability. I really wish they would collaborate a lot more and fragment their solutions in smaller module while keeping everything simple. I think they have a great product, but too much growing pain.

tinco 10y ago

To provide some weight the other way, we've been using Docker in production for about 3 years now, and have not had any big issues. Obviously, you guys probably have a bit more extreme use cases at AWS. Things that bug us are generally missing features, but those gradually get added in over the course of the years, though some get less love than others. For example for some reason it's still not possible to ADD something and change its permissions/ownership in one layer, resulting basically in a doubling of the size of such layers. I wouldn't go as far as saying it's in any kind of a 'sad' state though. It's a neat wrapper over some cool Linux kernel features, and it's been that way since before 1.0. I'm curious how you even get performance issues from Docker, what feature did cause performance issues for you?

andrewguenther 10y ago
Always fun to hear experiences from other production veterans. Glad to hear things are working well for you guys. Our use case involves rapid creation and destruction of containers. Granted, this use case was pretty unheard of when we first adopted Docker, but it is becoming much more common. Before Docker moved over to containerd, the docker daemon was riddled with locks which resulted in frequent dead-locking scenarios under load as well as poor performance. Thankfully, containerd now uses an event loop and is lock-free. This was a huge motivating factor for us to move forward to Docker 1.11. To me, the sad state has more to do with Docker the company pushing new features out as quickly as possible and leaving stabilization to contributors. There are some days where it really feels like Docker is open-source so that Docker Inc can get free QA. To most users things may not feel in a sad state, but it can really suck for contributors.
- tinco 10y ago
  Alright that makes sense, most of our containers are long running, usually months. Only the containers that have our apps that are under active development will see multiple rollovers per day. Now that I think about it we did have one semi-serious bug in Docker, though that was also our own fault. Our containers would log a lot, and we hadn't configured our rsyslog very well so under some circumstances its buffers would fill up and the log writes would become blocking and be real slow. When this would happen some commands like `docker ps` would totally lock up, which messed with our (hand rolled) orchestration system. It wasn't until one of us noticed the logs would be minutes behind that we discovered killing rsyslog would make docker responsive again and thus found out what was happening. Since it didn't actually affect our service I didn't remember it as particularily bad, but I can imagine that if our service depended on having fast interactions with Docker that would have hurt bad. IIRC they did recognize the severity of the issue and quickly had a fix ready. I bet Docker Inc. has a tough mission, building out Docker services far enough to compete with the dozens of platforms that integrate Docker such as AWS or OpenStack so they can actually make money off the enterprise.
- byroot 10y ago
  > the docker daemon was riddled with locks which resulted in frequent dead-locking scenarios under load as well as poor performance I second this. We use Docker in a similar scenario for a distributed CI. So we spawn between 70k and 90k containers every day. Up to very recently we were running 1.9 and got a staggering 9% of failures due to diverse Docker bugs. It's getting better though, since we upgraded to 1.12 a few days ago we're down to a more manageable 4%, but I'd still consider this very unreliable for an infrastructure tool. edit: my metrics were slightly flawed, we're down to 4% not 0.5%
  - andrewguenther 10y ago
    You were likely seeing the bug that kept us from deploying 1.9 which was related to corruption of the bit mask which managed IP address application. We saw failure rates very similar to yours with that issue.
  - nogox 10y ago
piva00 10y ago
I've been using Docker in production for the past 2 and a half years (in two different companies) and even with no extreme use cases we've had problems with: performance of volumes/devicemapper, random breaking bugs: daemon would restart without warning or errors in 1.4, randomly killing containers in 1.9, having to restart the daemon in 1.8 when it hung pulling images (consequently killing the containers in the process). I still like Docker and can see myself, team and company using it for a long time if nothing MUCH better show up (rkt is promising to take some of the complexity pain away but we are not diving into it yet) but I can't say I've not been bitten enough to completely avoid upgrading Docker if it isn't needed, we follow a rule to only upgrade to ".1" releases as most of our problems have been with ".0" ones.
- SEJeff 10y ago
  My favorite was docker exec -it $container bash would cause a nil pointer deterrence in docker 1.6.0 and kill the docker daemon. We've seen gobs of bugs since, but that was the most wtf gnarly one

cyphar 10y ago

I'd recommend looking at using runC (which is the underlying runtime underneath Docker). Currently we're heading for a 1.0, and the Open Containers Initiative is working on specifications that will make container runtimes interoperable and eventually provide tooling that works with all OCI runtimes. If you have anything to contribute, I would hope that you can give us a hand. :D

andrewguenther 10y ago
I'm a huge fan of the work being done on runC and would love to give you guys a hand! You'll probably see me around soon :)

StreamBright 10y ago

Would rkt be a worth to try alternative?

andrewguenther 10y ago
I really haven't looked at rkt as much as I should, but we're more likely to invest in looking at lower level tools like runC moving forward.
- StreamBright 10y ago
  Amazing thank you! I need to chose a containerization tech in the next month and I am pretty worried to go with Docker because I hear many stories about how it is not really production ready. Thanks for mentioning runC I will check it out.
  - colemickens 10y ago
    runC isn't an orchestration solution. It's a low-level component that can be (or is already) used by higher-level orchestration technologies.
    - StreamBright 10y ago
      Great, I do not need any orchestration solution at all. I need a container running thing that can encapsulate any software that we are developing (Java, C#, Node.JS, etc.). And now we are approaching the question what is my problem with Docker. I believe it is a misconception to compete with already existing tools like systemd. I especially do not want any mediocre orchestration solutions in my infrastructure that introduce big overhead and complexity that I do not need at all. One thing I learned along the way of managing large clusters (5K+ nodes) that Swarm like frameworks are extremely error prone. If you flip the problem and build a startup script the pulls down the container configuration from S3 for example and the container itself has code that attaches the instance to the right service (EKB, Haproxy, etc.) you can achieve the same without introducing services that sole purpose is to maintain a state that you do not need.

lacker 10y ago

If I were you I would add a disclaimer when criticizing Docker, mentioning that you work on AWS, since the products are competitive in some ways like EC2 Container Registry vs Docker Hub. It would be great for AWS if Docker simply focused on open source bug-fixing and let AWS provide the profitable services....

andrewguenther 10y ago
Added a disclaimer, however, the product I work on does not compete with Docker in any way. We actually rely on Docker quite heavily. No conspiracy here.
paulddraper 10y ago
"It'd be great if Docker wasn't profitable." I sort of agree, but that's not entirely realistic :)
- lcarlson 10y ago
  Why the anti capitalistic sentiment? Is programmers want to get paid for our work, right?
  - brianwawok 10y ago
    No. Programmers want huge paychecks but everyone ELSE should be FOSS and code for us for free
    - cyphar 10y ago
      ahem Free software does not need to be gratis. There are several examples of companies which charge money for free software.
      - softawre 10y ago
        ahem That's obviously not what is being discussed here. > It would be great for AWS if Docker simply focused on open source bug-fixing and let AWS provide the profitable services....

mtanski 10y ago

The same experiences we switched to using rkt, supervised by upstart (and now systemd). We have an "application" state template in our salt config and every docker update something would cause all of them to fail. Thankful the "application" state template abstracted running container enough were we switched from docker -> rkt under the covers without anybody noticing, except now we no longer fearing of container software updates.

pescerosso 10y ago
Hi mtanski, How did you replace docker with rkt? Do you have an howto that you can share?
- flexd 10y ago
  I haven't replaced Docker with rkt on a big scale (or ran Docker on a big scale), but I recently changed over some Docker containers to rkt. First off, this and the rest of the rkt docs is a good starting point https://coreos.com/rkt/docs/latest/rkt-vs-other-projects.html https://coreos.com/rkt/docs/latest/rkt-vs-other-projects.htm... Second, rkt runs Docker images without modifications, so you can swap over really easily https://coreos.com/rkt/docs/latest/running-docker-images.html https://coreos.com/rkt/docs/latest/running-docker-images.htm... rkt uses acbuild (which is part of the application container specification, see https://github.com/appc/spec https://github.com/appc/spec) to build images, and I had a very tiny Docker image just running a single Go process. I just created a shell script that ran the required acbuild commands to get as similar image. A good place to get started is the getting started guide https://coreos.com/rkt/docs/latest/getting-started-guide.html https://coreos.com/rkt/docs/latest/getting-started-guide.htm... Docker runs as a daemon, and rkt doesn't (which is one of the benefits). I just start my rkt container using systemd, so I have a systemd file with 'ExecStart=/usr/bin/rkt run myimage:1.23.4', but you can start the containers with whatever you want. It's also possible to use rkt with Kubernetes, but I have not tried that yet. http://kubernetes.io/docs/getting-started-guides/rkt/ http://kubernetes.io/docs/getting-started-guides/rkt/
mtanski 10y ago
An example of changing behavior that broke us not to long ago: https://github.com/docker/distribution/issues/1662 https://github.com/docker/distribution/issues/1662 . By the time this happened we were already working on the transition, just more motivation.

kordless 10y ago

If you haven't heard of Giant Swarm, I encourage you to contact them. They have a scalable microservices provisioning solution that can use either Docker or Kubernetes. German company. Disclaimer: I worked for them last year. Holler if you need an intro.

cookiecaper 10y ago

We're in the process of converting our 100+ cloud nodes to Docker+k8s and I have a lot of the same reservations -- the space is very immature and the tooling has a lot of kinks to work out, not only functionally but also aesthetically. It's already been a nightmare and we're not even deployed to prod yet.

justinsb 10y ago
If "cloud" is AWS, you should join the kubernetes slack sig-aws channel. Lots of community people figuring out those kinks together.
jacques_chester 10y ago
It's obligatory for me to recommend Cloud Foundry here. We've already built the platform, there's no need to build and maintain your own. It just works. Really well, actually. Disclosure: I work for Pivotal, we donate the majority of engineering to Cloud Foundry.

RRRA 10y ago

Not to mention 1.11's restart timer never were reset to 0 even after the container ran well for more than 10 seconds. (ie: after a few restart, your container would be waiting hours to start!). This and I can attest to 1.12 problems listed in this article. Can't remember the specific with 1.10, but basically, nothing really ever works as promised which make people waste a lot of time trying to make something work when it can't and second, doesn't give much trust in the product's stability. I really wish they would collaborate a lot more and fragment their solutions in smaller module while keeping everything simple. I think they have a great product, but too much growing pain.