3 ms·

As a Kubernetes user who's been running 500+ containers in production for a little over a year now, my chief complaint is not the Docker platform stuff, it's th

As a Kubernetes user who's been running 500+ containers in production for a little over a year now, my chief complaint is not the Docker platform stuff, it's the reliability of Docker itself. We've suffered through a number of bugs that cause node instability and lock-ups and some of the underlying issues have been open for close to a year. I don't think that this is the exact bug for our issue but it's closely related: https://github.com/docker/docker/issues/10355 https://github.com/docker/docker/issues/10355 I really want to give rkt a shot but our tooling investment in Docker is huge. I want to wait and see what the runtime ecosystem looks like in another 6 months before considering a switch.

hatred 10y ago
There are already ongoing efforts that are trying to get rid of that dependency too: - https://github.com/kelseyhightower/cri-o-tutorial https://github.com/kelseyhightower/cri-o-tutorial - Mesos already has something called UCR that allows one to run docker images minus the dependency on the docker runtime. (http://mesos.apache.org/documentation/latest/mesos-containerizer/ http://mesos.apache.org/documentation/latest/mesos-container...) Disclaimer: I work at Mesosphere on Mesos.
- yarou 10y ago
  Thanks for the link, that's a pretty neat project. I'm not a huge fan of Docker's runtime because it tends to be rather sluggish. I haven't used it in quite a while, though.
zbobet2012 10y ago
To echo this we have run 10k+ containers in production for close to a year, and docker crashes all the time. Sometimes in ways undetectable by any reasonable health polling mechanism messing up Kube as well. Edit: Relevant details https://news.ycombinator.com/item?id=13110897 https://news.ycombinator.com/item?id=13110897 Note these were not issues with using docker poorly, or out of mainline use cases. They are all issues with dockers core networking or runtime stack that are acknowledged bugs.
- cpuguy83 10y ago
  For #22502, I'm looking to get in a ring-buffer implementation for log drivers: https://github.com/docker/docker/pull/28762 https://github.com/docker/docker/pull/28762 The same could also be done for attached clients. #28889 is resolved in 1.12.4 #26492 was found to be not a Docker issue, but if you are still having an issue here, please open a new one, thanks!
  - zbobet2012 10y ago
    We have a workaround in place for 26492 (a program that just queries all containers, and there VETH info and checks if they are in the bridge, force adding them if they are not) luckily so we don't have to build custom versions of Systemd. We also instituted a similar fix for 28518 (we just kill the broken container when it's detected). However, our integration suite repeatedly caught deadlocks and some panics in 1.12.* so we have yet to upgraded to that. I'll ask our infrastructure teams to do a custom CoreOS with 1.12.4 and run it through the paces; though, I know we've had issues with Kube in the CoreOS alpha channel so it may be a no go. Thanks for the reply!
justincormack 10y ago
That issue is a kernel issue in old Ubuntu kernels. If you are experiencing that, upgrade your kernel. It is not something Docker can fix. If that is not your issue, please open a new one. If your system locks up, that is a kernel issue, and as the container runtimes all use pretty much the same kernel features, switching will not fix it.
cpuguy83 10y ago
Recent versions (1.11+) have had various race conditions around stdio handling. 1.12.4 (fingers-crossed) should resolves all the issues we've come up to at this point (hopefully that's it). The rest of the lockup issues have been related to kernel bugs which would also be observed with other tools. One major issue we've had (which is really multiple kernel issues with multiple causes) is netlink (a kernel interface) not responding and the container's mutex is held forever while we wait on netlink. The locking up part should be resolved in 1.12.4 for this as well, where it uses a timeout on the netlink socket... still going to have errors from this since if netlink isn't responding there's something else weird happening, but at least the container mutex can be released you can interact with the daemon. In 1.14 we should have a lock-free container object so that any new issues that come up in this regard are isolated to a particular thread. This makes detecting that there is an issue harder, but something we could potentially track and even report on. note that the daemon isn't really locking up fully, just that any command that tries to lock the container's mutex will get stuck, which includes listing it in `docker ps`, start/stop/restart/etc on that particular container.
- chrissnell 10y ago
  The netlink not responding issue sounds like the one we've been battling. Now we just have to wait for a CoreOS release with 1.12.4.
- deleted 10y ago
  [deleted]

hatred 10y ago

There are already ongoing efforts that are trying to get rid of that dependency too: - https://github.com/kelseyhightower/cri-o-tutorial https://github.com/kelseyhightower/cri-o-tutorial - Mesos already has something called UCR that allows one to run docker images minus the dependency on the docker runtime. (http://mesos.apache.org/documentation/latest/mesos-containerizer/ http://mesos.apache.org/documentation/latest/mesos-container...) Disclaimer: I work at Mesosphere on Mesos.

yarou 10y ago
Thanks for the link, that's a pretty neat project. I'm not a huge fan of Docker's runtime because it tends to be rather sluggish. I haven't used it in quite a while, though.

zbobet2012 10y ago

To echo this we have run 10k+ containers in production for close to a year, and docker crashes all the time. Sometimes in ways undetectable by any reasonable health polling mechanism messing up Kube as well. Edit: Relevant details https://news.ycombinator.com/item?id=13110897 https://news.ycombinator.com/item?id=13110897 Note these were not issues with using docker poorly, or out of mainline use cases. They are all issues with dockers core networking or runtime stack that are acknowledged bugs.

cpuguy83 10y ago
For #22502, I'm looking to get in a ring-buffer implementation for log drivers: https://github.com/docker/docker/pull/28762 https://github.com/docker/docker/pull/28762 The same could also be done for attached clients. #28889 is resolved in 1.12.4 #26492 was found to be not a Docker issue, but if you are still having an issue here, please open a new one, thanks!
- zbobet2012 10y ago
  We have a workaround in place for 26492 (a program that just queries all containers, and there VETH info and checks if they are in the bridge, force adding them if they are not) luckily so we don't have to build custom versions of Systemd. We also instituted a similar fix for 28518 (we just kill the broken container when it's detected). However, our integration suite repeatedly caught deadlocks and some panics in 1.12.* so we have yet to upgraded to that. I'll ask our infrastructure teams to do a custom CoreOS with 1.12.4 and run it through the paces; though, I know we've had issues with Kube in the CoreOS alpha channel so it may be a no go. Thanks for the reply!

justincormack 10y ago

That issue is a kernel issue in old Ubuntu kernels. If you are experiencing that, upgrade your kernel. It is not something Docker can fix. If that is not your issue, please open a new one. If your system locks up, that is a kernel issue, and as the container runtimes all use pretty much the same kernel features, switching will not fix it.

cpuguy83 10y ago

Recent versions (1.11+) have had various race conditions around stdio handling. 1.12.4 (fingers-crossed) should resolves all the issues we've come up to at this point (hopefully that's it). The rest of the lockup issues have been related to kernel bugs which would also be observed with other tools. One major issue we've had (which is really multiple kernel issues with multiple causes) is netlink (a kernel interface) not responding and the container's mutex is held forever while we wait on netlink. The locking up part should be resolved in 1.12.4 for this as well, where it uses a timeout on the netlink socket... still going to have errors from this since if netlink isn't responding there's something else weird happening, but at least the container mutex can be released you can interact with the daemon. In 1.14 we should have a lock-free container object so that any new issues that come up in this regard are isolated to a particular thread. This makes detecting that there is an issue harder, but something we could potentially track and even report on. note that the daemon isn't really locking up fully, just that any command that tries to lock the container's mutex will get stuck, which includes listing it in `docker ps`, start/stop/restart/etc on that particular container.

chrissnell 10y ago
The netlink not responding issue sounds like the one we've been battling. Now we just have to wait for a CoreOS release with 1.12.4.
deleted 10y ago
[deleted]