5 ms·

To my eyes, this signals that Docker hears the complaints from systems like Kubernetes that want to build on top of docker, but don't want all the Docker platfo

To my eyes, this signals that Docker hears the complaints from systems like Kubernetes that want to build on top of docker, but don't want all the Docker platform stuff. Layering matters, and Docker seems to be offering a solution that is Docker compatible but more usable by layered systems. Devil's in the details, of course, but this seems like what our customers want - docker but not Docker. Disclosure: Kubernetes dev

chrissnell 10y ago
As a Kubernetes user who's been running 500+ containers in production for a little over a year now, my chief complaint is not the Docker platform stuff, it's the reliability of Docker itself. We've suffered through a number of bugs that cause node instability and lock-ups and some of the underlying issues have been open for close to a year. I don't think that this is the exact bug for our issue but it's closely related: https://github.com/docker/docker/issues/10355 https://github.com/docker/docker/issues/10355 I really want to give rkt a shot but our tooling investment in Docker is huge. I want to wait and see what the runtime ecosystem looks like in another 6 months before considering a switch.
- hatred 10y ago
  There are already ongoing efforts that are trying to get rid of that dependency too: - https://github.com/kelseyhightower/cri-o-tutorial https://github.com/kelseyhightower/cri-o-tutorial - Mesos already has something called UCR that allows one to run docker images minus the dependency on the docker runtime. (http://mesos.apache.org/documentation/latest/mesos-containerizer/ http://mesos.apache.org/documentation/latest/mesos-container...) Disclaimer: I work at Mesosphere on Mesos.
  - yarou 10y ago
    Thanks for the link, that's a pretty neat project. I'm not a huge fan of Docker's runtime because it tends to be rather sluggish. I haven't used it in quite a while, though.
- zbobet2012 10y ago
  To echo this we have run 10k+ containers in production for close to a year, and docker crashes all the time. Sometimes in ways undetectable by any reasonable health polling mechanism messing up Kube as well. Edit: Relevant details https://news.ycombinator.com/item?id=13110897 https://news.ycombinator.com/item?id=13110897 Note these were not issues with using docker poorly, or out of mainline use cases. They are all issues with dockers core networking or runtime stack that are acknowledged bugs.
  - cpuguy83 10y ago
    For #22502, I'm looking to get in a ring-buffer implementation for log drivers: https://github.com/docker/docker/pull/28762 https://github.com/docker/docker/pull/28762 The same could also be done for attached clients. #28889 is resolved in 1.12.4 #26492 was found to be not a Docker issue, but if you are still having an issue here, please open a new one, thanks!
    - zbobet2012 10y ago
      We have a workaround in place for 26492 (a program that just queries all containers, and there VETH info and checks if they are in the bridge, force adding them if they are not) luckily so we don't have to build custom versions of Systemd. We also instituted a similar fix for 28518 (we just kill the broken container when it's detected). However, our integration suite repeatedly caught deadlocks and some panics in 1.12.* so we have yet to upgraded to that. I'll ask our infrastructure teams to do a custom CoreOS with 1.12.4 and run it through the paces; though, I know we've had issues with Kube in the CoreOS alpha channel so it may be a no go. Thanks for the reply!
- justincormack 10y ago
  That issue is a kernel issue in old Ubuntu kernels. If you are experiencing that, upgrade your kernel. It is not something Docker can fix. If that is not your issue, please open a new one. If your system locks up, that is a kernel issue, and as the container runtimes all use pretty much the same kernel features, switching will not fix it.
- cpuguy83 10y ago
  Recent versions (1.11+) have had various race conditions around stdio handling. 1.12.4 (fingers-crossed) should resolves all the issues we've come up to at this point (hopefully that's it). The rest of the lockup issues have been related to kernel bugs which would also be observed with other tools. One major issue we've had (which is really multiple kernel issues with multiple causes) is netlink (a kernel interface) not responding and the container's mutex is held forever while we wait on netlink. The locking up part should be resolved in 1.12.4 for this as well, where it uses a timeout on the netlink socket... still going to have errors from this since if netlink isn't responding there's something else weird happening, but at least the container mutex can be released you can interact with the daemon. In 1.14 we should have a lock-free container object so that any new issues that come up in this regard are isolated to a particular thread. This makes detecting that there is an issue harder, but something we could potentially track and even report on. note that the daemon isn't really locking up fully, just that any command that tries to lock the container's mutex will get stuck, which includes listing it in `docker ps`, start/stop/restart/etc on that particular container.
  - chrissnell 10y ago
    The netlink not responding issue sounds like the one we've been battling. Now we just have to wait for a CoreOS release with 1.12.4.
  - deleted 10y ago
    [deleted]
vidarh 10y ago
I see it largely as a response to rkt, which again was largely a response to Docker either not wanting to, or being slow at, doing this kind of separation. Avoiding thing Docker daemon, for example, was a big motivation with rkt. But of course it's gotten even more important as Docker has ballooned in complexity.
- wlamartin 10y ago
  I suspect it's more likely an immediate response to https://github.com/kubernetes-incubator/cri-o https://github.com/kubernetes-incubator/cri-o People still want to use runc because it's a great piece of battle tested kit but don't want the kitchen sink and projects like CRI-O and Garden-Runc/Guardian are pushing the much simpler, smaller surface area angle.
  - deleted 10y ago
    [deleted]
  - philips 10y ago
    Yes, I agree runc is a solid piece of kit. The rkt team is starting to use runc inside as well and hope to enable that in forthcoming releases to stay close to the Open Container Initiative specs as they move to 1.0. You can read more about that on our recent update post[1] or see a demo from a recent meeting[2]. Overall I appreciate all of the discussion on architecture of these systems and nailing this stuff down into the right format. My end goal is to ensure people get a strong reliable system, the exact technologies and projects used to get there is less important. [1] https://coreos.com/blog/rkt-and-kubernetes.html https://coreos.com/blog/rkt-and-kubernetes.html [2] https://www.youtube.com/watch?v=x84Nes2P3Z4&t=124s https://www.youtube.com/watch?v=x84Nes2P3Z4&t=124s
    - thockingoog 10y ago
      IF this announcement is all it seems, and I am expecting CoreOS folks to find any real trouble spots, this makes the various options at the lower-part of the stack much more similar.
  - 10y ago
xorcist 10y ago
If that's what you are looking for, why not use rkt? (Perspective: Docker in production. Would rather bite my arm off than do it again. rkt looks like a sane Docker, but I haven't used it in anger.)
markbnj 10y ago
Absolutely agree with this. Now I need to figure out for myself how this fits in with or contrasts with runc.
- shykes 10y ago
  containerd is built on top of runc. Optionally you can swap out runc and replace it with any oci-compatible backend, like for exame runv.

chrissnell 10y ago

As a Kubernetes user who's been running 500+ containers in production for a little over a year now, my chief complaint is not the Docker platform stuff, it's the reliability of Docker itself. We've suffered through a number of bugs that cause node instability and lock-ups and some of the underlying issues have been open for close to a year. I don't think that this is the exact bug for our issue but it's closely related: https://github.com/docker/docker/issues/10355 https://github.com/docker/docker/issues/10355 I really want to give rkt a shot but our tooling investment in Docker is huge. I want to wait and see what the runtime ecosystem looks like in another 6 months before considering a switch.

hatred 10y ago
There are already ongoing efforts that are trying to get rid of that dependency too: - https://github.com/kelseyhightower/cri-o-tutorial https://github.com/kelseyhightower/cri-o-tutorial - Mesos already has something called UCR that allows one to run docker images minus the dependency on the docker runtime. (http://mesos.apache.org/documentation/latest/mesos-containerizer/ http://mesos.apache.org/documentation/latest/mesos-container...) Disclaimer: I work at Mesosphere on Mesos.
- yarou 10y ago
  Thanks for the link, that's a pretty neat project. I'm not a huge fan of Docker's runtime because it tends to be rather sluggish. I haven't used it in quite a while, though.
zbobet2012 10y ago
To echo this we have run 10k+ containers in production for close to a year, and docker crashes all the time. Sometimes in ways undetectable by any reasonable health polling mechanism messing up Kube as well. Edit: Relevant details https://news.ycombinator.com/item?id=13110897 https://news.ycombinator.com/item?id=13110897 Note these were not issues with using docker poorly, or out of mainline use cases. They are all issues with dockers core networking or runtime stack that are acknowledged bugs.
- cpuguy83 10y ago
  For #22502, I'm looking to get in a ring-buffer implementation for log drivers: https://github.com/docker/docker/pull/28762 https://github.com/docker/docker/pull/28762 The same could also be done for attached clients. #28889 is resolved in 1.12.4 #26492 was found to be not a Docker issue, but if you are still having an issue here, please open a new one, thanks!
  - zbobet2012 10y ago
    We have a workaround in place for 26492 (a program that just queries all containers, and there VETH info and checks if they are in the bridge, force adding them if they are not) luckily so we don't have to build custom versions of Systemd. We also instituted a similar fix for 28518 (we just kill the broken container when it's detected). However, our integration suite repeatedly caught deadlocks and some panics in 1.12.* so we have yet to upgraded to that. I'll ask our infrastructure teams to do a custom CoreOS with 1.12.4 and run it through the paces; though, I know we've had issues with Kube in the CoreOS alpha channel so it may be a no go. Thanks for the reply!
justincormack 10y ago
That issue is a kernel issue in old Ubuntu kernels. If you are experiencing that, upgrade your kernel. It is not something Docker can fix. If that is not your issue, please open a new one. If your system locks up, that is a kernel issue, and as the container runtimes all use pretty much the same kernel features, switching will not fix it.
cpuguy83 10y ago
Recent versions (1.11+) have had various race conditions around stdio handling. 1.12.4 (fingers-crossed) should resolves all the issues we've come up to at this point (hopefully that's it). The rest of the lockup issues have been related to kernel bugs which would also be observed with other tools. One major issue we've had (which is really multiple kernel issues with multiple causes) is netlink (a kernel interface) not responding and the container's mutex is held forever while we wait on netlink. The locking up part should be resolved in 1.12.4 for this as well, where it uses a timeout on the netlink socket... still going to have errors from this since if netlink isn't responding there's something else weird happening, but at least the container mutex can be released you can interact with the daemon. In 1.14 we should have a lock-free container object so that any new issues that come up in this regard are isolated to a particular thread. This makes detecting that there is an issue harder, but something we could potentially track and even report on. note that the daemon isn't really locking up fully, just that any command that tries to lock the container's mutex will get stuck, which includes listing it in `docker ps`, start/stop/restart/etc on that particular container.
- chrissnell 10y ago
  The netlink not responding issue sounds like the one we've been battling. Now we just have to wait for a CoreOS release with 1.12.4.
- deleted 10y ago
  [deleted]

vidarh 10y ago

I see it largely as a response to rkt, which again was largely a response to Docker either not wanting to, or being slow at, doing this kind of separation. Avoiding thing Docker daemon, for example, was a big motivation with rkt. But of course it's gotten even more important as Docker has ballooned in complexity.

wlamartin 10y ago
I suspect it's more likely an immediate response to https://github.com/kubernetes-incubator/cri-o https://github.com/kubernetes-incubator/cri-o People still want to use runc because it's a great piece of battle tested kit but don't want the kitchen sink and projects like CRI-O and Garden-Runc/Guardian are pushing the much simpler, smaller surface area angle.
- deleted 10y ago
  [deleted]
- philips 10y ago
  Yes, I agree runc is a solid piece of kit. The rkt team is starting to use runc inside as well and hope to enable that in forthcoming releases to stay close to the Open Container Initiative specs as they move to 1.0. You can read more about that on our recent update post[1] or see a demo from a recent meeting[2]. Overall I appreciate all of the discussion on architecture of these systems and nailing this stuff down into the right format. My end goal is to ensure people get a strong reliable system, the exact technologies and projects used to get there is less important. [1] https://coreos.com/blog/rkt-and-kubernetes.html https://coreos.com/blog/rkt-and-kubernetes.html [2] https://www.youtube.com/watch?v=x84Nes2P3Z4&t=124s https://www.youtube.com/watch?v=x84Nes2P3Z4&t=124s
  - thockingoog 10y ago
    IF this announcement is all it seems, and I am expecting CoreOS folks to find any real trouble spots, this makes the various options at the lower-part of the stack much more similar.
- 10y ago

xorcist 10y ago

If that's what you are looking for, why not use rkt? (Perspective: Docker in production. Would rather bite my arm off than do it again. rkt looks like a sane Docker, but I haven't used it in anger.)

markbnj 10y ago

Absolutely agree with this. Now I need to figure out for myself how this fits in with or contrasts with runc.

shykes 10y ago
containerd is built on top of runc. Optionally you can swap out runc and replace it with any oci-compatible backend, like for exame runv.