4 ms·

As one of the maintainers of runc, I cannot agree with your general thrust more (I am obviously biased since I work for a distribution company -- SUSE). I don't

by cyphar 8y ago

As one of the maintainers of runc, I cannot agree with your general thrust more (I am obviously biased since I work for a distribution company -- SUSE). I don't agree it should be containment-per-binary, but I think that containment should be transparent to the operating system so it can be managed. Currently the idea with k8s is that you just destroy anything that has broken state and rebuild it somewhere else. While this is the easiest solution to the problem, I think that containers should not always be treated like disposable objects. I think in many ways, Solaris Zones (and FreeBSD Jails to a lesser extent) did this right. Every administrative command in Solaris is zone-aware and you can manage your zones like you manage your host. On Linux, containers are completely opaque root filesystems which contain who-knows-what and are completely unmanagable or introspectable by the host. Everyone is cargo-culting and creating hacks around this fundamentally opaque system to try to make it less opaque (or just giving up and making it easy to purge them). I'm working on some improvements to the OCI image-spec that _might_ help with introspection of images[1], but runtime introspection is going to be quite hard (needs runtime agreement). I get that it's no longer sexy to work on container runtimes, now that everyone is losing their mind over orchestrators (and I imagine soon they'll be losing their mind over the next big thing). But I think that improving these fundamental blocks is pretty interesting and might result in better systems. [1]: https://github.com/openSUSE/umoci/issues/256 https://github.com/openSUSE/umoci/issues/256

pm90 8y ago
It was never sexy to work on container runtimes, and nobody is losing their minds over orchestrators either. There was a real need for a platform to deploy to without being tied in to {aws, gcp, etc}. First it was OpenStack, now its kubernetes.
imhoguy 8y ago
There is always systemd-nspawn to do it the simple way.
- cyphar 8y ago
  systemd-nspawn doesn't really help do any of what I complained about. Besides, LXC is a much better "simple" container runtime in my opinion.
ofrzeta 8y ago
> I think that containment should be transparent to the operating system so it can be managed you mean, as in "VM"? :)
- cyphar 8y ago
  VMs are entirely opaque to the host operating system. I'm not sure what the parallel is (though I assume you were being tongue-in-cheek).
  - ofrzeta 8y ago
    Well, I always had problems differentiating "opaque" and "transparent" because depending on your point of view often both concepts can be applied interchangeably. For instance in your post I didn't really get your point because at one time you seem to favor containers as a single entity from the perspective of the OS so they can be easily managed. This is what I understand you called "transparent to the OS". The is opposed to the container root fs that you call "opaque", supposedly because it cannot be easily mounted from the host OS etc. As the implementor of container runtimes you know that there's not a container entity but an agglomeration of isolation techniques such as namespaces, cgroups etc. including some kind of overlay fs that make up an actual container. For me that's not very transparent. VMs on the other hand are in that respect – in a similar way Solaris zones and Jails are, as they are actual entities that can be seen and managed on the host OS.
    - cyphar 8y ago
      I mean "transparent to the OS" in reference to operating system tooling rather than the kernel objects that make up containers (there is an argument that these are related concepts -- but I think we can have cohesive tooling without redesigning how Linux handles containers). The lack of a 'struct container_t' in Linux definitely makes containers quite complicated to set up, and is a constant source of pain. But this also allows for lots of flexibility with different userspace programs to use various subsets of containers as part of their operation (the Chrome sandbox is a good example of this). > The is opposed to the container root fs that you call "opaque", supposedly because it cannot be easily mounted from the host OS etc. Well, you can mount it and you can manually funge with it using 'nsenter' but that's not really ideal. Why (from the perspective of a user) can't you run a package upgrade across your containers? Why can't you get the free memory across all containers? Free disk space? And so on. This information is _available_ but it's not cohesive for _users_. > VMs on the other hand are in that respect – in a similar way Solaris zones and Jails are I agree that Solaris Zones and FreeBSD Jails are transparent (in fact that level of tooling transparency is what I wish we had on Linux). I don't agree that VMs are transparent in the same way -- they are an opaque DRAM blob from the perspective of the host kernel. You can dtrace zones/jails, you cannot dtrace a VM (and get meaningful results about what the guest kernel is doing) AFAIK.

by cyphar 8y ago

pm90 8y ago

It was never sexy to work on container runtimes, and nobody is losing their minds over orchestrators either. There was a real need for a platform to deploy to without being tied in to {aws, gcp, etc}. First it was OpenStack, now its kubernetes.

imhoguy 8y ago

There is always systemd-nspawn to do it the simple way.

cyphar 8y ago
systemd-nspawn doesn't really help do any of what I complained about. Besides, LXC is a much better "simple" container runtime in my opinion.

ofrzeta 8y ago

> I think that containment should be transparent to the operating system so it can be managed you mean, as in "VM"? :)

cyphar 8y ago
VMs are entirely opaque to the host operating system. I'm not sure what the parallel is (though I assume you were being tongue-in-cheek).
- ofrzeta 8y ago
  Well, I always had problems differentiating "opaque" and "transparent" because depending on your point of view often both concepts can be applied interchangeably. For instance in your post I didn't really get your point because at one time you seem to favor containers as a single entity from the perspective of the OS so they can be easily managed. This is what I understand you called "transparent to the OS". The is opposed to the container root fs that you call "opaque", supposedly because it cannot be easily mounted from the host OS etc. As the implementor of container runtimes you know that there's not a container entity but an agglomeration of isolation techniques such as namespaces, cgroups etc. including some kind of overlay fs that make up an actual container. For me that's not very transparent. VMs on the other hand are in that respect – in a similar way Solaris zones and Jails are, as they are actual entities that can be seen and managed on the host OS.
  - cyphar 8y ago
    I mean "transparent to the OS" in reference to operating system tooling rather than the kernel objects that make up containers (there is an argument that these are related concepts -- but I think we can have cohesive tooling without redesigning how Linux handles containers). The lack of a 'struct container_t' in Linux definitely makes containers quite complicated to set up, and is a constant source of pain. But this also allows for lots of flexibility with different userspace programs to use various subsets of containers as part of their operation (the Chrome sandbox is a good example of this). > The is opposed to the container root fs that you call "opaque", supposedly because it cannot be easily mounted from the host OS etc. Well, you can mount it and you can manually funge with it using 'nsenter' but that's not really ideal. Why (from the perspective of a user) can't you run a package upgrade across your containers? Why can't you get the free memory across all containers? Free disk space? And so on. This information is _available_ but it's not cohesive for _users_. > VMs on the other hand are in that respect – in a similar way Solaris zones and Jails are I agree that Solaris Zones and FreeBSD Jails are transparent (in fact that level of tooling transparency is what I wish we had on Linux). I don't agree that VMs are transparent in the same way -- they are an opaque DRAM blob from the perspective of the host kernel. You can dtrace zones/jails, you cannot dtrace a VM (and get meaningful results about what the guest kernel is doing) AFAIK.