4 ms·

OT but why do you want to push containers? In my opinion it's a misadventure from the real issue of a proper package manager (which in turn is a misadventure fr

OT but why do you want to push containers? In my opinion it's a misadventure from the real issue of a proper package manager (which in turn is a misadventure from a proper build system) The issue with current containers is it's the wrong level of abstraction, containment should be per binary and ideally inline with code (see pledge(2)).

barrkel 8y ago
K8s is about cluster orchestration. It's for building apps that allocate memory and compute from a pool of machines almost like malloc() and free(). It's also an abstraction for allocating compute away from the specifics of a provider, so you theoretically don't care if you deploy to gcloud or Azure or AWS. Containers and images merely define the interface with the agnostically allocated compute. They're not the essential bit though, not the magic.
- djsumdog 8y ago
  Right, and there are some scheduling systems like Maraton/mesos that can be used to control jobs or VMs as well (although they're more often used with containers than anything else these days). Can K8s run VMs or is its only abstraction containers/pods?
  - dharmab 8y ago
    Containers in pods only. In practice we run all of our Mesos tasks and Marathon services in containers anyway, as you mentioned.
  - peteridah 8y ago
    The pod abstraction can be implemented as a vm. Take a look at clear containers
- nwmcsween 8y ago
  I never said K8s wasn't about orchestration, I said why push containers in their current horribly broken form
deleted 8y ago
[deleted]
cyphar 8y ago
As one of the maintainers of runc, I cannot agree with your general thrust more (I am obviously biased since I work for a distribution company -- SUSE). I don't agree it should be containment-per-binary, but I think that containment should be transparent to the operating system so it can be managed. Currently the idea with k8s is that you just destroy anything that has broken state and rebuild it somewhere else. While this is the easiest solution to the problem, I think that containers should not always be treated like disposable objects. I think in many ways, Solaris Zones (and FreeBSD Jails to a lesser extent) did this right. Every administrative command in Solaris is zone-aware and you can manage your zones like you manage your host. On Linux, containers are completely opaque root filesystems which contain who-knows-what and are completely unmanagable or introspectable by the host. Everyone is cargo-culting and creating hacks around this fundamentally opaque system to try to make it less opaque (or just giving up and making it easy to purge them). I'm working on some improvements to the OCI image-spec that _might_ help with introspection of images[1], but runtime introspection is going to be quite hard (needs runtime agreement). I get that it's no longer sexy to work on container runtimes, now that everyone is losing their mind over orchestrators (and I imagine soon they'll be losing their mind over the next big thing). But I think that improving these fundamental blocks is pretty interesting and might result in better systems. [1]: https://github.com/openSUSE/umoci/issues/256 https://github.com/openSUSE/umoci/issues/256
- pm90 8y ago
  It was never sexy to work on container runtimes, and nobody is losing their minds over orchestrators either. There was a real need for a platform to deploy to without being tied in to {aws, gcp, etc}. First it was OpenStack, now its kubernetes.
- imhoguy 8y ago
  There is always systemd-nspawn to do it the simple way.
  - cyphar 8y ago
    systemd-nspawn doesn't really help do any of what I complained about. Besides, LXC is a much better "simple" container runtime in my opinion.
Carpetsmoker 8y ago
I think pledge (and unveil) are really good ideas, but that doesn't mean it's a substitute for containers. They're kind of different: pledge mostly just enforces correctness of a program, whereas containers enforce separation. I think containers are more analogous to privsep (used extensively in OpenBSD) than pledge. My biggest issue with the current state is that the common tools (e.g. Docker, k8s) are pretty terrible. Docker overcomplicates a great many things, which is one reason for the many bugs. And k8s ... well, let's not even go there. I don't think much more is needed than just `run-container /usr/containers/cool-container` and presto, a container with the files in that directory. No need for buggy/slow platform-dependent FS abstraction layers that change every year because they never quite work right in a bazillion edge cases. This is how e.g. FreeBSD jails work. This won't prevent implementing tools like `docker pull`; at it simplest it could be `curl ... | tar xf - -C /usr/containers/cool-container`. Running multiple containers based of a single image? `cp -r`. Maybe I'm just an old Unix guy, but I think that composition from generic tools is a much better approach for a lot of this stuff. It's easier, less platform-dependent, has less code (meaning less bugs), etc.
- nwmcsween 8y ago
  Containers are basically a few syscalls wrapped in a ball of mud, pledge is containment and the Linux somewhat broken and ugly alternative seccomp bpf is used in most containers _but_ not properly as most containers are bundles of software (multiple binaries) which makes it hard to contain a container (ha) as you have to allow more syscalls than needed. A better way would just use .preinit_array and inject a .seccomp section, .unshare, etc to create per binary containment or just chain-load everything in one big line.
  - Carpetsmoker 8y ago
    > Containers are basically a few syscalls wrapped in a ball of mud I don't think it has to be; I mean, the current Linux/Docker situation is kind of that, but that's just a single implementation. > most containers are bundles of software (multiple binaries) Yeah, exactly. Containers aren't just about security, also convenience. Something like "docker pull someimage" can be a handy cross-platform way to run stuff. I think "docker pull" is the reason for Docker's popularity. You can build a self-contained container containing Ruby, all your Ruby on Rails stuff and JS stuff, and whatnot, and then send that to any server and just "docker run" it. Getting some of this dynamic stuff to run can be tricky and error-prone: need to get the correct Ruby version, Rails version, muck about with bundle, etc. Want to add a new app-server or migrate to somewhere else? Need to spend a day setting everything up again and hope you didn't make a mistake that will bite you later :-( A self-contained container is a lot easier, even with good OS package management or automation tools like Puppet. Many people also use Docker as "Vagrant" (e.g. to run dev versions or tests and whatnot). Again, "docker pull" is convenient here, especially when there are certain dependencies/requirements, or for running testing against different versions of some dependency, etc. Again, I don't think Docker is very good as such, but I do think there are some ideas worth keeping, both for security and plain convenience (not often those two go hand-in-hand!)

barrkel 8y ago

K8s is about cluster orchestration. It's for building apps that allocate memory and compute from a pool of machines almost like malloc() and free(). It's also an abstraction for allocating compute away from the specifics of a provider, so you theoretically don't care if you deploy to gcloud or Azure or AWS. Containers and images merely define the interface with the agnostically allocated compute. They're not the essential bit though, not the magic.

djsumdog 8y ago
Right, and there are some scheduling systems like Maraton/mesos that can be used to control jobs or VMs as well (although they're more often used with containers than anything else these days). Can K8s run VMs or is its only abstraction containers/pods?
- dharmab 8y ago
  Containers in pods only. In practice we run all of our Mesos tasks and Marathon services in containers anyway, as you mentioned.
- peteridah 8y ago
  The pod abstraction can be implemented as a vm. Take a look at clear containers
nwmcsween 8y ago
I never said K8s wasn't about orchestration, I said why push containers in their current horribly broken form

deleted 8y ago

[deleted]

cyphar 8y ago

As one of the maintainers of runc, I cannot agree with your general thrust more (I am obviously biased since I work for a distribution company -- SUSE). I don't agree it should be containment-per-binary, but I think that containment should be transparent to the operating system so it can be managed. Currently the idea with k8s is that you just destroy anything that has broken state and rebuild it somewhere else. While this is the easiest solution to the problem, I think that containers should not always be treated like disposable objects. I think in many ways, Solaris Zones (and FreeBSD Jails to a lesser extent) did this right. Every administrative command in Solaris is zone-aware and you can manage your zones like you manage your host. On Linux, containers are completely opaque root filesystems which contain who-knows-what and are completely unmanagable or introspectable by the host. Everyone is cargo-culting and creating hacks around this fundamentally opaque system to try to make it less opaque (or just giving up and making it easy to purge them). I'm working on some improvements to the OCI image-spec that _might_ help with introspection of images[1], but runtime introspection is going to be quite hard (needs runtime agreement). I get that it's no longer sexy to work on container runtimes, now that everyone is losing their mind over orchestrators (and I imagine soon they'll be losing their mind over the next big thing). But I think that improving these fundamental blocks is pretty interesting and might result in better systems. [1]: https://github.com/openSUSE/umoci/issues/256 https://github.com/openSUSE/umoci/issues/256

pm90 8y ago
It was never sexy to work on container runtimes, and nobody is losing their minds over orchestrators either. There was a real need for a platform to deploy to without being tied in to {aws, gcp, etc}. First it was OpenStack, now its kubernetes.
imhoguy 8y ago
There is always systemd-nspawn to do it the simple way.
- cyphar 8y ago
  systemd-nspawn doesn't really help do any of what I complained about. Besides, LXC is a much better "simple" container runtime in my opinion.

Carpetsmoker 8y ago

I think pledge (and unveil) are really good ideas, but that doesn't mean it's a substitute for containers. They're kind of different: pledge mostly just enforces correctness of a program, whereas containers enforce separation. I think containers are more analogous to privsep (used extensively in OpenBSD) than pledge. My biggest issue with the current state is that the common tools (e.g. Docker, k8s) are pretty terrible. Docker overcomplicates a great many things, which is one reason for the many bugs. And k8s ... well, let's not even go there. I don't think much more is needed than just `run-container /usr/containers/cool-container` and presto, a container with the files in that directory. No need for buggy/slow platform-dependent FS abstraction layers that change every year because they never quite work right in a bazillion edge cases. This is how e.g. FreeBSD jails work. This won't prevent implementing tools like `docker pull`; at it simplest it could be `curl ... | tar xf - -C /usr/containers/cool-container`. Running multiple containers based of a single image? `cp -r`. Maybe I'm just an old Unix guy, but I think that composition from generic tools is a much better approach for a lot of this stuff. It's easier, less platform-dependent, has less code (meaning less bugs), etc.

nwmcsween 8y ago
Containers are basically a few syscalls wrapped in a ball of mud, pledge is containment and the Linux somewhat broken and ugly alternative seccomp bpf is used in most containers _but_ not properly as most containers are bundles of software (multiple binaries) which makes it hard to contain a container (ha) as you have to allow more syscalls than needed. A better way would just use .preinit_array and inject a .seccomp section, .unshare, etc to create per binary containment or just chain-load everything in one big line.
- Carpetsmoker 8y ago
  > Containers are basically a few syscalls wrapped in a ball of mud I don't think it has to be; I mean, the current Linux/Docker situation is kind of that, but that's just a single implementation. > most containers are bundles of software (multiple binaries) Yeah, exactly. Containers aren't just about security, also convenience. Something like "docker pull someimage" can be a handy cross-platform way to run stuff. I think "docker pull" is the reason for Docker's popularity. You can build a self-contained container containing Ruby, all your Ruby on Rails stuff and JS stuff, and whatnot, and then send that to any server and just "docker run" it. Getting some of this dynamic stuff to run can be tricky and error-prone: need to get the correct Ruby version, Rails version, muck about with bundle, etc. Want to add a new app-server or migrate to somewhere else? Need to spend a day setting everything up again and hope you didn't make a mistake that will bite you later :-( A self-contained container is a lot easier, even with good OS package management or automation tools like Puppet. Many people also use Docker as "Vagrant" (e.g. to run dev versions or tests and whatnot). Again, "docker pull" is convenient here, especially when there are certain dependencies/requirements, or for running testing against different versions of some dependency, etc. Again, I don't think Docker is very good as such, but I do think there are some ideas worth keeping, both for security and plain convenience (not often those two go hand-in-hand!)