6 ms·

So, a funny thing happened on the way to Linux dominance of the server: We started container-izing things and focusing on building one-service per container (or

So, a funny thing happened on the way to Linux dominance of the server: We started container-izing things and focusing on building one-service per container (or VM). Suddenly the OS matters a lot less; each container only needs the libs and kernel services that it needs to do its job. It doesn't need the whole OS and doesn't benefit from the whole OS being present. I suspect there's an opportunity for a lot of alternative systems to make inroads in that space. But, then again, if the OS doesn't matter...it may be that we all just end up using the path of least resistance, which is almost always a small Linux distro with all the stuff we're used to. But, for someone that loves Plan 9 or Solaris or OpenBSD and wants to deliver service containers, they can probably get away with deploying containers using those systems without people balking at the idea.

sophacles 10y ago
I've had similar thoughts - that containerization might start opening the path for microkernel style OSes.
- pjmlp 10y ago
  Microkernels have actually won in the embedded and real time space.
  - groovy2shoes 10y ago
    Not sure why you're being downvoted, because this is absolutely the case. Examples include L4, QNX, FreeRTOS, Minix, etc. The vast majority of kernels that have been designed specifically for embedded or real-time applications are microkernels. That, of course, doesn't mean that only microkernels are used in embedded and real-time applications. It's not uncommon to see, for example, a stripped-down Linux or even Windows in some applications. In some cases, even MS-DOS and its kin are still used. But by and large, the most of the kernels you're looking at in this space are indeed microkernels. The microkernel architecture also offers some advantages in the high-assurance space. See seL4, for example. This doesn't necessarily have anything to do with any technological superiority of microkernels, per se, but they tend to be far smaller than monolithic kernels, which in turn makes them much easier to audit and verify. The Minix 3 kernel, for example, is something like 6,000 lines of C -- roughly 2/3 the size of GNU grep -- which makes it comparatively easy to audit, simply because there's less code to audit.
ori_b 10y ago
> each container only needs the libs and kernel services that it needs to do its job. It doesn't need the whole OS and doesn't benefit from the whole OS being present. Yes, but the kernel services are the part that's incompatible between systems. Containers do not run their own kernel, but instead run on their host OS kernel.
- SwellJoe 10y ago
  A Linux/UNIX ABI is probably the minimum cost of entry for an OS to participate in this idea, at least in the short-term, since everything for the server is currently built primarily for Linux.
  - geofft 10y ago
    And NT and Solaris both support the Linux ABI now. OS X / iOS is basically the only holdout with nontrivial market share.
    - jclulow 10y ago
      I don't know that Oracle Solaris supports the Linux ABI. The revival of Linux ABI support in zones (containers) is a SmartOS (and illumos) thing, and we've diverged significantly from Solaris at this point.
      - geofft 10y ago
        Aw, looks like it was Oracle Solaris 10 only. (Is there a good generic word for Oracle Solaris + SmartOS + Illumos + etc.?)
        jclulow 10y ago
        SmartOS is really just a distribution of illumos; so are OmniOS, OpenIndiana, etc. We are all very similar as we push and pull from our common source base. When Oracle took the source closed again, they essentially forked their own private product. They have diverged significantly enough from everybody else that it's not really helpful to include them in the same family anymore. tl,dr: we just call it all "illumos" now.
wrp 10y ago
Suppose we remove everything from an OS that the particular service doesn't need, then you tie the app+OS together as a unit. How is that different from just a regular application running on the host OS? If it is something better, then why don't we write applications to be that way to begin with? Serious question.
- zbuttram 10y ago
  That's pretty much what people do with containers.
  - thaumasiotes 10y ago
    Right, the question is "how is that different from a statically-linked application?"
    - stonogo 10y ago
      It allows developers to operate like they have a statically-linked application without having to undergo the cognitive dissonance of questioning the received "wisdom" that they are bad.
    - ferbivore 10y ago
      Security?
      - thaumasiotes 10y ago
        Some details would be a little more helpful.
      - SwellJoe 10y ago
        In some regards that's currently a myth perpetrated by the folks making money on container-based deployments. Containers don't actually contain very well on Linux (yet, though some nice things are happening on that front, and you can already build very solid things with SELinux, but I haven't seen anybody actually doing so; except probably the Project Atomic folks). But, yes, in theory, a container-based deployment is hella tight on the security front. One service being compromised is no big deal: Kill it and instantly spin up a new one on a new IP (hopefully with a fix for the security issue). If your visibility into the system is high enough, you may even be able to architect detection of compromises; e.g. if your container for database service starts sending email or opens an IRC port, you know it's fucked, and you kill it with prejudice. In a system built to think of containers as black boxes with APIs, you don't need to keep any particular one around if you have reason to tear it down. Note that I haven't actually seen people doing any of this with containers. But, it's a thing that one could do that cannot be done when all of your services are applications running on a single big machine. The narrower the purpose of your container, the more secure it can become. You still have lots of moving parts in your total infrastructure (more, actually, since the container management has a cost, too), but each moving part has very well-defined boundaries, and misbehavior is easier to spot and easier to shut down quickly and via automated means.
api 10y ago
Containers and the direction they're evolving is the revenge of Tenenbaum: https://en.wikipedia.org/wiki/Tanenbaum–Torvalds_debate https://en.wikipedia.org/wiki/Tanenbaum–Torvalds_debate My take on that debate: They were both right, but they were optimizing for different things. Tenenbaum was right in a theoretical sense and an eventual sense. Microkernels are better. Separation of concerns is better. Torvalds was right in the early-mid 1990s and right in a pragmatic sense. More monolithic (or vertically integrated) kernels are easier to build and ship and are faster on smaller machines. A microkernel made little pragmatic sense on a early-mid 90s box. But keep in mind that these machines were far less powerful than a Raspberry Pi. My first Linux machine was a 66mhz 80586 clone with 32MB of RAM. Once you have machines with dozens or even hundreds of cores, NUMA, and that host massively diverse multi-tenant work loads, microkernels start making a whole lot of sense. Once you have to containerize apps for security, they make sense too. They also make sense if we want to eliminate reboots, etc. They make much more pragmatic sense now than they did in the 90s. Ultimately microkernels or microkernel-like systems will win.
- mwcampbell 10y ago
  What do containers have to do with microkernels? If we're talking about BSD jails, Solaris/Illumos zones, and Linux cgroups + namespaces, there are basically two scenarios: Either multi-tenancy is handled through hardware virtualization (mainstream cloud providers), or all of the containers are run on a shared monolithic kernel (Joyent's Triton, running Linux binaries in containers via LX-branded zones on Illumos). I suppose the former scenario could be seen as a vindication of microkernels, if you consider the Xen hypervisor a microkernel. But that has nothing to do with containers.
  - MustardTiger 10y ago
    >What do containers have to do with microkernels? Xen is becoming a major OS, and it is a microkernel OS.
    - vertex-four 10y ago
      Containers are not VMs.
pjmlp 10y ago
For me as a developer, the OS stopped to matter when I moved away from C and C++ back into languages with richer runtimes and libraries. I also started to see POSIX as the extended C runtime that didn't make it into the ANSI C standard.
- eatbitseveryday 10y ago
  Interestingly, this argument was also examined in a recent conference [1] and the same conclusions are drawn. Specifically: applications are using high-level frameworks more (and POSIX less) which use platform-specific extensions, ioctl dominates, and new (disjoint) abstractions are introduced by the major OS platforms. [1] http://dx.doi.org/10.1145/2901318.2901350 http://dx.doi.org/10.1145/2901318.2901350
  - deleted 10y ago
    [deleted]
ris 10y ago
Your OS is now your hypervisor. If you don't see that, then your containers are so loosely coupled they barely have to interact with each other. Wait until you need to tightly couple several components and you'll be back to square one.

sophacles 10y ago

I've had similar thoughts - that containerization might start opening the path for microkernel style OSes.

pjmlp 10y ago
Microkernels have actually won in the embedded and real time space.
- groovy2shoes 10y ago
  Not sure why you're being downvoted, because this is absolutely the case. Examples include L4, QNX, FreeRTOS, Minix, etc. The vast majority of kernels that have been designed specifically for embedded or real-time applications are microkernels. That, of course, doesn't mean that only microkernels are used in embedded and real-time applications. It's not uncommon to see, for example, a stripped-down Linux or even Windows in some applications. In some cases, even MS-DOS and its kin are still used. But by and large, the most of the kernels you're looking at in this space are indeed microkernels. The microkernel architecture also offers some advantages in the high-assurance space. See seL4, for example. This doesn't necessarily have anything to do with any technological superiority of microkernels, per se, but they tend to be far smaller than monolithic kernels, which in turn makes them much easier to audit and verify. The Minix 3 kernel, for example, is something like 6,000 lines of C -- roughly 2/3 the size of GNU grep -- which makes it comparatively easy to audit, simply because there's less code to audit.

ori_b 10y ago

> each container only needs the libs and kernel services that it needs to do its job. It doesn't need the whole OS and doesn't benefit from the whole OS being present. Yes, but the kernel services are the part that's incompatible between systems. Containers do not run their own kernel, but instead run on their host OS kernel.

SwellJoe 10y ago
A Linux/UNIX ABI is probably the minimum cost of entry for an OS to participate in this idea, at least in the short-term, since everything for the server is currently built primarily for Linux.
- geofft 10y ago
  And NT and Solaris both support the Linux ABI now. OS X / iOS is basically the only holdout with nontrivial market share.
  - jclulow 10y ago
    I don't know that Oracle Solaris supports the Linux ABI. The revival of Linux ABI support in zones (containers) is a SmartOS (and illumos) thing, and we've diverged significantly from Solaris at this point.
    - geofft 10y ago
      Aw, looks like it was Oracle Solaris 10 only. (Is there a good generic word for Oracle Solaris + SmartOS + Illumos + etc.?)
      - jclulow 10y ago
        SmartOS is really just a distribution of illumos; so are OmniOS, OpenIndiana, etc. We are all very similar as we push and pull from our common source base. When Oracle took the source closed again, they essentially forked their own private product. They have diverged significantly enough from everybody else that it's not really helpful to include them in the same family anymore. tl,dr: we just call it all "illumos" now.

wrp 10y ago

Suppose we remove everything from an OS that the particular service doesn't need, then you tie the app+OS together as a unit. How is that different from just a regular application running on the host OS? If it is something better, then why don't we write applications to be that way to begin with? Serious question.

zbuttram 10y ago
That's pretty much what people do with containers.
- thaumasiotes 10y ago
  Right, the question is "how is that different from a statically-linked application?"
  - stonogo 10y ago
    It allows developers to operate like they have a statically-linked application without having to undergo the cognitive dissonance of questioning the received "wisdom" that they are bad.
  - ferbivore 10y ago
    Security?
    - thaumasiotes 10y ago
      Some details would be a little more helpful.
    - SwellJoe 10y ago
      In some regards that's currently a myth perpetrated by the folks making money on container-based deployments. Containers don't actually contain very well on Linux (yet, though some nice things are happening on that front, and you can already build very solid things with SELinux, but I haven't seen anybody actually doing so; except probably the Project Atomic folks). But, yes, in theory, a container-based deployment is hella tight on the security front. One service being compromised is no big deal: Kill it and instantly spin up a new one on a new IP (hopefully with a fix for the security issue). If your visibility into the system is high enough, you may even be able to architect detection of compromises; e.g. if your container for database service starts sending email or opens an IRC port, you know it's fucked, and you kill it with prejudice. In a system built to think of containers as black boxes with APIs, you don't need to keep any particular one around if you have reason to tear it down. Note that I haven't actually seen people doing any of this with containers. But, it's a thing that one could do that cannot be done when all of your services are applications running on a single big machine. The narrower the purpose of your container, the more secure it can become. You still have lots of moving parts in your total infrastructure (more, actually, since the container management has a cost, too), but each moving part has very well-defined boundaries, and misbehavior is easier to spot and easier to shut down quickly and via automated means.

api 10y ago

Containers and the direction they're evolving is the revenge of Tenenbaum: https://en.wikipedia.org/wiki/Tanenbaum–Torvalds_debate https://en.wikipedia.org/wiki/Tanenbaum–Torvalds_debate My take on that debate: They were both right, but they were optimizing for different things. Tenenbaum was right in a theoretical sense and an eventual sense. Microkernels are better. Separation of concerns is better. Torvalds was right in the early-mid 1990s and right in a pragmatic sense. More monolithic (or vertically integrated) kernels are easier to build and ship and are faster on smaller machines. A microkernel made little pragmatic sense on a early-mid 90s box. But keep in mind that these machines were far less powerful than a Raspberry Pi. My first Linux machine was a 66mhz 80586 clone with 32MB of RAM. Once you have machines with dozens or even hundreds of cores, NUMA, and that host massively diverse multi-tenant work loads, microkernels start making a whole lot of sense. Once you have to containerize apps for security, they make sense too. They also make sense if we want to eliminate reboots, etc. They make much more pragmatic sense now than they did in the 90s. Ultimately microkernels or microkernel-like systems will win.

mwcampbell 10y ago
What do containers have to do with microkernels? If we're talking about BSD jails, Solaris/Illumos zones, and Linux cgroups + namespaces, there are basically two scenarios: Either multi-tenancy is handled through hardware virtualization (mainstream cloud providers), or all of the containers are run on a shared monolithic kernel (Joyent's Triton, running Linux binaries in containers via LX-branded zones on Illumos). I suppose the former scenario could be seen as a vindication of microkernels, if you consider the Xen hypervisor a microkernel. But that has nothing to do with containers.
- MustardTiger 10y ago
  >What do containers have to do with microkernels? Xen is becoming a major OS, and it is a microkernel OS.
  - vertex-four 10y ago
    Containers are not VMs.

pjmlp 10y ago

For me as a developer, the OS stopped to matter when I moved away from C and C++ back into languages with richer runtimes and libraries. I also started to see POSIX as the extended C runtime that didn't make it into the ANSI C standard.

eatbitseveryday 10y ago
Interestingly, this argument was also examined in a recent conference [1] and the same conclusions are drawn. Specifically: applications are using high-level frameworks more (and POSIX less) which use platform-specific extensions, ioctl dominates, and new (disjoint) abstractions are introduced by the major OS platforms. [1] http://dx.doi.org/10.1145/2901318.2901350 http://dx.doi.org/10.1145/2901318.2901350
- deleted 10y ago
  [deleted]

ris 10y ago

Your OS is now your hypervisor. If you don't see that, then your containers are so loosely coupled they barely have to interact with each other. Wait until you need to tightly couple several components and you'll be back to square one.