3 ms·

> If you're running 20-30 containers on a beefy VM, you're not really condensing anything. You're just moving from running hundreds of small VMs on a single ser

by raulk 9y ago

> If you're running 20-30 containers on a beefy VM, you're not really condensing anything. You're just moving from running hundreds of small VMs on a single server to running much less larger VMs. Of course you are condensing. Those small VMs would have been running a full OS runtime each. Assuming you relocate those workloads onto a single machine (1 process = 1 container), you no longer run 20-30 copies of an OS, just a single one. And the container engine takes the place of the hypervisor.

acdha 9y ago
> Of course you are condensing. Those small VMs would have been running a full OS runtime each. That's something but probably not as much as you think since hypervisors can share identical pages (e.g. the Linux kernel) across guests and the base footprint for a server Linux install is not that high as a percentage of the private data most applications use. Unless you're running a ton of unnecessary services on those guests or have an application which uses almost no RAM you're talking about a fairly modest percentage savings even before you factor in all of the things you might be running for container management and other overhead on that side. The other thing to remember is that this works both ways: containers are great for being able to upgrade one component independently but that means that e.g. you might have a dozen different versions of a common shared library because not all of your containers are using the same base image & version and with Docker your storage driver might actually force shared libraries to be duplicated across all processes anyway.
- sdeziel 9y ago
  > since hypervisors can share identical pages (e.g. the Linux kernel) With ASLR, I'm now sure the gains are that substantial.
  - derefr 9y ago
    The parent doesn't mean that the hypervisor will merge identical MMU->physical page mappings (like a Copy-on-Write process fork would); they mean that VM pages' underlying host virtual pages literally get periodically hashed for their current content by a background process on the dom0 and merged when they are found to have identical hashes. The underlying virtual page is then made copy-on-write. Or, to put that another way: the host memory for most modern hypervisors consists of a heap of "new" pages, and then a generational garbage collector that moves said pages, if still alive, into a content-addressible "old" store. As such, if two VMs each have a process that 1. calls malloc() 1000 times to get 1000 1-page buffers randomly spaced through their memory, the mappings different for each VM; and then 2. uses a fixed PRNG seed to generate random data [but the same random data] to fill those pages; then those two processes' pages will still get collapsed together for a 50% savings.
patrickg_zill 9y ago
I'm not sure that e.g. OpenVZ or LXC would have that much larger a footprint in RAM, even if they do have a full OS runtime. A few more daemons running, that would be it. Everything else is mapped onto the single Linux kernel, and many pages are shared as a result; I think that even libs are able to be shared across VMs; so if you had 2 identical versions of glibc in 2 VMs, only 1 would be loaded and used.
- chc 9y ago
  Doesn't LXC also mean containers? Like, the C literally stands for "Container."
  - sdeziel 9y ago
    LXC (with LXD) or OpenVZ containers are typically shipping full OS in the container. Docker is different in that it typically only have a few processes per containers.
    - derefr 9y ago
      "Full OS" is confusing here. Let's be explicit: 1. "application containers" are effectively a single process [though that can fork more] with some kernel process-struct fields set to nonzero values, indicating that the kernel should present this process a different view of its environment. 2. "virtual machines" are the processor providing a separate virtualized view of the CPU, on which is then booted another virtualized kernel, which brings up with it virtualized OS services and eventually an app. 3. Between them, "OS containers" are a hybrid: they start up all the userland virtualized OS services that a VM does, but they do so on top of a kernel that's not actually a fresh, separate kernel; but instead a kernel that has been told (through setting tons of containerization process flags) to present to this group of processes a view of the world where this kernel looks like a fresh kernel in a newly-started VM. "OS containers" are basically a raw optimization over VMs by asking one kernel to pretend to be multiple kernels, and to manage one pool of memory instead of having multiple pools of memory. Anything you can do with raw VMs, you should (in theory, given good inter-container isolation+quota logic) be able to do with OS containers as well.

by raulk 9y ago

acdha 9y ago

> Of course you are condensing. Those small VMs would have been running a full OS runtime each. That's something but probably not as much as you think since hypervisors can share identical pages (e.g. the Linux kernel) across guests and the base footprint for a server Linux install is not that high as a percentage of the private data most applications use. Unless you're running a ton of unnecessary services on those guests or have an application which uses almost no RAM you're talking about a fairly modest percentage savings even before you factor in all of the things you might be running for container management and other overhead on that side. The other thing to remember is that this works both ways: containers are great for being able to upgrade one component independently but that means that e.g. you might have a dozen different versions of a common shared library because not all of your containers are using the same base image & version and with Docker your storage driver might actually force shared libraries to be duplicated across all processes anyway.

sdeziel 9y ago
> since hypervisors can share identical pages (e.g. the Linux kernel) With ASLR, I'm now sure the gains are that substantial.
- derefr 9y ago
  The parent doesn't mean that the hypervisor will merge identical MMU->physical page mappings (like a Copy-on-Write process fork would); they mean that VM pages' underlying host virtual pages literally get periodically hashed for their current content by a background process on the dom0 and merged when they are found to have identical hashes. The underlying virtual page is then made copy-on-write. Or, to put that another way: the host memory for most modern hypervisors consists of a heap of "new" pages, and then a generational garbage collector that moves said pages, if still alive, into a content-addressible "old" store. As such, if two VMs each have a process that 1. calls malloc() 1000 times to get 1000 1-page buffers randomly spaced through their memory, the mappings different for each VM; and then 2. uses a fixed PRNG seed to generate random data [but the same random data] to fill those pages; then those two processes' pages will still get collapsed together for a 50% savings.

patrickg_zill 9y ago

I'm not sure that e.g. OpenVZ or LXC would have that much larger a footprint in RAM, even if they do have a full OS runtime. A few more daemons running, that would be it. Everything else is mapped onto the single Linux kernel, and many pages are shared as a result; I think that even libs are able to be shared across VMs; so if you had 2 identical versions of glibc in 2 VMs, only 1 would be loaded and used.

chc 9y ago
Doesn't LXC also mean containers? Like, the C literally stands for "Container."
- sdeziel 9y ago
  LXC (with LXD) or OpenVZ containers are typically shipping full OS in the container. Docker is different in that it typically only have a few processes per containers.
  - derefr 9y ago
    "Full OS" is confusing here. Let's be explicit: 1. "application containers" are effectively a single process [though that can fork more] with some kernel process-struct fields set to nonzero values, indicating that the kernel should present this process a different view of its environment. 2. "virtual machines" are the processor providing a separate virtualized view of the CPU, on which is then booted another virtualized kernel, which brings up with it virtualized OS services and eventually an app. 3. Between them, "OS containers" are a hybrid: they start up all the userland virtualized OS services that a VM does, but they do so on top of a kernel that's not actually a fresh, separate kernel; but instead a kernel that has been told (through setting tons of containerization process flags) to present to this group of processes a view of the world where this kernel looks like a fresh kernel in a newly-started VM. "OS containers" are basically a raw optimization over VMs by asking one kernel to pretend to be multiple kernels, and to manage one pool of memory instead of having multiple pools of memory. Anything you can do with raw VMs, you should (in theory, given good inter-container isolation+quota logic) be able to do with OS containers as well.