6 ms·

Doesn't this article fail to consider the bigger, practical picture, i.e. overall resource efficiency/footprint in multi-component architectures? Containers al

by raulk 9y ago

Doesn't this article fail to consider the bigger, practical picture, i.e. overall resource efficiency/footprint in multi-component architectures? Containers allow us to condense workloads in a single OS runtime –while preserving isolation– where otherwise the same workloads would have spanned multiple machines or VMs, each with its overhead and slack (unused resources). Example: consider you need to deploy not just a single instance of Wordpress, Redis, Postgres, etc., but a complex application consisting of many of those components. You can either choose to (a) deploy each in a different machine [incurring in the overhead of OS + unused resources]; (b) in different VMs [incurring in the cost of OS, but being able to share a resource pool]; or (c) in containers, sharing the OS and incurring in the overhead of the container daemon. I would love to see an article that takes these points into account.

CrLf 9y ago
> Containers allow us to condense workloads in a single OS runtime –while preserving isolation– where otherwise the same workloads would have spanned multiple machines or VMs, each with its overhead and slack (unused resources). I'm starting to believe that the increased density provided by containerization is a myth, in practice. Both because orchestration tools bring their own overhead (compare all the proxies and filesystem/network overlays with multiple virtualized kernels) but also because containerization goes hand-in-hand with microservices thus increasing the number of components (often times with no real reason other than to be hip). If you're running 20-30 containers on a beefy VM, you're not really condensing anything. You're just moving from running hundreds of small VMs on a single server to running much less larger VMs.
- Chris2048 9y ago
  But you can argue the overheads are birthing pains.
- AnsemWise 9y ago
  Containers are an abstraction that exist using cgroups and namespaces for isolation. They use the hosts' kernel, it's not virtualized. Containers are only limited by the capabilities of namespaces and cgroups, unlike vms. You might be dismissing microservices too quickly. They do have overhead but so does any level of abstract; the benefit of them though is clear separation of responsibilities between services and residency(Swarms, clusters, etc). Both can be achieved with Vms but VMs weren't built with these goals in mind
  - AnsemWise 9y ago
    Resiliency
  - raulk 9y ago
    Using VMs to isolate single processes is like owning multiple toasters, and buying a different house to plug each toaster in.
    - angry_octet 9y ago
      Nah, that is separate physical machines. For VMs it's more like a multitenant toaster colo, with every toaster in its own asbestos cage, but sharing power and network^Wbread. For containers, it's like putting them all in one house but putting a fuse and RCD on every toaster. A traditional server is building a custom house each time and the toasters are all plugged into the same socket. I'm not sure the toaster analogy will gain mass acceptance.
    - derefr 9y ago
      What if you're hosting toasters owned by people who might be rude teenagers who want to set any house with a rival toaster in it on fire? When Docker can safely protect a Minecraft server in one container from a local DoS attack coming from a bot running in a sibling container, I'll reconsider using VMs. :P
- raulk 9y ago
  > If you're running 20-30 containers on a beefy VM, you're not really condensing anything. You're just moving from running hundreds of small VMs on a single server to running much less larger VMs. Of course you are condensing. Those small VMs would have been running a full OS runtime each. Assuming you relocate those workloads onto a single machine (1 process = 1 container), you no longer run 20-30 copies of an OS, just a single one. And the container engine takes the place of the hypervisor.
  - acdha 9y ago
    > Of course you are condensing. Those small VMs would have been running a full OS runtime each. That's something but probably not as much as you think since hypervisors can share identical pages (e.g. the Linux kernel) across guests and the base footprint for a server Linux install is not that high as a percentage of the private data most applications use. Unless you're running a ton of unnecessary services on those guests or have an application which uses almost no RAM you're talking about a fairly modest percentage savings even before you factor in all of the things you might be running for container management and other overhead on that side. The other thing to remember is that this works both ways: containers are great for being able to upgrade one component independently but that means that e.g. you might have a dozen different versions of a common shared library because not all of your containers are using the same base image & version and with Docker your storage driver might actually force shared libraries to be duplicated across all processes anyway.
    - sdeziel 9y ago
      > since hypervisors can share identical pages (e.g. the Linux kernel) With ASLR, I'm now sure the gains are that substantial.
      - derefr 9y ago
        The parent doesn't mean that the hypervisor will merge identical MMU->physical page mappings (like a Copy-on-Write process fork would); they mean that VM pages' underlying host virtual pages literally get periodically hashed for their current content by a background process on the dom0 and merged when they are found to have identical hashes. The underlying virtual page is then made copy-on-write. Or, to put that another way: the host memory for most modern hypervisors consists of a heap of "new" pages, and then a generational garbage collector that moves said pages, if still alive, into a content-addressible "old" store. As such, if two VMs each have a process that 1. calls malloc() 1000 times to get 1000 1-page buffers randomly spaced through their memory, the mappings different for each VM; and then 2. uses a fixed PRNG seed to generate random data [but the same random data] to fill those pages; then those two processes' pages will still get collapsed together for a 50% savings.
- coding123 9y ago
  Well, at least for our company we definitely see the gains and we absolutely don't see it as a Myth.. Previous project without docker to support all of our dev and QA environments: 9 VMs that we paid for. Most recent project with Docker: 2 VMs we pay for which includes all Dev and QA environments, with natively installed Nginx to hide the port differences.
- digi_owl 9y ago
  It really do seem like we can't win. Try to get more control in one place, and things sprawl out of control somewhere else. containers bundled up libs and binaries in a single package, only for an "app" to come reliant on a zoo of containers doing one little part of the whole. Makes one wonder if the stack is made of rabbits rather than turtles...
naasking 9y ago
> Containers allow us to condense workloads in a single OS runtime –while preserving isolation– where otherwise the same workloads would have spanned multiple machines or VMs, each with its overhead and slack (unused resources). Ultimately there's no reason why containerization can't be pushed right down to the language level. Consider .NET's AppDomains, or even further, a capability-secure programming language which isolates at the object level with zero overhead over ordinary languages.
- xyzzy_plugh 9y ago
  For many classes of software, sure. But there are plenty of reasons to isolate the entire OS. And Linux containers (not docker per se) are very inexpensive. Being able to compose using off the shelf packages and binaries and use userspace effectively provides a wonderful space to build solutions. I can use any language I want, any tools I want.
- infogulch 9y ago
  > no reason why containerization can't be pushed right down to the language level I started my reply by listing a plethora of reasons why this wouldn't work. (For one, this would work because it does work, right now in Erlang.) But they all came down to that it seems like you're missing some of the problems that containerization solves. You can wrap up almost any service -- regardless of language or versions or runtimes or how it interacts with the filesystem or what versions of libraries it depends on or what global configs it expects or anything -- and ship it as a self-contained normalized service that can run right alongside any other number of other self-contained normalized services that require their own global configs and libraries etc etc even if they're incompatible and whoever is deploying them doesn't even need to care. Everyone has their own language and toolset that they're comfortable with and productive in. That will never change. Containerization abstracts over all of them and normalizes their deployment. You can never get that with a solution at the language level.
  - derefr 9y ago
    > For one, this would work because it does work, right now in Erlang. Eh? Erlang has no equivalent to cgroups/namespaces, or even an equivalent of non-UID-0 code execution on its VM. There is in fact no isolation mechanism in the Erlang VM; all code is "privileged." Untrusted multitenant code execution is a pipe-dream for now, unless you graft on another sandbox inside Erlang, ala CouchDB's V8 C-port [and more recently luerl] sandboxes. (I've been very much considering contributing code for "non-privileged Erlang processes" and "Erlang process namespaces"—adding things like "namespace outboxes" that will crash their own virtual nodes rather than flood peers—but it's not there right now.)
pbh101 9y ago
d) Deploy them directly to the same bare-metal non-virtual machine, without a VM or container. Yes, you can get cross-app deployment conflicts (packaging, etc), and limits your cloud deployment options, but it is definitely another option and has lower overhead than any of the first three.
- raulk 9y ago
  That would defeat a major goal of isolation: security. It also creates fragility. What happens if a process is buggy and rallies up to 100% CPU? It affects all others. Although to solve these issues you could use cgroups and namespaces... Aaaand we're back to containers again.
  - pbh101 9y ago
    Agreed on both points. Either I missed your note about isolation on the first read or you edited.
- Veratyr 9y ago
  On packaging, can't this be solved by static linking or something like Nix?

by raulk 9y ago

CrLf 9y ago

> Containers allow us to condense workloads in a single OS runtime –while preserving isolation– where otherwise the same workloads would have spanned multiple machines or VMs, each with its overhead and slack (unused resources). I'm starting to believe that the increased density provided by containerization is a myth, in practice. Both because orchestration tools bring their own overhead (compare all the proxies and filesystem/network overlays with multiple virtualized kernels) but also because containerization goes hand-in-hand with microservices thus increasing the number of components (often times with no real reason other than to be hip). If you're running 20-30 containers on a beefy VM, you're not really condensing anything. You're just moving from running hundreds of small VMs on a single server to running much less larger VMs.

Chris2048 9y ago
But you can argue the overheads are birthing pains.
AnsemWise 9y ago
Containers are an abstraction that exist using cgroups and namespaces for isolation. They use the hosts' kernel, it's not virtualized. Containers are only limited by the capabilities of namespaces and cgroups, unlike vms. You might be dismissing microservices too quickly. They do have overhead but so does any level of abstract; the benefit of them though is clear separation of responsibilities between services and residency(Swarms, clusters, etc). Both can be achieved with Vms but VMs weren't built with these goals in mind
- AnsemWise 9y ago
  Resiliency
- raulk 9y ago
  Using VMs to isolate single processes is like owning multiple toasters, and buying a different house to plug each toaster in.
  - angry_octet 9y ago
    Nah, that is separate physical machines. For VMs it's more like a multitenant toaster colo, with every toaster in its own asbestos cage, but sharing power and network^Wbread. For containers, it's like putting them all in one house but putting a fuse and RCD on every toaster. A traditional server is building a custom house each time and the toasters are all plugged into the same socket. I'm not sure the toaster analogy will gain mass acceptance.
  - derefr 9y ago
    What if you're hosting toasters owned by people who might be rude teenagers who want to set any house with a rival toaster in it on fire? When Docker can safely protect a Minecraft server in one container from a local DoS attack coming from a bot running in a sibling container, I'll reconsider using VMs. :P
raulk 9y ago
> If you're running 20-30 containers on a beefy VM, you're not really condensing anything. You're just moving from running hundreds of small VMs on a single server to running much less larger VMs. Of course you are condensing. Those small VMs would have been running a full OS runtime each. Assuming you relocate those workloads onto a single machine (1 process = 1 container), you no longer run 20-30 copies of an OS, just a single one. And the container engine takes the place of the hypervisor.
- acdha 9y ago
  > Of course you are condensing. Those small VMs would have been running a full OS runtime each. That's something but probably not as much as you think since hypervisors can share identical pages (e.g. the Linux kernel) across guests and the base footprint for a server Linux install is not that high as a percentage of the private data most applications use. Unless you're running a ton of unnecessary services on those guests or have an application which uses almost no RAM you're talking about a fairly modest percentage savings even before you factor in all of the things you might be running for container management and other overhead on that side. The other thing to remember is that this works both ways: containers are great for being able to upgrade one component independently but that means that e.g. you might have a dozen different versions of a common shared library because not all of your containers are using the same base image & version and with Docker your storage driver might actually force shared libraries to be duplicated across all processes anyway.
  - sdeziel 9y ago
    > since hypervisors can share identical pages (e.g. the Linux kernel) With ASLR, I'm now sure the gains are that substantial.
    - derefr 9y ago
      The parent doesn't mean that the hypervisor will merge identical MMU->physical page mappings (like a Copy-on-Write process fork would); they mean that VM pages' underlying host virtual pages literally get periodically hashed for their current content by a background process on the dom0 and merged when they are found to have identical hashes. The underlying virtual page is then made copy-on-write. Or, to put that another way: the host memory for most modern hypervisors consists of a heap of "new" pages, and then a generational garbage collector that moves said pages, if still alive, into a content-addressible "old" store. As such, if two VMs each have a process that 1. calls malloc() 1000 times to get 1000 1-page buffers randomly spaced through their memory, the mappings different for each VM; and then 2. uses a fixed PRNG seed to generate random data [but the same random data] to fill those pages; then those two processes' pages will still get collapsed together for a 50% savings.
coding123 9y ago
Well, at least for our company we definitely see the gains and we absolutely don't see it as a Myth.. Previous project without docker to support all of our dev and QA environments: 9 VMs that we paid for. Most recent project with Docker: 2 VMs we pay for which includes all Dev and QA environments, with natively installed Nginx to hide the port differences.
digi_owl 9y ago
It really do seem like we can't win. Try to get more control in one place, and things sprawl out of control somewhere else. containers bundled up libs and binaries in a single package, only for an "app" to come reliant on a zoo of containers doing one little part of the whole. Makes one wonder if the stack is made of rabbits rather than turtles...

naasking 9y ago

> Containers allow us to condense workloads in a single OS runtime –while preserving isolation– where otherwise the same workloads would have spanned multiple machines or VMs, each with its overhead and slack (unused resources). Ultimately there's no reason why containerization can't be pushed right down to the language level. Consider .NET's AppDomains, or even further, a capability-secure programming language which isolates at the object level with zero overhead over ordinary languages.

xyzzy_plugh 9y ago
For many classes of software, sure. But there are plenty of reasons to isolate the entire OS. And Linux containers (not docker per se) are very inexpensive. Being able to compose using off the shelf packages and binaries and use userspace effectively provides a wonderful space to build solutions. I can use any language I want, any tools I want.
infogulch 9y ago
> no reason why containerization can't be pushed right down to the language level I started my reply by listing a plethora of reasons why this wouldn't work. (For one, this would work because it does work, right now in Erlang.) But they all came down to that it seems like you're missing some of the problems that containerization solves. You can wrap up almost any service -- regardless of language or versions or runtimes or how it interacts with the filesystem or what versions of libraries it depends on or what global configs it expects or anything -- and ship it as a self-contained normalized service that can run right alongside any other number of other self-contained normalized services that require their own global configs and libraries etc etc even if they're incompatible and whoever is deploying them doesn't even need to care. Everyone has their own language and toolset that they're comfortable with and productive in. That will never change. Containerization abstracts over all of them and normalizes their deployment. You can never get that with a solution at the language level.
- derefr 9y ago
  > For one, this would work because it does work, right now in Erlang. Eh? Erlang has no equivalent to cgroups/namespaces, or even an equivalent of non-UID-0 code execution on its VM. There is in fact no isolation mechanism in the Erlang VM; all code is "privileged." Untrusted multitenant code execution is a pipe-dream for now, unless you graft on another sandbox inside Erlang, ala CouchDB's V8 C-port [and more recently luerl] sandboxes. (I've been very much considering contributing code for "non-privileged Erlang processes" and "Erlang process namespaces"—adding things like "namespace outboxes" that will crash their own virtual nodes rather than flood peers—but it's not there right now.)

pbh101 9y ago

d) Deploy them directly to the same bare-metal non-virtual machine, without a VM or container. Yes, you can get cross-app deployment conflicts (packaging, etc), and limits your cloud deployment options, but it is definitely another option and has lower overhead than any of the first three.

raulk 9y ago
That would defeat a major goal of isolation: security. It also creates fragility. What happens if a process is buggy and rallies up to 100% CPU? It affects all others. Although to solve these issues you could use cgroups and namespaces... Aaaand we're back to containers again.
- pbh101 9y ago
  Agreed on both points. Either I missed your note about isolation on the first read or you edited.
Veratyr 9y ago
On packaging, can't this be solved by static linking or something like Nix?