4 ms·

Hi, I'm the author of the article. > Containers have been mature for longer than the article implies (freebsd jails, Solaris zones). Mature containers have be

Hi, I'm the author of the article. > Containers have been mature for longer than the article implies (freebsd jails, Solaris zones). Mature containers have been around since the the days of mainframes. The recent (say, last decade) disparity has been lack of mature containers on the platform of choice, which is Linux for most cloud-based projects. I don't know many people who would switch their projects to run on FreeBSD or Solaris just to use containers instead of VMs. Also, the container types you mention don't support independent, fine-grained choices of isolate-or-not for networking, the file system, user IDs, process IDs, and scheduling of system resources. > Modern virt hardware+hypervisors is blazing fast and (I believe) will eventually outpace bare metal for specific workloads on big boxes due to the improved isolation for many performance critical operations (I.e. separate TLBs for flushes, smarter cache invalidation). It's unclear why you think such optimization would appear in hypervisors but not kernels, especially given how much more insight a kernel has into the workloads it directly runs compared to a hypervisor running a kernel running workloads. Even if hypervisors achieve CPU performance parity for running systems, you still haven't addressed the memory and storage overhead (in terms of resident footprint) of running full OS images instead of containers. > Even if they don't reach parity, the gap certainty wouldn't justify giving up all the benefits of full virtualization. You're portraying virtualization as having benefits versus containerization while ignoring that containerization also has many benefits over virtualization, especially w.r.t. cost efficiency.

aliguori 13y ago
Hi David, > Mature containers have been around since the the days of mainframes. Citation needed. I'd go as far as to say that there is no such thing as a mature container technology. The fundamental problem of containers is that mainstreams kernels are not designed to be multi-tenant. They support multiple users quite well but when you try to have multiple root users, badness ensues. Even the best container systems today for Linux still have fundamental gaps. > Even if hypervisors achieve CPU performance parity for running systems, you still haven't addressed the memory and storage overhead (in terms of resident footprint) of running full OS images instead of containers. Re: memory, same page merging can eliminate a lot of that overhead when running mostly homogenous workloads. Re: storage using CoW not only makes provisioning instant (not 5-10 minutes) but also addresses the storage concern. If you mean the overhead of running two kernels, well, actually using namespace has a fair amount of practical overhead too. Of course, benchmarks speak louder than words here. No one has published a container based result for SPECvirt because I'm quite sure it's not faster than virtualization.
- justincormack 13y ago
  Linux containers are pretty immature, they only started being implemented recently; mainframes, Solaris, FreeBSD are much older. The Linux implementation is interesting though in terms of granularity.
  - makomk 13y ago
    If you ignore stuff like OpenVZ, maybe, but there's probably a reason why that was mostly displaced by full virtualization in the first place.
- davidstrauss 13y ago
  > Citation needed. First, I define containers by their capabilities, not a vendor or kernel calling them "containers." The Linux kernel, internally, has no concept of containers; it merely provides the resource-sharing and security-isolation building blocks to implement containers. It's userland utilities like LXC that assemble the necessary blocks together to provide what administrators recognize as containers. So, let's talk non-virtualization-based resource- and security-isolation. Workload Manager (WLM) [1] has been a part of IBM z/OS since before it was even called z/OS. WLM implemented something like cgroups scheduling, in that existing utilization samples feed into future scheduling to ensure responsiveness and fair access to resources under contention. The Resource Access Control Facility (RACF) has mapped allowed program access to resources since 1985 [2], which functions similarly to modern mandatory access control. > I'd go as far as to say that there is no such thing as a mature container technology. The fundamental problem of containers is that mainstreams kernels are not designed to be multi-tenant. It's your turn to provide some citations. You could argue that the Linux kernel has failed to achieve good multi-tenancy support, but saying it hasn't been designed for multi-tenancy just makes you look ignorant of the last five years of development on namespaces, cgroups, and general multiprocess resource scheduling. > They support multiple users quite well but when you try to have multiple root users, badness ensues. Containers aren't about having multiple root users, even though that's technically possible if you opt to use Linux UID namespaces, which aren't mandatory. > I'm quite sure it's not faster than virtualization. Citation needed, especially since you're arguing that more layers of abstraction (kernel + hypervisor + kernel) is at or above the same speed of fewer (kernel + cgroups/namespaces). [1] http://en.wikipedia.org/wiki/Workload_Manager http://en.wikipedia.org/wiki/Workload_Manager [2] http://www-03.ibm.com/systems/z/os/zos/features/racf/racfhist.html http://www-03.ibm.com/systems/z/os/zos/features/racf/racfhis...
gwgarry 13y ago
> The recent (say, last decade) disparity has been lack of mature containers on the platform of choice, which is Linux for most cloud-based projects. I don't know many people who would switch their projects to run on FreeBSD or Solaris just to use containers instead of VMs. OpenVZ. VServer. Many people did that in the old days. Before Xen it was all about FreeBSD.
amscanne 13y ago
> Hi, I'm the author of the article. Awesome! Although I don't agree with the thesis, any systems article is a great article :) > Also, the container types you mention don't support independent, fine-grained choices of isolate-or-not for networking, the file system, user IDs, process IDs, and scheduling of system resources. Granted, containers have gotten better over time. My beef isn't with containers, just that they're often compared to virtualization as if they're a substitute or replacement. I really disagree with that. Containers are still one kernel, with a huge attack surface. Forget about other OSes (including other versions of linux -- system calls do change over time, e.g. -- epoll, inotify, etc.). It's still a single buffer cache, so policing memory (beyond RSS) and disk I/O is tricky. It's one thing for a platform like Heroku, but containers don't make sense for a general purpose, multi-tenant infrastructure. > It's unclear why you think such optimization would appear in hypervisors but not kernels, especially given how much more insight a kernel has into the workloads it directly runs compared to a hypervisor running a kernel running workloads. Re: The long-term performance of virtualization. It's a different abstraction which I believe lends itself better to using all the resources of large physical machines. We're talking future here (so to be sure, we'll just have to wait and see), but as a philosophy it makes sense to me. The same way we hit the limit on cranking the frequency of CPUs and eventually starting adding cores, a single operating system won't scale linearly forever. (Not that Linux isn't doing a laudable job). The optimizations I mentioned specifically apply to the use of virt hardware (i.e. the VT extensions effectively give you multiple TLBs because they are tagged) because there are different set of semantics associated with how those threads of execution run on the hardware. The restricted physical interface presented to VMs can free the hardware by shifting some low-level problems of scale and co-ordination into software (effectively achieving performance by forcing a more scalable software architecture). > Even if hypervisors achieve CPU performance parity for running systems, you still haven't addressed the memory and storage overhead (in terms of resident footprint) of running full OS images instead of containers. There are techniques to reduce these overheads, but you've got a good point -- there is a price to pay for full virtualization. But fundamentally, I think this is part of the trade-off of better isolation. A common buffer cache in Linux will improve performance of containers, but it becomes very difficult or impossible to enforce exact resource policies. > You're portraying virtualization as having benefits versus containerization while ignoring that containerization also has many benefits over virtualization, especially w.r.t. cost efficiency. I acknowledge that containers can be more efficient -- I just don't think it justifies giving up the benefits that full virtualization provides. I also think that virtualization is valuable because it enables some additional technologies, but that's a different story... Ultimately, I think it boils down to the fact that I just don't think containers and VMs are comparable. I don't see how containers could displace virtualization for infrastructure use cases beyond a platform for specific applications. The EC2+Heroku model makes perfect sense -- some people will run containers inside VMs on a general purpose infrastructure, some won't.

aliguori 13y ago

Hi David, > Mature containers have been around since the the days of mainframes. Citation needed. I'd go as far as to say that there is no such thing as a mature container technology. The fundamental problem of containers is that mainstreams kernels are not designed to be multi-tenant. They support multiple users quite well but when you try to have multiple root users, badness ensues. Even the best container systems today for Linux still have fundamental gaps. > Even if hypervisors achieve CPU performance parity for running systems, you still haven't addressed the memory and storage overhead (in terms of resident footprint) of running full OS images instead of containers. Re: memory, same page merging can eliminate a lot of that overhead when running mostly homogenous workloads. Re: storage using CoW not only makes provisioning instant (not 5-10 minutes) but also addresses the storage concern. If you mean the overhead of running two kernels, well, actually using namespace has a fair amount of practical overhead too. Of course, benchmarks speak louder than words here. No one has published a container based result for SPECvirt because I'm quite sure it's not faster than virtualization.

justincormack 13y ago
Linux containers are pretty immature, they only started being implemented recently; mainframes, Solaris, FreeBSD are much older. The Linux implementation is interesting though in terms of granularity.
- makomk 13y ago
  If you ignore stuff like OpenVZ, maybe, but there's probably a reason why that was mostly displaced by full virtualization in the first place.
davidstrauss 13y ago
> Citation needed. First, I define containers by their capabilities, not a vendor or kernel calling them "containers." The Linux kernel, internally, has no concept of containers; it merely provides the resource-sharing and security-isolation building blocks to implement containers. It's userland utilities like LXC that assemble the necessary blocks together to provide what administrators recognize as containers. So, let's talk non-virtualization-based resource- and security-isolation. Workload Manager (WLM) [1] has been a part of IBM z/OS since before it was even called z/OS. WLM implemented something like cgroups scheduling, in that existing utilization samples feed into future scheduling to ensure responsiveness and fair access to resources under contention. The Resource Access Control Facility (RACF) has mapped allowed program access to resources since 1985 [2], which functions similarly to modern mandatory access control. > I'd go as far as to say that there is no such thing as a mature container technology. The fundamental problem of containers is that mainstreams kernels are not designed to be multi-tenant. It's your turn to provide some citations. You could argue that the Linux kernel has failed to achieve good multi-tenancy support, but saying it hasn't been designed for multi-tenancy just makes you look ignorant of the last five years of development on namespaces, cgroups, and general multiprocess resource scheduling. > They support multiple users quite well but when you try to have multiple root users, badness ensues. Containers aren't about having multiple root users, even though that's technically possible if you opt to use Linux UID namespaces, which aren't mandatory. > I'm quite sure it's not faster than virtualization. Citation needed, especially since you're arguing that more layers of abstraction (kernel + hypervisor + kernel) is at or above the same speed of fewer (kernel + cgroups/namespaces). [1] http://en.wikipedia.org/wiki/Workload_Manager http://en.wikipedia.org/wiki/Workload_Manager [2] http://www-03.ibm.com/systems/z/os/zos/features/racf/racfhist.html http://www-03.ibm.com/systems/z/os/zos/features/racf/racfhis...

gwgarry 13y ago

> The recent (say, last decade) disparity has been lack of mature containers on the platform of choice, which is Linux for most cloud-based projects. I don't know many people who would switch their projects to run on FreeBSD or Solaris just to use containers instead of VMs. OpenVZ. VServer. Many people did that in the old days. Before Xen it was all about FreeBSD.

amscanne 13y ago

> Hi, I'm the author of the article. Awesome! Although I don't agree with the thesis, any systems article is a great article :) > Also, the container types you mention don't support independent, fine-grained choices of isolate-or-not for networking, the file system, user IDs, process IDs, and scheduling of system resources. Granted, containers have gotten better over time. My beef isn't with containers, just that they're often compared to virtualization as if they're a substitute or replacement. I really disagree with that. Containers are still one kernel, with a huge attack surface. Forget about other OSes (including other versions of linux -- system calls do change over time, e.g. -- epoll, inotify, etc.). It's still a single buffer cache, so policing memory (beyond RSS) and disk I/O is tricky. It's one thing for a platform like Heroku, but containers don't make sense for a general purpose, multi-tenant infrastructure. > It's unclear why you think such optimization would appear in hypervisors but not kernels, especially given how much more insight a kernel has into the workloads it directly runs compared to a hypervisor running a kernel running workloads. Re: The long-term performance of virtualization. It's a different abstraction which I believe lends itself better to using all the resources of large physical machines. We're talking future here (so to be sure, we'll just have to wait and see), but as a philosophy it makes sense to me. The same way we hit the limit on cranking the frequency of CPUs and eventually starting adding cores, a single operating system won't scale linearly forever. (Not that Linux isn't doing a laudable job). The optimizations I mentioned specifically apply to the use of virt hardware (i.e. the VT extensions effectively give you multiple TLBs because they are tagged) because there are different set of semantics associated with how those threads of execution run on the hardware. The restricted physical interface presented to VMs can free the hardware by shifting some low-level problems of scale and co-ordination into software (effectively achieving performance by forcing a more scalable software architecture). > Even if hypervisors achieve CPU performance parity for running systems, you still haven't addressed the memory and storage overhead (in terms of resident footprint) of running full OS images instead of containers. There are techniques to reduce these overheads, but you've got a good point -- there is a price to pay for full virtualization. But fundamentally, I think this is part of the trade-off of better isolation. A common buffer cache in Linux will improve performance of containers, but it becomes very difficult or impossible to enforce exact resource policies. > You're portraying virtualization as having benefits versus containerization while ignoring that containerization also has many benefits over virtualization, especially w.r.t. cost efficiency. I acknowledge that containers can be more efficient -- I just don't think it justifies giving up the benefits that full virtualization provides. I also think that virtualization is valuable because it enables some additional technologies, but that's a different story... Ultimately, I think it boils down to the fact that I just don't think containers and VMs are comparable. I don't see how containers could displace virtualization for infrastructure use cases beyond a platform for specific applications. The EC2+Heroku model makes perfect sense -- some people will run containers inside VMs on a general purpose infrastructure, some won't.