19 ms·

Containers, Not Virtual Machines, Are the Future Cloud

228 points by dave1010uk 13y ago

api 13y ago
Where I work we use Parallels Virtuozzo for a private cloud. I can see why more companies don't use it for public clouds-- you do run into a few weird edge cases with containers vs. full VMs and these would be a tech support issue for a large-scale hosted. But the density you achieve with it is at least one order of magnitude better than KVM, possibly two, and the performance is also better. There is zero I/O overhead since you are in fact running bare-metal, so it's very good for high-performance computing (one of our use cases). Most of the edge cases you do encounter are due to poorly-written software. The most common thing is server software that depends on Linux's default overcommit behavior and malloc() never failing and then uses that to try to badly implement a memory-sensitive cache in competition with the kernel. Containers change overcommit behavior and malloc() CAN fail, so such software will eat all the memory in a container and barf. But there are hacks to get around this, and it's mostly an issue with POS rube goldberg machine web stacks like Apache/mod_php. I've been following LXC for a while and see that some people are now using it in production. Once Linux has a main-line kernel container solution I think you'll see it used quite a bit more. Eventually the developers of crappy server code will be forced to fix their code to behave properly.
- sedachv 13y ago
  > But there are hacks to get around this, and it's mostly an issue with POS rube goldberg machine web stacks like Apache/mod_php. It's a common technique used by a lot of virtual machines and memory management code to get a large contiguous region of address space to work with (most JVMs work this way, SBCL does too). I'm not exactly sure why Xen can handle overcommit fine and Virtuozzo can't, the latter does need a custom kernel so in principle they have access to all the low-level virtual memory stuff.
  - api 13y ago
    JVM's always worked fine for us in Virtuozzo. PHP is what we have issues with, but we've found workarounds.
ebbv 13y ago
Container virtualization certainly has its uses, and we will see hosting products based on it; no question. But to say that it is "the future" and virtual machines are on their way out is a naive view. There's plenty of reasons why someone might want a virtualized OS; customization. What if my application is built to run on Debian but yours is built to run on CentOS? With container based virtualization we have to be put on different nodes. With VMs, we can share a node. This is why VMs exist. This is why they will continue to exist. Disclaimer; I work in the hosting industry.
- justincormack 13y ago
  You can run debian containers in a centOS host, so long as they can share the same kernel, which Linux distros mostly can now, as we dont see the degree of distro patching we used to at one point.
  - ebbv 13y ago
    It was just a simplistic example. The point remains the same; there are use cases where people want an OS image that has a different kernel.
    - viraptor 13y ago
      There's a question of "which people?" Sure, there are some cases where people will want their own OSes / images / kernels. But there's also a huge group of "people" who would prefer to make the whole issue of "image" disappear. They're more happy to deploy to heroku / dotcloud / something similar where they can have the NoOps environment. And for those providers secure containers will have a much lower overhead than full blown virtual machines.
      - ebbv 13y ago
        Yes this is true and I said that we will definitely see hosting products based on containers. I'm not disputing that point. You know what the most popular hosting product is right now? It's not VPSes. It's shared hosting. Good old simple shared hosting outsells VPSes by a huge margin. There's plenty of room for a variety of products, all the way from shared, to VPS, to container based to dedicated servers. What I took issue with was the original article saying that containers would make VMs a thing of the past, which is silly.
Nux 13y ago
Containers certainly have their uses, but what I think will happen is hardware will become better and better at virtualisation just as the (K)VM software will become better. LXC is good for stuff like OpenShift (PAAS), but for "VPS" offerings the VM will continue to reign supreme.
- zb 13y ago
  Actually, OpenShift doesn't use LXC (it uses SELinux to separate gears) because containers are not secure in the way that VMs are.
  - lil_cain 13y ago
    In the near future, it'll use kernel namespacing (like LXC) and SELinux.
- alinspired 13y ago
  I agree about VMs for VPS offering. But i believe no improvements in hardware or (para)virtualization will get VMs closer to Containers in other (non-VPS) scenarios
deleted 13y ago
[deleted]
- sciurus 13y ago
  How do VMs "abstract finance" any more than containers?
  - deleted 13y ago
    [deleted]
    - deleted 13y ago
      [deleted]
susi22 13y ago
I'm surprised the article doesn't mention SmartOS. It's a superb OS which let's you spin up zones and KVMs. http://smartos.blueprint.org/home/why-smartos-in-my-lab http://smartos.blueprint.org/home/why-smartos-in-my-lab http://dtrace.org/blogs/brendan/2013/01/11/virtualization-performance-zones-kvm-xen/ http://dtrace.org/blogs/brendan/2013/01/11/virtualization-pe... http://www.cloudcomp.ch/2013/04/openstack-on-smartos/ http://www.cloudcomp.ch/2013/04/openstack-on-smartos/
- wmf 13y ago
  I wouldn't expect much Solaris[1] coverage in Linux Journal. [1] Illumos/SmartOS proponents seem to go out of their way to avoid saying the S-word.
  - helper 13y ago
    Illumos has KVM support, Solaris does not. The distinction is made because Solaris and Illumos have diverged into different operating systems.
j_s 13y ago
Is it possible to have the same degree of security with a container as is provided by a virtual machine?
- susi22 13y ago
  Yes, the security is there. Though, resource fairness is not (since you have basically bare metal access). SmartOS is trying to fix this: http://wiki.smartos.org/display/DOC/Tuning+the+IO+Throttle http://wiki.smartos.org/display/DOC/Tuning+the+IO+Throttle http://dtrace.org/blogs/brendan/2012/12/19/the-use-method-smartos-performance-checklist/ http://dtrace.org/blogs/brendan/2012/12/19/the-use-method-sm...
  - wmf 13y ago
    Linux now has fair sharing for CPU, disk, and network if you can figure out how to configure it. Or you can use cgroups to guarantee resources to make containers act more like VMs.
    - davidstrauss 13y ago
      cgroups are the method of doing fair resource sharing.
  - Estragon 13y ago
    How can the security be comparable? I would have thought that the kernel shared between containerized processes presents a much bigger attack surface than a virtual machine monitor does.
  - chubot 13y ago
    That's not true -- Linux has had local root exploits pretty much at ALL times. If you're in a container, you can break out with one of these exploits. You'll be root and have access to all other containers on the machine. In a VM, you don't share a kernel with other users, so the local root exploit doesn't buy you anything. I prefer containers and think VMs are a huge hack for most problems. But as it stands now, the number of local root exploits vs say remote exploits on Linux is probably greater by a factor of 100 or so, so you're taking much more risk. Security is one of the problems with Linux containers that still needs to be resolved. (see http://mattoncloud.org/2012/07/16/are-lxc-containers-enough/ http://mattoncloud.org/2012/07/16/are-lxc-containers-enough/, Google "lxc security")
amirmc 13y ago
Folks interested in this should also take note of the projects that target Xen [1,2 are ones I'm aware of]. Startup times that can be tuned to be on the order of 10s of milliseconds (see comment in Mirage paper). [1] Mirage - http://openmirage.org http://openmirage.org (paper at http://anil.recoil.org/papers/2013-asplos-mirage.pdf http://anil.recoil.org/papers/2013-asplos-mirage.pdf) [2] Zerg - http://zerg.erlangonxen.org http://zerg.erlangonxen.org
- marshray 13y ago
  I agree, these things are the future. Containers may be useful for single-tenant partitioning, but no one has managed to produce an OS kernel without an order of magnitude more local root security vulnerabilities than the HVM or PV virutualization systems.
  - contingencies 13y ago
    Honestly, on Linux we can reach a similar point now by careful application of existing tools. For example, by whitelisting syscalls, devices, hardware and network resources and access patterns, in combination with containers. Most of the challenge is automating that process adequately, which is really nothing more than traditional code profiling meets existing policy-based kernel-level security toolkits meets containers meets development processes (ie. build/test/release). After ~3 years' with LXC, I believe that most of the attack surfaces are reasonably well understood and/or not required for most code to function. Sure, there will be kernel-level vulnerabilities, but they won't be exposed to such a broad range of userland applications. With security you also get so much more insight, power and predictability with regards to resource profiling than with paravirtualized solutions. Real kernel-level metrics versus load! Think of the impact for operations/capacity planning. So... short soundbite to summarize? Docker is a toy compared to where we are going.
    - shykes 13y ago
      Hi, I'm the author of Docker. I agree with everything you said about lxc, but I'm not sure how you reached the conclusion that docker is a toy. Since Docker is a portability layer on top of lxc, it automatically benefits from all the awesome work happening in the kernel. Or do you just use "toy" as a synonymous for "userland"?
aetimmes 13y ago
Solaris/illumos have been running containers for almost ten years now. They're just a layer of abstraction on the other side of the OS layer - a different tool, not necessarily a better one.
lsc 13y ago
Having sold both containers and VMs to untrusted users? I disagree. Now, my container experience is old, and containerization technology is improving all the time, so over time, what I say here becomes less true, but that's the thing; I'm willing to pay a fairly high cost in (cheap) ram for relatively small gains in compartmentalization. (which is to say, I'm willing to spend a fair bit of money to make my pager go off less.) See, the expensive part? it's not the ram, it's sysadmin time. That's why VPSs are killing shared hosting; it's actually easier to manage a bunch of VPSs than a bunch of people with shell accounts on the same box. Same goes for all the containerization technologies I've tried; it's easier to just throw xen/kvm on there and pay for the extra ram than to manage one user stepping on resources required by another user. For me? that was the beautiful part of switching from jails to Xen; suddenly, I could just ignore customers who tried to burn all the cpu and ram they could. Now, if I can figure out how to do the same with disk? I'll be a very happy man. That, at least, is where I'm putting my money; in more ram.
- jauer 13y ago
  SmartOS has disk IO throttling for KVM guests.
  - lsc 13y ago
    and you can throttle with ionice on xen running a Linux dom0[1]. It works pretty great if you have some user who is going overboard and you want to make it suck for just them and suck less for everyone else. It doesn't work like the CPU scheduler; which is to say, I can't set ionice on all the guests and expect good performance on the light-use guests while a heavy-use guest is thrashing the disk. (I can expect the good performance for the light-use guests when the resource in contention is CPU time.) It's possible that either ionice isn't a good enough tool, or that I don't know enough about it... but I think it's far more likely that the variable nature of disk throughput makes balancing requests harder... (e.g. how many IOPS a second I get varies a lot from second to second, based on where the sectors I want to operate on are in relation to the read/write head. How much disk bandwidth I get varies even more.) I am pretty curious about how much this situation will change with SSD storage. [1]http://wiki.prgmr.com/mediawiki/index.php/Chapter_7:_Hosting_Untrusted_Users_Under_Xen:_Lessons_from_the_Trenches#Regulating_Disk_Access_with_ionice http://wiki.prgmr.com/mediawiki/index.php/Chapter_7:_Hosting...
CptCodeMonkey 13y ago
Just got a client off a container based provider and it wasn't too great an experience ( dramatically slow networking, unreliable file I/O ). Granted I could, and can continue to, say the same thing about virtualization but virtualization is here and reliable enough that it seems like a hard sell to want to try something that is even less performant.
fridder 13y ago
Also see FreeBSD Jails:
- davidstrauss 13y ago
  Hi, I'm the author of the article. FreeBSD Jails lack the same fine-grained isolation choices versus the base system that the Linux kernel exposes through namespaces and cgroups. That's not to say that Jails don't capture most of the value I argue containers have, but they're a different generation of design. Also, I wrote the article for the Linux Journal, which obviously affects the solutions explored.
  - oijaf888 13y ago
    What can cgroups and namespaces do that rctl and capsicum can't? I assume with namespaces you can have multiple processes running with the same PID?
    - davidstrauss 13y ago
      RCTL can enforce specific limits, which is good if you either want to divide resources such that there can't be (or is unlikely to be) contention. cgroups offers hard limits for some things, like memory, but it mostly opts for a model using "shares" that determine the fractional access to resources versus other cgroups holding shares against the same resource. For example, assume there's CPU contention. cgroup A has 10 CPU shares and cgroup B has 90. Processes in cgroup A will get 90% of the CPU time, but it will not starve cgroup A because cgroup A will still get 10%. This shares-based model also has a major effect when there isn't contention. Shares-based resources are burstable. Even cgroup A (with 10 CPU shares) can use 100% of CPU if nothing else needs it. This "burstable" nature can be good or bad. It's good in the sense that most users will probably get more CPU than their shares guarantee most of the time. It's bad because users can start expecting more than their shares guarantee and get a nasty surprise when resources get under contention. It's time to drop some analogies. cgroups are very much like a highway with an HOV lane (or more): anyone can go very fast when there's no contention. But, during rush hour, lanes get distributed as "shares" of the road to the HOV and non-HOV groups. Neither the HOV nor the non-HOV drivers get starved for road access (though responsiveness may not be equivalent, by design). Traditional "nice" is like emergency vehicle traffic. An ambulance every now and then works fine as "-20 nice" traffic. But, if you filled the road with ambulances, it would starve normal traffic of roadway access. RCTL is sort of like a person riding reserved right-of-way public transit. From the time the person hops on the train at point A to when they get off at point B, it will be the same duration any time of day. They don't get to go faster during low-traffic times, but they also don't have to worry about a significantly worse experience during rush hour. Capsicum seems focused on intra-application isolation; I'm not sure how to compare it to other OS-level containers.
ewams 13y ago
Good article that discusses something I have not touched in years, simply because of a lot of the drawbacks he talks about. Also is a wonderful sales pitch, someone has been practicing. In the virtualization world we argue often about how to do things “right” and what the “industry standards” are. I started a series a while back to help define and explain virtualization and various related topics. Specifically I even talk about this subject ( http://ewams.net/?date=2012/01/04&view=The_4_Types_of_Virtualization#b http://ewams.net/?date=2012/01/04&view=The_4_Types_of_Virtua... ). The main problem with saying VM’s are “no longer the future” is they are mainstream and have been for a while, so they probably are not not cool enough for hackers but they are not going anywhere fast. Containers have a place and full blown virtual machines have a place. The trick is finding which one will benefit your organization and for which purpose. Most likely both can work depending on the requirements. The other topic I notice that is usually overlooked when talking about virtualization is the actual OS. Here in this article it discusses mostly Red Hat based distro’s. Some organizations strictly use OSX, others Ubuntu, others RHEL, while others use some sort of mix. Each is different and even for closely related like Fedora and RHEL, what works on one may not work on another without hacking it or some other black magic. (Or it may work just fine, you know how it goes). The other point of the OS talk is where is Windows? We can pretend to be ostriches but Windows server is out there and in large numbers. Lots of people make lots of money writing applications that work strictly on Windows (same for *nix). Just do not forget about it. If we move forward with containers, and since the topic has come up at least twice in the last 6 months, let’s start to see recommendations on how to run containers. How to pick workloads. How to optimize applications and kernels to run them. What kind of hardware to run them on. How to code for them. Etc.
toolslive 13y ago
IMNSHO: the future should be both VM-less and container-less. meanwhile, containers are more efficient.
- davidstrauss 13y ago
  How would you suggest we efficiently partition and sell computing resources, then? It's not cost-effective or even power-efficient to run separate hardware for each scale of computing. For example, many projects only need a container or VM with 512MB of memory and one CPU core.
  - VLM 13y ago
    Can't speak for the original poster but my guess is the obvious drop in power/cost required for computing. "Just wanna email, word process, and some web and some gaming" has relatively recently gone from a kilobuck P4 couple hundred watt space heater to a hundredbuck tablet that needs charging every couple days. Unless you all can find a way to make silicon more expensive or dramatically increase processing demand, all these elaborate schemes to dole out processor are doomed to something like the per minute long distance business model where eventually more money was being spent on detailed billing than on actually providing the service... When the TCO of an elaborate virtualization system and its provisioning and billing and firewalling system exceeds the TCO of just spinning up a ubiquitous cheap low power system on a chip...
AndrewGaspar 13y ago
It's pretty bad they don't even mention Windows Azure - their Web Sites service is already using containers, though only on the free tier for right now.
- davidstrauss 13y ago
  I'm the author of the article. It doesn't mention Windows Azure because I wrote it for the Linux Journal, and a full discussion of containerization outside of Linux wasn't possible in the allotted article space.
amscanne 13y ago
Containers have been mature for longer than the article implies (freebsd jails, Solaris zones). Technology just doesn't work this way. Things don't simpler over time (and you don't shed support for different kernels, OSes, better isolation, etc.) Not to mention the fact that virtualization has gotten dramatically better since EC2 was launched, since that seems to be the benchmark here (they are still running Xen 3.X first released in 2005). Modern virt hardware+hypervisors is blazing fast and (I believe) will eventually outpace bare metal for specific workloads on big boxes due to the improved isolation for many performance critical operations (I.e. separate TLBs for flushes, smarter cache invalidation). Even if they don't reach parity, the gap certainty wouldn't justify giving up all the benefits of full virtualization.
- davidstrauss 13y ago
  Hi, I'm the author of the article. > Containers have been mature for longer than the article implies (freebsd jails, Solaris zones). Mature containers have been around since the the days of mainframes. The recent (say, last decade) disparity has been lack of mature containers on the platform of choice, which is Linux for most cloud-based projects. I don't know many people who would switch their projects to run on FreeBSD or Solaris just to use containers instead of VMs. Also, the container types you mention don't support independent, fine-grained choices of isolate-or-not for networking, the file system, user IDs, process IDs, and scheduling of system resources. > Modern virt hardware+hypervisors is blazing fast and (I believe) will eventually outpace bare metal for specific workloads on big boxes due to the improved isolation for many performance critical operations (I.e. separate TLBs for flushes, smarter cache invalidation). It's unclear why you think such optimization would appear in hypervisors but not kernels, especially given how much more insight a kernel has into the workloads it directly runs compared to a hypervisor running a kernel running workloads. Even if hypervisors achieve CPU performance parity for running systems, you still haven't addressed the memory and storage overhead (in terms of resident footprint) of running full OS images instead of containers. > Even if they don't reach parity, the gap certainty wouldn't justify giving up all the benefits of full virtualization. You're portraying virtualization as having benefits versus containerization while ignoring that containerization also has many benefits over virtualization, especially w.r.t. cost efficiency.
ausjke 13y ago
LXC will never replace XEN/KVM/whatever. LXC has its places, but saying it is the future for all VMs is just naive. Since it's from linuxjournal I want to check who dares to make such bold statements, then realize this is probably the real purpose of that article, a twisted ads for its Drupal business? In short, the article is pointless and shallow, and you can't stop checking who is making such claims, and then you find yet another drupal hosting site, this sucks.
alinspired 13y ago
Containers are efficient(and most likely the future) for PaaS/SaaS type of applications, where individual Instance (VM/Container) is not treated as a "computer" or Infrastructure. Another benefit for PaaS/SaaS is that you don't necessarily sell RAM or DISK, you sell an application/service and thus can take FULL advantage of containers, like: - more efficient resources sharing(on host) - dynamic resources re-balancing - quicker start/stop/etc Containers can be used to offer IaaS, but as was already mentioned in this thread, a few edge cases (custom kernel or kernel module, any filesystems support, any firewall support) prevent it due to support/operations overhead.
- gundy 13y ago
  They are not really efficient. All they do is add an extra layer of security so you can encapsulate that specific process in a chroot with its own networking stack. Other than that the CPU is still fairshared and the memory still has the same limits as if you ran it without the container. It adds nothing but security.
  - wmf 13y ago
    In the cloud, KVM/Xen overhead is considered the baseline so containers are definitely more efficient by comparison.
  - alinspired 13y ago
    There are container technologies (openvz, LXC) that offer lot more than "fairshared" to manage resources between containers. Plus the efficiencies of lower "virtualization overhead", because you don't have full OS in containers compared to VMs (consider full VM kernel, VM HW drivers, etc)
j45 13y ago
Three thoughts: 1) I'm not sure if this article has accounted for the one massive achilles heel of paravirtualization (containers): Corrupting the core OS by way of upgrade or patches, can take all containers all out. 2) I have used VMware since the early betas in 98. The motivation was simple: Replacing servers sucks. The day you buy or lease a server, in essence, you purchase it's death and dealing with a hasty or painful migration. 3) From my 7th or so generation of server migrations in doing complex hosting since 95, one thing is clear: The single most valuable thing is a hard separation between the bare metal and the OS. So much so that you can move the VMs at will anywhere, because, you just bought the death of your server the day it's brand new. I ran a CentOS box with a BSD VM as a firewall, with Windows and Linux VM's behind it to serve one project because each did their unique and highly valuable part. Ultimately, for me: Paravirtualization not only uses the host os, but shares it between multiple configurations. This is in my mind still introduces risks that should be minimized, if not avoided. I'm sure PV is improving all the time. It's simply newer to the game than what VMware started. If VMWare isn't for you, suites like ProxMox are serious options to consider. Hosting isn't somewhere to have preferences and opinions. If I'm wrong, I'll change at any moment. Issues with PV might be decreasing, but I've been ultimately burnt by PV every time I used it in production for a prolonged time. An example of where sysadmin time cost way more than a little bit of more ram for Virtualization: Parallels Virtuozzo, corrupted itself so badly during an update to the core OS that all the containers on the server were rendered inoperable. Parallels support had one of their high level techs login from Russia to manually install corrupted DLL files so the server would at least boot, and we could get the heck off it. At the end of the repair we were told the VM was no longer upgradeable and all updates would have to be done manually. The customer had taken on PV at the behest of the hosting company who insisted it was amazing despite my concerns. Said hosting company also now has switched over entirely to VMware. The unforgivable kicker of PV for my customer: tons of customization and creating re-usable containers were all lost when leaving, because moving to something like VMware meant having to set them up from scratch again.
- alinspired 13y ago
  Your comment must be entirely about Containers for Windows (by Parallels), which is not what the article and most of the discussions are about. Containers for Linux on the contrary are not prone to host "corruption"
thelarry 13y ago
Linux containers have been awesome for helping me sandbox user generate code. Has anyone actually had experience with using Mesos in the real world?
- gundy 13y ago
  That's all that they are good for adding security that would otherwise not exist. Outside of that under a PaaS, I don't see what they add.
davidgerard 13y ago
This is totally the case. Back before it got Oracled, Solaris 10 zones were one of the finest things ever - and completely controllable in resource usage, with a few percent CPU overhead at most (for comparison, running ZFS takes more). Zones for Linux would win, comprehensively.
gundy 13y ago
In other news OPs startup not the future of PaaS but OpenShift, Redhat's open source alternative is.
cornet 13y ago
Containers are zones are great if you're building your own platform that you control. However I'm not sure building a multi-tenancy hosting environment based on containers or zones is the best idea. Suppose the tun driver has a bug in it which can cause a kernel panic under some circumstances and that someone running OpenVPN in a container or zone hits this condition. You can now wave bye bye to all the zones running on that host. https://github.com/joyent/illumos-extra/commit/9412039a18f2f52b24b18ad8c8642a55d3b50d93 https://github.com/joyent/illumos-extra/commit/9412039a18f2f...
gnel 13y ago
I currently have a OpenVZ vps on the internet to host my projects and I chose it for its better performance in comparison to a real VM (xen,kvm,etc) but my what I'm still to find it's a provider that offers OpenVZ containers (or virtuozzo containers) that can scale programatically, like the VM's on Amazon EC2. I think it should be possible, that I could program if my container cpu load is over 70% for more than 40 seconds to launch a new copy of my container in the same or other machine.. This lack of a service to scale containers it's the first thing that needs to be done to compete with common VM clouds.
csense 13y ago
What's ironic is that the entire point of an OS that supports multiple processes is to provide them each with an illusion that they're totally in control of their own machine. This abstraction was kind of leaky -- different processes typically share various namespaces, like filesystems, process tables and user accounts. As a result, virtualizing an entire OS became a way to make a tight abstraction of having an independent machine at your disposal. The reason virtualization happened was because emulating the system hardware was easier than changing every OS to adequately separate their namespaces. Now that virtualization has the attention of OS developers, you have development like LXC / cgroup stuff in Linux, bringing isolation into the OS, where it should have existed all along. AFAIK the mainframe world -- which, by the way, still exists -- has never gotten out of the early-computing mindset of having an expensive machine which must be shared between applications, so isolation and containerization has always been a feature of mainframe OS's.
grogers 13y ago
Maybe, but the linux kernel has had its share of local privilege escalation bugs. If I were a cloud provider (or user) I'd be hesitant to trust containers for hard isolation.
jpollock 13y ago
Sun/Oracle have been championing containers (Zones) for a long time now. They have their places. However, businesses still want full virtualised systems (LDOMs). Businesses like treating applications like appliances. They want to start up a system and not touch it again. Migrating an application to a new OS release costs them a lot of money. A zone forces an upgrade - everything on the machine needs to be rolled forward to the new release. Since businesses typically have a patchwork of software in various levels of support, continuous upgrades can be an expensive (and possibly impossible) task. For example, the application vendor may require a license purchase in order to get support for the new OS release. The vendor may even have gone out of business, and the replacement is still 6-12 months away. In a full virtualised platform, you're not forced to continually pay that maintenance cost. You are free to store it as technical debt and pay it in a more managed fashion.
- wmf 13y ago
  But both Solaris and Linux let you run old userspace in a container on a newer kernel.
  - jpollock 13y ago
    Actually, there are limits, as I was informed in the Solaris 11 launch. I didn't pay too much attention because it didn't affect me, but the guys it did affect weren't that impressed!
mark_l_watson 13y ago
A question: which hosting companies sell container hosting services for specific platforms like Sintra+Postgres, Java+Hybernate+Mysql, etc? If I understand this correctly, you would want to be able to rent container slices with the specific stack you are going to use, already set up.
- wmf 13y ago
  If you want a container with a specific stack that's usually called PaaS. Personally I hope that tools like Docker allow some companies to focus on hosting containers (hopefully not inside Xen VMs...) and others to focus on building great stacks and container templates.
kleiba 13y ago
A short (5min) talk by Solomon Hykes of dotCloud about how their "docker" tool makes use of containers: https://www.youtube.com/watch?v=9xciauwbsuo https://www.youtube.com/watch?v=9xciauwbsuo
bmullan 13y ago
One future for LXC is with ARM based servers such as Calxeda. Approx 800 core with gigabit Ethernet fabric & I/O. And sata built-in for each core. About size of a Cisco catalyst 5000 box but less than 1000 watts. Runs ubuntu or Red Hat and uses LXC for virtualization. Last I saw it was <$100k

228 points by dave1010uk 13y ago

api 13y ago

Where I work we use Parallels Virtuozzo for a private cloud. I can see why more companies don't use it for public clouds-- you do run into a few weird edge cases with containers vs. full VMs and these would be a tech support issue for a large-scale hosted. But the density you achieve with it is at least one order of magnitude better than KVM, possibly two, and the performance is also better. There is zero I/O overhead since you are in fact running bare-metal, so it's very good for high-performance computing (one of our use cases). Most of the edge cases you do encounter are due to poorly-written software. The most common thing is server software that depends on Linux's default overcommit behavior and malloc() never failing and then uses that to try to badly implement a memory-sensitive cache in competition with the kernel. Containers change overcommit behavior and malloc() CAN fail, so such software will eat all the memory in a container and barf. But there are hacks to get around this, and it's mostly an issue with POS rube goldberg machine web stacks like Apache/mod_php. I've been following LXC for a while and see that some people are now using it in production. Once Linux has a main-line kernel container solution I think you'll see it used quite a bit more. Eventually the developers of crappy server code will be forced to fix their code to behave properly.

sedachv 13y ago
> But there are hacks to get around this, and it's mostly an issue with POS rube goldberg machine web stacks like Apache/mod_php. It's a common technique used by a lot of virtual machines and memory management code to get a large contiguous region of address space to work with (most JVMs work this way, SBCL does too). I'm not exactly sure why Xen can handle overcommit fine and Virtuozzo can't, the latter does need a custom kernel so in principle they have access to all the low-level virtual memory stuff.
- api 13y ago
  JVM's always worked fine for us in Virtuozzo. PHP is what we have issues with, but we've found workarounds.

ebbv 13y ago

Container virtualization certainly has its uses, and we will see hosting products based on it; no question. But to say that it is "the future" and virtual machines are on their way out is a naive view. There's plenty of reasons why someone might want a virtualized OS; customization. What if my application is built to run on Debian but yours is built to run on CentOS? With container based virtualization we have to be put on different nodes. With VMs, we can share a node. This is why VMs exist. This is why they will continue to exist. Disclaimer; I work in the hosting industry.

justincormack 13y ago
You can run debian containers in a centOS host, so long as they can share the same kernel, which Linux distros mostly can now, as we dont see the degree of distro patching we used to at one point.
- ebbv 13y ago
  It was just a simplistic example. The point remains the same; there are use cases where people want an OS image that has a different kernel.
  - viraptor 13y ago
    There's a question of "which people?" Sure, there are some cases where people will want their own OSes / images / kernels. But there's also a huge group of "people" who would prefer to make the whole issue of "image" disappear. They're more happy to deploy to heroku / dotcloud / something similar where they can have the NoOps environment. And for those providers secure containers will have a much lower overhead than full blown virtual machines.
    - ebbv 13y ago
      Yes this is true and I said that we will definitely see hosting products based on containers. I'm not disputing that point. You know what the most popular hosting product is right now? It's not VPSes. It's shared hosting. Good old simple shared hosting outsells VPSes by a huge margin. There's plenty of room for a variety of products, all the way from shared, to VPS, to container based to dedicated servers. What I took issue with was the original article saying that containers would make VMs a thing of the past, which is silly.

Nux 13y ago

Containers certainly have their uses, but what I think will happen is hardware will become better and better at virtualisation just as the (K)VM software will become better. LXC is good for stuff like OpenShift (PAAS), but for "VPS" offerings the VM will continue to reign supreme.

zb 13y ago
Actually, OpenShift doesn't use LXC (it uses SELinux to separate gears) because containers are not secure in the way that VMs are.
- lil_cain 13y ago
  In the near future, it'll use kernel namespacing (like LXC) and SELinux.
alinspired 13y ago
I agree about VMs for VPS offering. But i believe no improvements in hardware or (para)virtualization will get VMs closer to Containers in other (non-VPS) scenarios

deleted 13y ago

[deleted]

sciurus 13y ago
How do VMs "abstract finance" any more than containers?
- deleted 13y ago
  [deleted]
  - deleted 13y ago
    [deleted]

susi22 13y ago

I'm surprised the article doesn't mention SmartOS. It's a superb OS which let's you spin up zones and KVMs. http://smartos.blueprint.org/home/why-smartos-in-my-lab http://smartos.blueprint.org/home/why-smartos-in-my-lab http://dtrace.org/blogs/brendan/2013/01/11/virtualization-performance-zones-kvm-xen/ http://dtrace.org/blogs/brendan/2013/01/11/virtualization-pe... http://www.cloudcomp.ch/2013/04/openstack-on-smartos/ http://www.cloudcomp.ch/2013/04/openstack-on-smartos/

wmf 13y ago
I wouldn't expect much Solaris[1] coverage in Linux Journal. [1] Illumos/SmartOS proponents seem to go out of their way to avoid saying the S-word.
- helper 13y ago
  Illumos has KVM support, Solaris does not. The distinction is made because Solaris and Illumos have diverged into different operating systems.

j_s 13y ago

Is it possible to have the same degree of security with a container as is provided by a virtual machine?

susi22 13y ago
Yes, the security is there. Though, resource fairness is not (since you have basically bare metal access). SmartOS is trying to fix this: http://wiki.smartos.org/display/DOC/Tuning+the+IO+Throttle http://wiki.smartos.org/display/DOC/Tuning+the+IO+Throttle http://dtrace.org/blogs/brendan/2012/12/19/the-use-method-smartos-performance-checklist/ http://dtrace.org/blogs/brendan/2012/12/19/the-use-method-sm...
- wmf 13y ago
  Linux now has fair sharing for CPU, disk, and network if you can figure out how to configure it. Or you can use cgroups to guarantee resources to make containers act more like VMs.
  - davidstrauss 13y ago
    cgroups are the method of doing fair resource sharing.
- Estragon 13y ago
  How can the security be comparable? I would have thought that the kernel shared between containerized processes presents a much bigger attack surface than a virtual machine monitor does.
- chubot 13y ago
  That's not true -- Linux has had local root exploits pretty much at ALL times. If you're in a container, you can break out with one of these exploits. You'll be root and have access to all other containers on the machine. In a VM, you don't share a kernel with other users, so the local root exploit doesn't buy you anything. I prefer containers and think VMs are a huge hack for most problems. But as it stands now, the number of local root exploits vs say remote exploits on Linux is probably greater by a factor of 100 or so, so you're taking much more risk. Security is one of the problems with Linux containers that still needs to be resolved. (see http://mattoncloud.org/2012/07/16/are-lxc-containers-enough/ http://mattoncloud.org/2012/07/16/are-lxc-containers-enough/, Google "lxc security")

amirmc 13y ago

Folks interested in this should also take note of the projects that target Xen [1,2 are ones I'm aware of]. Startup times that can be tuned to be on the order of 10s of milliseconds (see comment in Mirage paper). [1] Mirage - http://openmirage.org http://openmirage.org (paper at http://anil.recoil.org/papers/2013-asplos-mirage.pdf http://anil.recoil.org/papers/2013-asplos-mirage.pdf) [2] Zerg - http://zerg.erlangonxen.org http://zerg.erlangonxen.org

marshray 13y ago
I agree, these things are the future. Containers may be useful for single-tenant partitioning, but no one has managed to produce an OS kernel without an order of magnitude more local root security vulnerabilities than the HVM or PV virutualization systems.
- contingencies 13y ago
  Honestly, on Linux we can reach a similar point now by careful application of existing tools. For example, by whitelisting syscalls, devices, hardware and network resources and access patterns, in combination with containers. Most of the challenge is automating that process adequately, which is really nothing more than traditional code profiling meets existing policy-based kernel-level security toolkits meets containers meets development processes (ie. build/test/release). After ~3 years' with LXC, I believe that most of the attack surfaces are reasonably well understood and/or not required for most code to function. Sure, there will be kernel-level vulnerabilities, but they won't be exposed to such a broad range of userland applications. With security you also get so much more insight, power and predictability with regards to resource profiling than with paravirtualized solutions. Real kernel-level metrics versus load! Think of the impact for operations/capacity planning. So... short soundbite to summarize? Docker is a toy compared to where we are going.
  - shykes 13y ago
    Hi, I'm the author of Docker. I agree with everything you said about lxc, but I'm not sure how you reached the conclusion that docker is a toy. Since Docker is a portability layer on top of lxc, it automatically benefits from all the awesome work happening in the kernel. Or do you just use "toy" as a synonymous for "userland"?

aetimmes 13y ago

Solaris/illumos have been running containers for almost ten years now. They're just a layer of abstraction on the other side of the OS layer - a different tool, not necessarily a better one.

lsc 13y ago

Having sold both containers and VMs to untrusted users? I disagree. Now, my container experience is old, and containerization technology is improving all the time, so over time, what I say here becomes less true, but that's the thing; I'm willing to pay a fairly high cost in (cheap) ram for relatively small gains in compartmentalization. (which is to say, I'm willing to spend a fair bit of money to make my pager go off less.) See, the expensive part? it's not the ram, it's sysadmin time. That's why VPSs are killing shared hosting; it's actually easier to manage a bunch of VPSs than a bunch of people with shell accounts on the same box. Same goes for all the containerization technologies I've tried; it's easier to just throw xen/kvm on there and pay for the extra ram than to manage one user stepping on resources required by another user. For me? that was the beautiful part of switching from jails to Xen; suddenly, I could just ignore customers who tried to burn all the cpu and ram they could. Now, if I can figure out how to do the same with disk? I'll be a very happy man. That, at least, is where I'm putting my money; in more ram.

jauer 13y ago
SmartOS has disk IO throttling for KVM guests.
- lsc 13y ago
  and you can throttle with ionice on xen running a Linux dom0[1]. It works pretty great if you have some user who is going overboard and you want to make it suck for just them and suck less for everyone else. It doesn't work like the CPU scheduler; which is to say, I can't set ionice on all the guests and expect good performance on the light-use guests while a heavy-use guest is thrashing the disk. (I can expect the good performance for the light-use guests when the resource in contention is CPU time.) It's possible that either ionice isn't a good enough tool, or that I don't know enough about it... but I think it's far more likely that the variable nature of disk throughput makes balancing requests harder... (e.g. how many IOPS a second I get varies a lot from second to second, based on where the sectors I want to operate on are in relation to the read/write head. How much disk bandwidth I get varies even more.) I am pretty curious about how much this situation will change with SSD storage. [1]http://wiki.prgmr.com/mediawiki/index.php/Chapter_7:_Hosting_Untrusted_Users_Under_Xen:_Lessons_from_the_Trenches#Regulating_Disk_Access_with_ionice http://wiki.prgmr.com/mediawiki/index.php/Chapter_7:_Hosting...

CptCodeMonkey 13y ago

Just got a client off a container based provider and it wasn't too great an experience ( dramatically slow networking, unreliable file I/O ). Granted I could, and can continue to, say the same thing about virtualization but virtualization is here and reliable enough that it seems like a hard sell to want to try something that is even less performant.

fridder 13y ago

Also see FreeBSD Jails:

davidstrauss 13y ago
Hi, I'm the author of the article. FreeBSD Jails lack the same fine-grained isolation choices versus the base system that the Linux kernel exposes through namespaces and cgroups. That's not to say that Jails don't capture most of the value I argue containers have, but they're a different generation of design. Also, I wrote the article for the Linux Journal, which obviously affects the solutions explored.
- oijaf888 13y ago
  What can cgroups and namespaces do that rctl and capsicum can't? I assume with namespaces you can have multiple processes running with the same PID?
  - davidstrauss 13y ago
    RCTL can enforce specific limits, which is good if you either want to divide resources such that there can't be (or is unlikely to be) contention. cgroups offers hard limits for some things, like memory, but it mostly opts for a model using "shares" that determine the fractional access to resources versus other cgroups holding shares against the same resource. For example, assume there's CPU contention. cgroup A has 10 CPU shares and cgroup B has 90. Processes in cgroup A will get 90% of the CPU time, but it will not starve cgroup A because cgroup A will still get 10%. This shares-based model also has a major effect when there isn't contention. Shares-based resources are burstable. Even cgroup A (with 10 CPU shares) can use 100% of CPU if nothing else needs it. This "burstable" nature can be good or bad. It's good in the sense that most users will probably get more CPU than their shares guarantee most of the time. It's bad because users can start expecting more than their shares guarantee and get a nasty surprise when resources get under contention. It's time to drop some analogies. cgroups are very much like a highway with an HOV lane (or more): anyone can go very fast when there's no contention. But, during rush hour, lanes get distributed as "shares" of the road to the HOV and non-HOV groups. Neither the HOV nor the non-HOV drivers get starved for road access (though responsiveness may not be equivalent, by design). Traditional "nice" is like emergency vehicle traffic. An ambulance every now and then works fine as "-20 nice" traffic. But, if you filled the road with ambulances, it would starve normal traffic of roadway access. RCTL is sort of like a person riding reserved right-of-way public transit. From the time the person hops on the train at point A to when they get off at point B, it will be the same duration any time of day. They don't get to go faster during low-traffic times, but they also don't have to worry about a significantly worse experience during rush hour. Capsicum seems focused on intra-application isolation; I'm not sure how to compare it to other OS-level containers.

ewams 13y ago

Good article that discusses something I have not touched in years, simply because of a lot of the drawbacks he talks about. Also is a wonderful sales pitch, someone has been practicing. In the virtualization world we argue often about how to do things “right” and what the “industry standards” are. I started a series a while back to help define and explain virtualization and various related topics. Specifically I even talk about this subject ( http://ewams.net/?date=2012/01/04&view=The_4_Types_of_Virtualization#b http://ewams.net/?date=2012/01/04&view=The_4_Types_of_Virtua... ). The main problem with saying VM’s are “no longer the future” is they are mainstream and have been for a while, so they probably are not not cool enough for hackers but they are not going anywhere fast. Containers have a place and full blown virtual machines have a place. The trick is finding which one will benefit your organization and for which purpose. Most likely both can work depending on the requirements. The other topic I notice that is usually overlooked when talking about virtualization is the actual OS. Here in this article it discusses mostly Red Hat based distro’s. Some organizations strictly use OSX, others Ubuntu, others RHEL, while others use some sort of mix. Each is different and even for closely related like Fedora and RHEL, what works on one may not work on another without hacking it or some other black magic. (Or it may work just fine, you know how it goes). The other point of the OS talk is where is Windows? We can pretend to be ostriches but Windows server is out there and in large numbers. Lots of people make lots of money writing applications that work strictly on Windows (same for *nix). Just do not forget about it. If we move forward with containers, and since the topic has come up at least twice in the last 6 months, let’s start to see recommendations on how to run containers. How to pick workloads. How to optimize applications and kernels to run them. What kind of hardware to run them on. How to code for them. Etc.

toolslive 13y ago

IMNSHO: the future should be both VM-less and container-less. meanwhile, containers are more efficient.

davidstrauss 13y ago
How would you suggest we efficiently partition and sell computing resources, then? It's not cost-effective or even power-efficient to run separate hardware for each scale of computing. For example, many projects only need a container or VM with 512MB of memory and one CPU core.
- VLM 13y ago
  Can't speak for the original poster but my guess is the obvious drop in power/cost required for computing. "Just wanna email, word process, and some web and some gaming" has relatively recently gone from a kilobuck P4 couple hundred watt space heater to a hundredbuck tablet that needs charging every couple days. Unless you all can find a way to make silicon more expensive or dramatically increase processing demand, all these elaborate schemes to dole out processor are doomed to something like the per minute long distance business model where eventually more money was being spent on detailed billing than on actually providing the service... When the TCO of an elaborate virtualization system and its provisioning and billing and firewalling system exceeds the TCO of just spinning up a ubiquitous cheap low power system on a chip...

AndrewGaspar 13y ago

It's pretty bad they don't even mention Windows Azure - their Web Sites service is already using containers, though only on the free tier for right now.

davidstrauss 13y ago
I'm the author of the article. It doesn't mention Windows Azure because I wrote it for the Linux Journal, and a full discussion of containerization outside of Linux wasn't possible in the allotted article space.

amscanne 13y ago

Containers have been mature for longer than the article implies (freebsd jails, Solaris zones). Technology just doesn't work this way. Things don't simpler over time (and you don't shed support for different kernels, OSes, better isolation, etc.) Not to mention the fact that virtualization has gotten dramatically better since EC2 was launched, since that seems to be the benchmark here (they are still running Xen 3.X first released in 2005). Modern virt hardware+hypervisors is blazing fast and (I believe) will eventually outpace bare metal for specific workloads on big boxes due to the improved isolation for many performance critical operations (I.e. separate TLBs for flushes, smarter cache invalidation). Even if they don't reach parity, the gap certainty wouldn't justify giving up all the benefits of full virtualization.

davidstrauss 13y ago
Hi, I'm the author of the article. > Containers have been mature for longer than the article implies (freebsd jails, Solaris zones). Mature containers have been around since the the days of mainframes. The recent (say, last decade) disparity has been lack of mature containers on the platform of choice, which is Linux for most cloud-based projects. I don't know many people who would switch their projects to run on FreeBSD or Solaris just to use containers instead of VMs. Also, the container types you mention don't support independent, fine-grained choices of isolate-or-not for networking, the file system, user IDs, process IDs, and scheduling of system resources. > Modern virt hardware+hypervisors is blazing fast and (I believe) will eventually outpace bare metal for specific workloads on big boxes due to the improved isolation for many performance critical operations (I.e. separate TLBs for flushes, smarter cache invalidation). It's unclear why you think such optimization would appear in hypervisors but not kernels, especially given how much more insight a kernel has into the workloads it directly runs compared to a hypervisor running a kernel running workloads. Even if hypervisors achieve CPU performance parity for running systems, you still haven't addressed the memory and storage overhead (in terms of resident footprint) of running full OS images instead of containers. > Even if they don't reach parity, the gap certainty wouldn't justify giving up all the benefits of full virtualization. You're portraying virtualization as having benefits versus containerization while ignoring that containerization also has many benefits over virtualization, especially w.r.t. cost efficiency.

ausjke 13y ago

LXC will never replace XEN/KVM/whatever. LXC has its places, but saying it is the future for all VMs is just naive. Since it's from linuxjournal I want to check who dares to make such bold statements, then realize this is probably the real purpose of that article, a twisted ads for its Drupal business? In short, the article is pointless and shallow, and you can't stop checking who is making such claims, and then you find yet another drupal hosting site, this sucks.

alinspired 13y ago

Containers are efficient(and most likely the future) for PaaS/SaaS type of applications, where individual Instance (VM/Container) is not treated as a "computer" or Infrastructure. Another benefit for PaaS/SaaS is that you don't necessarily sell RAM or DISK, you sell an application/service and thus can take FULL advantage of containers, like: - more efficient resources sharing(on host) - dynamic resources re-balancing - quicker start/stop/etc Containers can be used to offer IaaS, but as was already mentioned in this thread, a few edge cases (custom kernel or kernel module, any filesystems support, any firewall support) prevent it due to support/operations overhead.

gundy 13y ago
They are not really efficient. All they do is add an extra layer of security so you can encapsulate that specific process in a chroot with its own networking stack. Other than that the CPU is still fairshared and the memory still has the same limits as if you ran it without the container. It adds nothing but security.
- wmf 13y ago
  In the cloud, KVM/Xen overhead is considered the baseline so containers are definitely more efficient by comparison.
- alinspired 13y ago
  There are container technologies (openvz, LXC) that offer lot more than "fairshared" to manage resources between containers. Plus the efficiencies of lower "virtualization overhead", because you don't have full OS in containers compared to VMs (consider full VM kernel, VM HW drivers, etc)

j45 13y ago

Three thoughts: 1) I'm not sure if this article has accounted for the one massive achilles heel of paravirtualization (containers): Corrupting the core OS by way of upgrade or patches, can take all containers all out. 2) I have used VMware since the early betas in 98. The motivation was simple: Replacing servers sucks. The day you buy or lease a server, in essence, you purchase it's death and dealing with a hasty or painful migration. 3) From my 7th or so generation of server migrations in doing complex hosting since 95, one thing is clear: The single most valuable thing is a hard separation between the bare metal and the OS. So much so that you can move the VMs at will anywhere, because, you just bought the death of your server the day it's brand new. I ran a CentOS box with a BSD VM as a firewall, with Windows and Linux VM's behind it to serve one project because each did their unique and highly valuable part. Ultimately, for me: Paravirtualization not only uses the host os, but shares it between multiple configurations. This is in my mind still introduces risks that should be minimized, if not avoided. I'm sure PV is improving all the time. It's simply newer to the game than what VMware started. If VMWare isn't for you, suites like ProxMox are serious options to consider. Hosting isn't somewhere to have preferences and opinions. If I'm wrong, I'll change at any moment. Issues with PV might be decreasing, but I've been ultimately burnt by PV every time I used it in production for a prolonged time. An example of where sysadmin time cost way more than a little bit of more ram for Virtualization: Parallels Virtuozzo, corrupted itself so badly during an update to the core OS that all the containers on the server were rendered inoperable. Parallels support had one of their high level techs login from Russia to manually install corrupted DLL files so the server would at least boot, and we could get the heck off it. At the end of the repair we were told the VM was no longer upgradeable and all updates would have to be done manually. The customer had taken on PV at the behest of the hosting company who insisted it was amazing despite my concerns. Said hosting company also now has switched over entirely to VMware. The unforgivable kicker of PV for my customer: tons of customization and creating re-usable containers were all lost when leaving, because moving to something like VMware meant having to set them up from scratch again.

alinspired 13y ago
Your comment must be entirely about Containers for Windows (by Parallels), which is not what the article and most of the discussions are about. Containers for Linux on the contrary are not prone to host "corruption"

thelarry 13y ago

Linux containers have been awesome for helping me sandbox user generate code. Has anyone actually had experience with using Mesos in the real world?

gundy 13y ago
That's all that they are good for adding security that would otherwise not exist. Outside of that under a PaaS, I don't see what they add.

davidgerard 13y ago

This is totally the case. Back before it got Oracled, Solaris 10 zones were one of the finest things ever - and completely controllable in resource usage, with a few percent CPU overhead at most (for comparison, running ZFS takes more). Zones for Linux would win, comprehensively.

gundy 13y ago

In other news OPs startup not the future of PaaS but OpenShift, Redhat's open source alternative is.

cornet 13y ago

Containers are zones are great if you're building your own platform that you control. However I'm not sure building a multi-tenancy hosting environment based on containers or zones is the best idea. Suppose the tun driver has a bug in it which can cause a kernel panic under some circumstances and that someone running OpenVPN in a container or zone hits this condition. You can now wave bye bye to all the zones running on that host. https://github.com/joyent/illumos-extra/commit/9412039a18f2f52b24b18ad8c8642a55d3b50d93 https://github.com/joyent/illumos-extra/commit/9412039a18f2f...

gnel 13y ago

I currently have a OpenVZ vps on the internet to host my projects and I chose it for its better performance in comparison to a real VM (xen,kvm,etc) but my what I'm still to find it's a provider that offers OpenVZ containers (or virtuozzo containers) that can scale programatically, like the VM's on Amazon EC2. I think it should be possible, that I could program if my container cpu load is over 70% for more than 40 seconds to launch a new copy of my container in the same or other machine.. This lack of a service to scale containers it's the first thing that needs to be done to compete with common VM clouds.

csense 13y ago

What's ironic is that the entire point of an OS that supports multiple processes is to provide them each with an illusion that they're totally in control of their own machine. This abstraction was kind of leaky -- different processes typically share various namespaces, like filesystems, process tables and user accounts. As a result, virtualizing an entire OS became a way to make a tight abstraction of having an independent machine at your disposal. The reason virtualization happened was because emulating the system hardware was easier than changing every OS to adequately separate their namespaces. Now that virtualization has the attention of OS developers, you have development like LXC / cgroup stuff in Linux, bringing isolation into the OS, where it should have existed all along. AFAIK the mainframe world -- which, by the way, still exists -- has never gotten out of the early-computing mindset of having an expensive machine which must be shared between applications, so isolation and containerization has always been a feature of mainframe OS's.

grogers 13y ago

Maybe, but the linux kernel has had its share of local privilege escalation bugs. If I were a cloud provider (or user) I'd be hesitant to trust containers for hard isolation.

jpollock 13y ago

Sun/Oracle have been championing containers (Zones) for a long time now. They have their places. However, businesses still want full virtualised systems (LDOMs). Businesses like treating applications like appliances. They want to start up a system and not touch it again. Migrating an application to a new OS release costs them a lot of money. A zone forces an upgrade - everything on the machine needs to be rolled forward to the new release. Since businesses typically have a patchwork of software in various levels of support, continuous upgrades can be an expensive (and possibly impossible) task. For example, the application vendor may require a license purchase in order to get support for the new OS release. The vendor may even have gone out of business, and the replacement is still 6-12 months away. In a full virtualised platform, you're not forced to continually pay that maintenance cost. You are free to store it as technical debt and pay it in a more managed fashion.

wmf 13y ago
But both Solaris and Linux let you run old userspace in a container on a newer kernel.
- jpollock 13y ago
  Actually, there are limits, as I was informed in the Solaris 11 launch. I didn't pay too much attention because it didn't affect me, but the guys it did affect weren't that impressed!

mark_l_watson 13y ago

A question: which hosting companies sell container hosting services for specific platforms like Sintra+Postgres, Java+Hybernate+Mysql, etc? If I understand this correctly, you would want to be able to rent container slices with the specific stack you are going to use, already set up.

wmf 13y ago
If you want a container with a specific stack that's usually called PaaS. Personally I hope that tools like Docker allow some companies to focus on hosting containers (hopefully not inside Xen VMs...) and others to focus on building great stacks and container templates.

kleiba 13y ago

A short (5min) talk by Solomon Hykes of dotCloud about how their "docker" tool makes use of containers: https://www.youtube.com/watch?v=9xciauwbsuo https://www.youtube.com/watch?v=9xciauwbsuo

bmullan 13y ago

One future for LXC is with ARM based servers such as Calxeda. Approx 800 core with gigabit Ethernet fabric & I/O. And sata built-in for each core. About size of a Cisco catalyst 5000 box but less than 1000 watts. Runs ubuntu or Red Hat and uses LXC for virtualization. Last I saw it was <$100k