4 ms·
What are Linux containers and how did they come about?
- derefr 13y agoIn practice, containers are process groups with access restrictions. In theory, though, containers are simply an optimization of virtual machines, and are best understood that way. If you can use virtual machines to solve a problem--and your virtual machines are all generally based on a recent Linux--then you can achieve the same thing, using far fewer resources, by using containers.
- 3am 13y agoI think the virtual machine analogy is a bit dicey because lxc+cgroups/Docker containers have to be much more aware of the host kernel api and devices. edit: yes derefr, I agree. My point was also if there was a distribution or kernel specific Ruby or Ruby library bug, for example, then you might have to account for it in your code if you ran in containers (if you don't control the host on which your code is running), whereas you control the whole stack in the virtual machine.
- derefr 13y agoBut it's not idiomatic to take advantage of those close ties (by, say, having two containers share a named pipe mounted from the host.) I suppose what you mean is that containers aren't applicable in every instance that VMs are--like, say, providing VPS instances, where some of the instances might want to do funny fiddling things with their virtualized "hardware." This is true. But for the situations where the applicability of VMs and containers intersect--using them to run Heroku-like isolated "app slugs", for example--then you can think of containers as lighter-weight VMs, and you won't really go wrong.
- k3oni 13y agoYep, this is more like it
- gaius 13y agoBut there should be no such thing as a VPS. You should be able to just give n people a shell on a box and quotas, permissions etc would suffice. That you can't is a shortfall of the OS and a hypervisor is a bandaid.
- tbrownaw 13y agoWhich could be made to work until someone wants to run an alternative OS (ie, not Linux) or use features that are only available in newer kernel versions. Or until someone realizes that a shared kernel with a security policy is a larger attack surface than a tiny hypervisor with no concept of sharing.
- dpark 13y agoAnd what's your security model? Each user that needs to run services has to run them under their user account? Or are you going to also allocate some number of service accounts to each user and pay the cost to manage these accounts and the combined quotas they entail? Now how do your users' service accounts share resources? Someone will need a web server running under one account and a job processor running under a different account (different accounts in order to allow for limited privileges). Are they restricted to communicating through named pipes (hope no one else binds the pipe)? What if they want to share some disk-based resources? Shove them all in a group and set someone's home directory to 770? How do you handle it when one of your users needs a SQL server? Do they need to install this in their home directory, too? And lock down their install so that hopefully other users on the box can't call it? What about port binding? Just luck of the draw, and everyone binds on what they want? Then they file a request for you to forward to the port they happened to grab? And they hope that they never lose the port due to a server reboot, or a service crash? People could work in the environment you describe, but it would be miserable. And you (the server owner) would have to sink a massive amount of resources into maintenance. This isn't a shortcoming of the OS. It's a fundamental mismatch between the goal of sharing a server among many users and giving those users sufficient control to build what they need.
- chongli 13y agoHow do you handle it when one of your users needs a SQL server? Do they need to install this in their home directory, too? And lock down their install so that hopefully other users on the box can't call it? NixOS can handle situations like this. It enables per-user package management with deterministic, immutable installs. If multiple users install the same package, it only puts one copy on the system. Each user gets their own environment where they can customize the package without affecting the other users.
- MrBuddyCasino 13y agoI think we've used something similar some time ago (maybe XEN?) and had the following issues, how does Docker compare: - what happens when one container changes the system clock? - are iptables rules per container or global?
- ambiate 13y agoAs far as I understand, the clock and such are tied to the base host. Typing 'uptime' in my linux container shows 5 days, the uptime of the base host.
- justincormack 13y agoIn theory things like the clock could be namespaced too, if anyone found it useful.
- MrBuddyCasino 13y agoThat would be useful - sometimes you have to change the system time to debug some code, and if that trips up all the other containers, its a deal breaker for us.
- djchen 13y agoI was thinking more like FreeBSD jails. Containers can't change the system clock.
- qznc 13y agoNetworking for Docker containers is bridged, so iptables are per container. http://blog.docker.io/2013/08/containers-docker-how-secure-are-they/ http://blog.docker.io/2013/08/containers-docker-how-secure-a...
- mcpherrinm 13y agoLinux network namespaces let you have multiple independent network stacks in linux. That means routing tables, interfaces, routing tables, IP addresses, etc. Each container gets its own network namespace (along with other namespaces like hostname, pids, users, ipc, filesystem mounts). Anything not handled by one of the 6 namespaces is the same across all containers. That includes things like what kernel modules are loaded, the system clock, etc. Because a user with root can manipulate the kernel in many ways, I wouldn't give root to an untrusted user and assume containers were enough to contain them. Certainly if they can load a custom kernel module it's game over, but I'd bet there's plenty of other ways to break out too.
- k3oni 13y agoWelcome back OpenVZ... except that this will be part of the OS now.. and i wouldn't call these "an optimization of virtual machines".. containers are part of the kernel/OS so call it kernel/OS virtualization based if you want where virtual machines are or supposed to be hardware virtualization based. Ex. openvz,lxc - containers kvm,xen - virtual machines
- memracom 13y agoOpenVz was something that required hacked kernels. A better analogy is LXC = Solaris Zones KVM and XEN are hypervisors for virtualization. VirtualBox is a non-hypervisor virtual machine.
- vidarh 13y agoOpenVz is something that requires "hacked kernels" only because the changes were not accepted upstream. I don't think your analogy is any better. We could tack on an " = OpenVz" on there - all three of them are pretty similar.
- diminoten 13y agoExcept when you need snapshotting, live-migration, or have a highly disparate set of underlying OS requirements. The author of the article explicitly says, "This isn't to say that they're going to replace virtual machines." and I agree - there's no point in pretending like they're solving all of the same problems. I guess you could say that containers are an optimization of some use cases of VMs.
- gaius 13y agoBut that's simply not true. 20 years ago you could live migrate processes around a VAXcluster with nary a VM in sight.
- wmf 13y agoBut since history does not necessarily flow in one direction, today you need a hypervisor (or experiment with CRIU) to do live migration.
- ghaff 13y agoAlmost. Although I don't remember the details, IBM has Live Partition Mobility for WPARs on AIX. They were looking for differentiation from Solaris Containers at the time and bought a startup called Meiosys, which had some tech which helped them add live migration to WPARs.
- wmf 13y agoHeh. Even though I work for IBM I wouldn't bother mentioning AIX on HN; it simply doesn't exist inside the echo chamber.
- pigs 13y agoYou actually have to buy an add-on called WPAR Manager[1] to enable live mobility; it's not available in baseline AIX. They have been shipping this tech for several years now. Disclaimer: I was on the WPAR Manager dev team for several releases. [1] http://www-03.ibm.com/systems/power/software/aix/sysmgmt/wpar/ http://www-03.ibm.com/systems/power/software/aix/sysmgmt/wpa...
- zobzu 13y agoThats kinda true indeed. Containers people want to use are VMs that directly talks to the local kernel (i.e. it does not run another kernel on top, like virtualization does). You can make process groups without making the container look like a VM ... BUT. That's hard and time consuming for little gain.
- pjmlp 13y agoThat is why micro-hypervisors where all OS instances are virtualized even the one acting as master, like on mainframes, are gaining acceptance.
- cbsmith 13y agoAs someone who has used OpenVZ for a long time, I'd have to disagree with the notion that containers are simply process groups with access restrictions. You have a distinct namespace for them (including networking), which really changes everything.
- lsc 13y agoGenerally speaking, containers have worse resource separation. Last time I looked, a system with containers shared pagecache. Sharing pagecache means that one user thrashing the disk can make the system unusable for everyone, well, to the extent that no system without pagecache is usable. Xen and KVM mitigate this by giving you your own ram and thus your own pagecache. Personally? I think this makes a way bigger difference than any cgroup limiting of iops. Pagecache is not free ram, and should not be treated as such. I think containers are one of those tradeoffs; they are more efficient when it comes to hardware, but they are less efficient when it comes to sysadmin time tracking down users who are using more resources than they ought. On the other hand, some find managing multiple containers easier than managing multiple complete virtual servers.
- jsmthrowaway 13y agoPart of the problem, Luke, is that you're thinking about containers in terms of a hosting provider. That's not your fault. You're a hosting provider. To you, cotenancy is not only a security concern but also performance and SLA. For almost everyone else, efficiency is way paramount. I still think right now the best way forward is: - Hosting provider: Xen/etc - Internal clusters: containers Containers merely bring commodity computing closer to HPC operational techniques. Containers are not useful for you at prgmr. Using every drop of RAM on the machine for page cache is a positive for my use case, not a negative. Virt overhead makes no sense in internal clusters, in my opinion. I've worked in an all virt environment and an all container environment. The latter is way better, and these are scales where Puppet burning a core for several seconds across the fleet has a demonstrable impact on Opex.
- lsc 13y ago>Part of the problem, Luke, is that you're thinking about containers in terms of a hosting provider. That's not your fault. You're a hosting provider. To you, cotenancy is not only a security concern but also performance and SLA. For almost everyone else, efficiency is way paramount. Oh yeah. More fair is almost always preferable to more efficient when you are in a multi-user situation, and certainly, containers are more efficient. I... have always had a problem, though, with this idea that you should virtualize clustered production applications; virtualization takes a big server and cuts it up into little servers (at a cost) - whereas clustering takes a bunch of little servers and combines them into a big server, again at a cost. It seems to me like once you are out of testing and actually need a full server worth of capacity, you don't want to use virtualization anymore. Multi-tenancy is a feature I need, sure, but for the cluster syadmin? it's all downside. containers/chroots do make a lot more sense than Xen or Kvm if you must virtualize your production clusters; I guess that's the idea behind docker; make a clean way to pass around directories full of binaries. I mean, to be fair, making packages /is/ a pain in the ass, so I can understand wanting a clean way to pass around your production code without using packages. Really, the more I think about it, the more I see containers as an easy way to roll back all the benefits and problems of using shared libraries without going through the incredible pain of statically linking everything. Distribute your "binary" along with all the libraries and the environment it requires. Hah. And actually? I can understand why you would want to do that; Before I was a service provider I was a cluster sysadmin, and I've put in my time resolving shared-library conflicts.
- gaius 13y agothe concept--and products based on that concept--has been around for almost a decade Closer to 6 decades, this is old hat for IBM mainframes. For me the question is tho', why virtualize? And the answer I keep coming back to is that a) it is too hard to retrofit good manageability onto processes as the basic unit of applications and b) people are used to it, the idea of having 1 machine per app, when really it isn't cost effective to do that.
- deleted 13y ago[deleted]
- ambiate 13y agoSegregate applications. If app one gets compromised, app two is off in another land. If my linode box gets hacked, without containers, all 16 websites will be compromised/go down. Desktop virtualization using the NX protocol and Linux containers is also really friendly. Our company uses old Pentium D boxes for desktops. The performance gain from using a thin client + lxc + lubuntu is insane! I can actually watch youtube videos and draw in OpenOffice Writer. (Also great for disaster recovery scenarios).
- gaius 13y agoBut processes - theoretically - are segregated. Own address space, running as different users, bound to a processor set, etc. The use of hypervisors is because processes in practice don't really do what they say on the tin, so you need to force another layer of protection and manageability in, and just eat the overhead.
- ambiate 13y agoI suppose I was thinking from the 'lazy throw up apache, mysql and wordpress' mindset. If example.com is in its own container with its own mysql database, I do not have to worry about www.test.com getting exploited and example.com's mysql data getting leaked into the wild. I'm also from the days when buffer overflows were on every daemon. Unauthorized remote shell access was always a threat back in those days.
- slig 13y agoThis is the current text: > to change the text of this page send 0.1 btc to 16JMNc3B5vCkuuPxNcNj388gmhP8UDKBuW Well done, it didn't take more than 1 hour to someone find a loophole.
- gtrubetskoy 13y agoThis article curiously fails to mention Linux VServer http://linux-vserver.org/ http://linux-vserver.org/ which has done this since 2003 and unlike Virtuozzo has always been completely open source. We've used it with great success in a hosting product.
- ghaff 13y agoI probably should have mentioned it. Virtuozzo was much higher profile at the time I was closely following various workload separation technologies--in part because it was making a serious attempt to position itself as an alternative to VMware in the enterprise space. (Something it ultimately failed to do although it continued to enjoy considerable success in hosting.) The fact is that there have been lots of different isolation approaches, including OS virtualization, and Virtuozzo's implementation of OS virtualization--while hardly unique--was the best known along with Solaris Containers.
- cbsmith 13y agoIt also didn't mention OpenVZ by name. Honestly, ever since OpenVZ came out, I haven't felt like the case for using VServer was terribly strong.
- octix 13y agoCan anyone please give a simple example or link of a use case of containers need? I use KVM/Qemu VMs at home and work(dev box), I have a dozen always running VMs. However, I started to read more and more about LXC containers, and still cannot grasp their importance... for example, I manage a few VPSs too, each has it's own purpose (web server, database), how could LXC help me? Thank you in advance.
- cbsmith 13y agoBasically you can use them just like you use your VM's. The main difference is you lose the ability to have different kernel's for each VM.
- octix 13y agoOk, thank you.
- ghaff 13y agoBasically every Platform-as-a-Service uses containers. They're much more efficient than VMs (resources, spin-up/down times) and the underlying infrastructure is standardized anyway. They may not very well not be relevant to a couple of home systems. We're talking about large-scale infrastructures.
- octix 13y agoYes, fast spin-up/down are an advantage. Thanks.