25 ms·

Docker - the Linux container runtime

273 points by radimm 14y ago

andyl 14y ago
Looks cool. When is it gonna be open sourced?
- niggler 14y ago
  "Docker will be open source soon" Soon could be in tomorrow, a week, a month, a year, or longer ...
  - shykes 14y ago
    ETA 2 weeks maximum.
asadjb 14y ago
I'm not familiar with any of the technologies used in this. Anybody care to comment on how strong the isolations would be security wise, compared to normal virtualization? If the security is almost at par and the isolation is good enough that one bad process can't bring the whole system down, might this be a good alternative to virtualization, since I imagine it would definitely use less resources.
- scarmig 14y ago
  Back in January I got a new laptop, installed Arch on it, got it all nicely set up. And I decided it was high time to start playing with LXC because container virtualization seems extremely promising to me. Created an Ubuntu container, seemed to work fine, and then used lxc-destroy, which took some time. It destroyed my entire file system. I have no clue how the hell it happened--it floors me that something like that would be possible--and I suspect it's probably simply the result of a newbie like myself somehow misusing userspace tools. But it was enough to turn me off of it for the time being.
- zdw 14y ago
  Barring kernel bugs, it should prevent against the mentioned resource monopolization issues. Normal virtualization is pretty resource wasteful, especially if the guests are not hypervisor aware. Getting away from huge per-VM block devices is a step in the right direction.
- teraflop 14y ago
  The Linux namespace stuff is evolving pretty fast, and I personally wouldn't trust it as the main line of defense for anything important. With virtualization, a buggy or malicious guest is still limited to its sandbox unless there's a flaw in the hypervisor itself. With containers/namespaces, the host and guest are just different sets of processes that see different "views" of the same kernel, so bugs are much more likely to be exploitable. Plus, if you enable user namespaces, some code paths (like on-demand filesystem module loading) that used to require root are now available to unprivileged users. There's already been at least one local root exploit that almost made it into 3.9: https://lkml.org/lkml/2013/3/13/361 https://lkml.org/lkml/2013/3/13/361
  - derefr 14y ago
    > The Linux namespace stuff is evolving pretty fast, and I personally wouldn't trust it as the main line of defense for anything important. If I recall, Heroku uses cgroups (EDIT: and namespaces) exclusively for multitenant isolation (and by the looks of this, dotCloud does too), so that's two big votes in the "if it's good enough for them" category.
    - teraflop 14y ago
      Sure, but cgroups and namespaces are kind-of-orthogonal features that both happen to be useful for making container-like things. cgroups are for limiting resource usage; namespaces are for providing the illusion of root access while actually being in a sandboxed environment. And as far as I'm aware (speaking as an interested non-expert, so please correct me if I'm wrong) cgroups have no effect on permissions, whereas UID namespaces required a lot of very invasive changes to the kernel.
      - deleted 14y ago
        [deleted]
      - derefr 14y ago
        Oh, no, you're right--I meant to include both things. I should have just been lazy and said "all that tech LXC uses." ;)
- wslh 14y ago
  I am familiar with Microsoft App-V, VMWare ThinApp, and Symantec Workspace Virtualization. They can help you as a security sandbox but not as a full protection. A virtual machine will be much more secure (and theoretically very strong), although there are security bugs there that enable you to escape it. Those products work at two levels: using filtering drivers for registry and the filesystem, and hooking into the Windows operating system API.
  - laumars 14y ago
    Virtual machines are not more secure. In fact there's been more documented attacks where root access on a guest VM has gained shell access on the host, than there's been against containers. This doesn't mean that containers are more secure than VMs either. Attacking VMs attracts more security researchers from what I've seen (but I may be wrong on that point). However whether your running a container or a virtual machine, you still need some shared processes (eg the 'ticks' of a system clock) and with any sufficiently complicated code WILL have bugs that can be potentially exploited. However the crux of the matter is regardless of whether you're running containers or full blown virtual machines, you cannot escape out of the sandbox without having elevated privileges on the guest to begin with. And if an attacker has that, then you've already lost - regardless of whether the attacker can or cannot escape the sandbox. Lastly, I'm not sure if you're aware of this or not, but this is a Linux solution and has nothing to do with Windows (I only say this because your post seemed tailored towards Windows-hosted virtualisation)
    - wslh 14y ago
      Are you saying that both approaches have the same level of security or probable insecurities? or that you can't currently estimate the difference? Even being aware that this is a Linux solution I mentioned the Windows technologies that I know technically.
      - laumars 14y ago
        > Are you saying that both approaches have the same level of security or probable insecurities? or that you can't currently estimate the difference? A bit of both, but mostly the former. In practical terms, they both have the same level of security. But -as with any software- something could be published tomorrow exposing some massive flaw that totally blows one or the other out of the water. However neither offer any technical advantage over the other from a security stand point and from a practical perspective, the real question of security is whether your guest OSs are locked down to begin with (eg it's no good arguing which home security system is the most effective if you leave the front door open to begin with). > Even being aware that this is a Linux solution I mentioned the Windows technologies that I know technically. That's fair enough and I had suspected that was the case. I just wanted to make sure that we were both talking about the same thing :)
- trotsky 14y ago
  Container based virtualization can provide an impressive amount of isolation while improving density dramatically on light duty loads over virtualization. Solaris zones are very well regarded and are used for multi-tenant by Joyent, and many many linux hosts provide multi-tenant solutions based on virtuozzo which predates linux containers by a good number of years. The main theoretical difference between hypervisor isolation and container isolation is one sits above the kernel, so a kernel level exploit only applies to a single virtual machine. With containers you're relying on the kernel to provide the isolation so you are still subject to (some) kernel level exploits. Practically linux containers (the mainline implementation) have only provided full isolation in recent patches and probably shouldn't be considered full shaken out for something like full in the wild root level multi-tenant access. They are super for application isolation for delivery of multiple single tenant workloads on one machine though - something people use hypervisors for quite a bit. The resources used can be a small fraction of what you're committing to with a hypervisor.
  - bcantrill 14y ago
    As trotsky mentions, we at Joyent are fervent believers in OS-based virtualization -- to the point that in SmartOS, we run hardware virtualization within an OS container. There are many reasons to favor OS-based virtualization over hardware-based virtualization, but first among these (in my opinion) is DRAM utilization: with OS-based virtualization, all unused DRAM is available to the system at large, and in the SmartOS case is used as adaptive replacement cache (ARC) that benefits all tenants. Given that few tenants consume every byte of their allocated DRAM, this alone leads to huge efficiencies from both the perspective of the cloud operator and the cloud user -- a higher-performing, higher-margin service. By contrast, for hardware-based virtualization, unused DRAM remains with the guest and is simply wasted (kludges like kernel samepage mapping and memory ballooning notwithstanding). DRAM isn't the only win, of course: for every other resource in the system (CPU, network, disk), OS-based virtualization offers tremendous (and insurmountable) efficiency advantages over hardware-based virtualization -- and it's great to see others make the same realization! For more details on the relative performance of OS-based virtualization, hardware-based virtualization and para-virtualization, see my colleague Brendan Gregg's excellent blog post on the subject[1]. [1] http://dtrace.org/blogs/brendan/2013/01/11/virtualization-performance-zones-kvm-xen/ http://dtrace.org/blogs/brendan/2013/01/11/virtualization-pe...
- bradrydzewski 14y ago
  This is still technically a virtualization technique, known as "operating system-level virtualization". http://en.wikipedia.org/wiki/Operating_system-level_virtualization http://en.wikipedia.org/wiki/Operating_system-level_virtuali... Here are some of the technologies explained: cgroups: Linux kernel feature that allows resource limiting and metering, as well as process isolation. The process isolation, also called namespaces, is important because it prevents a process from seeing or terminating other running processes. lxc: this is a utility that glues together cgroups and chroots to provide virtualization. It helps you easily setup a guest OS by downloading your favorite distro and unpacking it (kind of like debootstrap). It can then "boot" the guest OS by starting it's "init" process. The init process runs in its own namespace, inside a chroot. This is why they call LXC a chroot on steroids. It does everything that chroot does, with full process isolation and metering. aufs: this is sometimes called a "stacked" file system. It allows you to mount one file system on top of another. Why is this important? Because if you are managing a large number of virtual machines, each one with 1GB+ OS, it uses a lot of disk space. Also, the slowest part of creating a new container is copying the distro (can take up to 30 seconds). Using something like AUFS gives you much better performance. So what about security? Well, like every (relatively) new technology LXC has its issues. If you use Ubuntu 12.04 they provide a set of Apparmor scripts to mitigate known security risks (like disabling reboot or shutdown commands inside containers, and write access to the /sys filesystem).
zdw 14y ago
Are the aufs filesystems easily navigable from the host OS, ala Solaris's zones feature?
- zimbatm 14y ago
  The aufs mount is (probably) only mounted on the guest's VFS so I'm not sure how you would access it from the host.
  - shykes 14y ago
    Docker lets you visualize changes on any container's filesystem live, as they happen. It also lets you snapshot changes from any container into a new image, and immediately start running new containers from that image. No manual post-processing of the image, no configuration files to templatize. The whole thing is 2 commands and maybe half a second.
  - jpetazzo 14y ago
    The AUFS mountpoint is also reachable from the host. However, each container uses its own `mnt` namespace, so further mounts (done within the container) will not automatically be visible.
iand675 14y ago
I'm not entirely sure that I understand what this does. Is it some sort of hybrid between provisioning automation and deployment automation?
- derefr 14y ago
  It's a white-labelling of dotCloud's implementation of Heroku's "slug" concept (https://devcenter.heroku.com/articles/slug-compiler https://devcenter.heroku.com/articles/slug-compiler): basically, a SquashFS image with a known SHA, storing a precompiled runtime+libraries+code artifact that will never change, able to be union-mounted atop a "base image" (a chroot filesystem, possibly also a known-SHA SquashFS image), then spun up as an ephemeral LXC container. I actually use the idea in my own ad-hoc deployment process; they're very convenient for ensuring repeatability. (As a side-note, this is an example of an interesting bit of game theory: in a niche, the Majority Player will tend to keep their tech proprietary to stay ahead, while the Second String will tend to release everything OSS in order to remove the Majority Player's advantages. This one is dotCloud taking a stab at Heroku, but you can also think of, for example, Atlassian--who runs Github-competitor Bitbucket--poking at Github by releasing a generic Git GUI client, whereas Github released a Github client.)
  - shykes 14y ago
    This is mostly accurate :) I will add that our implementation predates Heroku's. Using a generic container layer early on (first OpenVZ-based prototypes in 2009) is what allowed us to launch multi-language support a year before any other paas. It's also how we operate both application servers and databases with the same underlying codebase, and the same ops team.
- zdw 14y ago
  Containerization has been around a long time. Probably the start would be chroot, which was developed further by BSD's jails, Solaris zones, and Linux containers. Basically, these are all ways a way of running programs with a partitioned subset of access to the OS - in some cases, it may even appear to be an entire installation under that subset. This (and many of the other mentioned solutions) adds to that by being able to capture and deploy an environment for running a specific app.
Harkins 14y ago
This sounds like a reimplementation of virtual machines at the os layer instead of hardware layer.
- voidlogic 14y ago
  It sounds like they are just building on cgroups etc, which are already part of the Linux kernel. I would argue that virtual machines at a hypervisor/hardware level were just a hack for OSs not living up to their isolation promises/obligations. Strong OS level isolation implementations (cgroups, namespaces etc) allow people to put isolation back where it belongs, the OS. The job of the OS is to control the hardware, wrapping the OS is software to emulate hardware is ridiculous and VMs generally have much more performance overhead than isolation containers.
- laumars 14y ago
  Containers have existed as long as virtual machines have. FreeBSD implemented "Jails" back in the late 90s / yr2000. Linux also has OpenVZ and Solaris has Zones. If you couple a container with a CoW file system that supports snapshotting (eg ZFS or BtrFS), then you can have most of the features you'd expect from virtualisation but without as heavy footprint. Containers are an underrated and often forgotten solution in my opinion.
- batgaijin 14y ago
  lxc leverages hvm I think... someone correct me? edit: it's too early, sorry this has nothing to do with your post... but I hope someone does correct me about hvm.
duked 14y ago
That sounds like a definitive improvement over LXC specially the isolation properties but I'm not sure what is the added value of Docker compared to OpenVZ ? Any ideas ?
- zimbatm 14y ago
  My guess is that the container specification is orthogonal to the sandboxing feature. In fact they're using LXC.
  - shykes 14y ago
    Correct, Docker is currently based on lxc, but that is an implementation detail. In theory it could be ported to any process-level isolation tech with similar features: OpenVZ, Solaris Zones - you could also try using BSD jails although I don't know if they have all the required features. To answer the original question: Docker extends LXC with a higher-level API which operates at the process level. OpenVZ helps you create "mini-servers". Docker lets you forget about servers and manage processes.
    - zimbatm 14y ago
      Nice. Can't way to try it out. I've built a similar tool in Go for PandaStream to isolate or encoding processes.
allerratio 14y ago
Why should I use this when systemd offers the same (minus the AUFS root fs)?
- shykes 14y ago
  We get that question a lot... Until people start playing with it. Then they never ask it again :)
nwg 14y ago
Does anyone know, or can anyone speculate on exactly how the "heterogenous payloads" are specified?
- shykes 14y ago
  The payload is anything which can be recorded on the filesystem by running a unix command. For example: # Run command which adds your payload $ docker run base apt-get install curl 5b4a1ee8 # Commit the result to a new image $ docker commit 5b4a1ee8 nwg/base-with-curl # Run a command from the new image. Your payload is available! $ docker run nwg/base-with-curl curl http://www.google.com Docker doesn't care how the payload was added. apt-get, shell script, pip install, gem bundle... All the same. In the end it's just a change to the filesystem.
rlpb 14y ago
How is this different from what you can already do with lxc on, for example, Ubuntu Server?
- derefr 14y ago
  It's not that it's any different, it's that it's standardized. The idea is that a Docker container would be portable between different PaaS hosts (and from your own staging environment to those hosts!) without rebuilding, because they'd all be using the "Docker standard for deployment." A PaaS host saying they supported Docker would imply that they'd be using, for example, SquashFS for container format, AuFS instead of OverlayFS for union-mounts, LXC instead of OpenVZ/Xen/KVM for isolation, and any other set of things your container might subtly rely upon. The culmination of this, I imagine, would be a PaaS host allowing you to specify the "stuff" you want to run just by the URL of the container-image.
  - shykes 14y ago
    Correct. Docker is the direct result of dotCloud's experience running hundreds of thousands of containers in production over the last 2 years. We tried very hard to put it in a form factor which makes it useful beyond the traditional PaaS. We think Docker's API is a fundamental building block for running any process on the server.
  - batgaijin 14y ago
    What about virtsandbox from fedora? how does this project overlap with that? can this handle certain situations better?
  - peterwwillis 14y ago
    Doesn't a standard involve, you know, standards? AFAIK a product name is not a standard. What if the namespace changes? What if AuFS changes? What if LXC changes? Independently or all together? ABI changes? Version changes? Feature changes? Are all the licenses compatible? Will it ever support platforms other than just certain versions of Linux? Or languages other than Go? I don't see a standard. I see marketing for a product and a mailing list to collect potential customers. But maybe i'm missing something.
shykes 14y ago
Wow! Did not expect this to show up on HN before actual release! (I work at dotCloud). We're still polishing a few rough edges. If you want early access add your github ID to this thread and we'll add you right away!
- themckman 14y ago
  Would REALLY love to take a look at this. Github: 198d
- buster 14y ago
  Yay, here too! Github: buster
- GuiA 14y ago
  There was a lightning talk about Docker at Pycon; I'd assume that's where OP got the info from :)
  - ARothfusz 14y ago
    I've posted a video of the lightning talk here: https://plus.google.com/photos/115695491015706412558/albums/5856171619418221505/5856171619150829938 https://plus.google.com/photos/115695491015706412558/albums/... It should be public so you can view it even without a G+ account (I think)
  - shykes 14y ago
    That's possible. When we submitted the lightning talk, we pictured a poorly lit backroom with maybe 15 fellow kernel geeks. Instead, we presented in the main room to at least 500 people... So much for discreetly asking for feedback :)
- efnx 14y ago
  Yes please and thank you! Github: schell
- deleted 14y ago
  [deleted]
- 14y ago
buster 14y ago
Wow, i need this! What would be the requirements for docker? I suppose it wouldn't run on linux 2.6.x? (CentOS 5.x)? Any other requirements?
- shykes 14y ago
  The main requirement is a modern kernel with the aufs module. We do most of our testing on Ubuntu 12.04 and 12.10. But any modern distro should be fine. There are a few people testing it on CentOS as I write this.
nicolast 14y ago
I recently changed our Jenkins CI infrastructure to use something like this: all jobs run in their own LXC container, using BTRFS and its snapshots/clones instead of AUFS. Works like a charm! The script is available at https://github.com/Incubaid/jenkins-lxc https://github.com/Incubaid/jenkins-lxc (no docs for now, and several improvements are possible). It expects CI job scripts to be contained in the job repository, e.g. https://github.com/Incubaid/arakoon/tree/1.6/jenkins https://github.com/Incubaid/arakoon/tree/1.6/jenkins
- niall_ohiggins 14y ago
  That's pretty awesome. At FrozenRidge.co, we work on our own CI server called Strider and have actually integrated Docker directly: You can read about it at: http://blog.frozenridge.co/next-generation-continuous-integration-deployment-with-dotclouds-docker-and-strider/ http://blog.frozenridge.co/next-generation-continuous-integr...
- jordanthoms 14y ago
  This is great, I've been wanting to do the same thing for our jenkins setup.
theatrus2 14y ago
Whats the interaction between systems based on Mesos, which can (and many do) use containers? Is this really designed for more multi-tenancy with lower trust over same-org clouds?
- shykes 14y ago
  Docker solves the problem of running things in a repeatable and infrastructure-agnostic way. Mesos solves the problem of telling many nodes what to run and when. In other words, Docker + Mesos is a killer combo. There is already experimentation underway to use Docker as an execution engine for Mesos.
pshc 14y ago
Will there be any possibility of running dev instances in OS X? Perhaps we'll be able to do brew install docker-compat at some point in the future and get a best-effort emulation layer even though the Linux APIs are missing. I hate messing around with virtualization.
- niall_ohiggins 14y ago
  I run a Docker host fine on OS X w/ Vagrant. The client runs on OS X but not Linux Containers.
zobzu 14y ago
Note that LXC DOES NOT PROVIDE SECURITY. It provides resource separation (to a point) and so on. Breaking out of a filesystem container is as easy as creating a root block device. Breaking out of a network container is as easy as creating a network device And in all cases, you can just inject memory, load lkms, etc. That's without mentioning the amount of weekly CVEs for Linux namespaces.
- shykes 14y ago
  (Copying my answer to a similar question) Yes, you probably don't want to run untrusted code with root privileges inside a container if anything valuable is running on the same host. However if that code is trusted, or if you're running it as an unprivileged user, or if nothing else of importance is sharing the same host, then I would not hesitate to use them. Containers are awesome because they represent a logical component in your software stack. You can also use them as a unit of hardware resource allocation and multi-tenancy, but you don't have to: you can map a container 1-to-1 to a physical box, for example. But the logical unit remains the same regardless of the underlying hardware and multi-tenancy setup, which is truly awesome. EDIT: details on multi-tenancy.
  - zobzu 14y ago
    you're trying to justify the use of lxc for security, IMO. Your webpage does state "strong guarantees of isolation" if you're sharing nothing of importance on the host, then, you don't really need LXC, unless you don't know how to setup mysql with more than one database, nginx with more than one virtual host, yada yada. Here's the trick: you CAN use LXC and SUPPLEMENT it by something providing security such as SELinux.
    - duked 14y ago
      or use OpenVZ which has been designed with security in mind ;)
      - jpetazzo 14y ago
wereHamster 14y ago
I'm interested to see how that standardization works across linux distributions. I mean, different distros use different versions of various libraries (openssl for example). So if your app links with libssl.so.X but the host only provides libssl.so.Y, then your app won't work. Of course, you could bundle libssl with your app. But then the standardization is at the level of kernel/libc ABI. In which case the container is basically a full LXC guest. But then why standardize an image format if you can create a small script which builds the image with lxc-create + installs whatever else necessary for your app. That script will be much smaller than the full image, even a barebones ubuntu lxc guest (debootstrap quantal) is ~400MB.
- wmf 14y ago
  Their base layer needs to specify exact versions of libraries. Deploying from images should be much faster and less fragile (oops, is Github down?) than from scripts.
  - shykes 14y ago
    Absolutely. Think of a Docker container as a universal build artifact. Your build step can be an arbitrary sequence of unix commands (download dependencies, build libssl, run shell scripts etc.). Docker can freeze the result of that build and guarantee that it will run in a repeatable and self-contained way, no matter where you run it. So you get clean separation of build and run, which is a hugely important part of reliable deployment.
    - wereHamster 14y ago
      So docker container ABI is basically the kernel? You just create a filesystem image and docker starts that as LXC guest. You could build a statically compiled binary and put that as /bin/init into the container image, right?
      - shykes 14y ago
        Yes! Exactly :) You wouldn't need to save it as /sbin/init. You would just type: $ docker run MYIMAGE /path/to/my/static/binary Here's the smallest image I've personally used to run a docker container: http://get.docker.io/images/busybox http://get.docker.io/images/busybox
NateDad 14y ago
Not to be a stick in the mud, but you're using a copyrighted image as your logo without any attribution or acknowledgement of the original owner of the copyright (the Lego Group). You should probably fix that.
- shykes 14y ago
  You're right! We put it up there when it was an internal project, it's probably time to take care of that. Taking it down until we found a correct way to do it. I'm a copyright noob: would simply acknowledging the copyright owner be enough and fall under fair use, or should we not use it unless we get written permission?
  - NateDad 14y ago
    I am not a copyright expert, however I am familiar with the Lego Group (the company that produces Lego toys) ... the set is pretty old (1986), which means they're not worried about making money off it. If you just stick a disclaimer after it that it's copyright 1986 the Lego Group, that's probably fine, especially since this looks to be an open source project, so you're not going to be charging people for something with their picture on it. This page has what is very likely the original image: http://www.peeron.com/scans/7823-1/ http://www.peeron.com/scans/7823-1/ Peeron.com has special permission directly from Lego to display the images, so if you wanted to be extra careful you could email dan@peeron.com and ask for permission to deep link to their picture (they'd probably say yes, the admin is a linux geek too). But honestly, a simple copyright disclaimer is probably fine. Lego won't reach out and swat you even if they do decide they don't like it, they'll just ask you to take it down.
  - binarycrusader 14y ago
    I would avoid using anyone else's images without their explicit permission. It's just safer that way.
undoware 14y ago
So what we have here is a presently-closed-source, linux-only implementation of closure copying, which is a tiny part of Nix, which is the platform-agnostic package manager behind the elegant NixOS (nixos.org). Or is there something I'm not getting? If it's more platform-dependent, less well-tested, and (ostensibly temporarily) less open, then why is this on the top of my page?
- shykes 14y ago
  As a fan of Nix, and someone who expected a lot from Conary/rPath back in the day, I can provide at least a partial answer: these distros are great as long as you use their packaging tools for everything. Which is simply not practical. There's a reason Nix and rPath never took off, and it's the same reason developers use virtualenv and rvm instead of system packages. They don't like being forced to use a single packaging tool for everything. Docker doesn't have an opinion about how you package things. It only cares about the resulting changes on the filesystem. So you are free to use the best tool for each job. By the way this means you can use Docker and Nix together. I would love to see that :)
epynonymous 14y ago
fyi, aufs which is the union mount filesystem required by docker (and cloud foundry warden), is not well supported on rhel, centos, and other linux distros, seems mainly support for ubuntu.
- kalmar 14y ago
  More than this, Ubuntu are trying to remove it from their distribution in favour of overlayfs [0]. As mentioned there, the reason is that aufs is not and will not be part of the mainline kernel. Precise 12.04 was going to be without AUFS, but some issues cropped up, keeping it around. It now looks like overlayfs will make it into the 3.10 mainline kernel [1], so it may be a better choice in the future. I think that the Under the hood stuff on docker's page are implementation details that can change, so a switch to overlayfs when that becomes more suitable could be possible. (Confirmation from the dotClouders present would be apprecaited.) [0] https://lists.ubuntu.com/archives/ubuntu-devel/2012-February/034850.html https://lists.ubuntu.com/archives/ubuntu-devel/2012-February... [1] http://lwn.net/Articles/542707/ http://lwn.net/Articles/542707/
felixr 14y ago
curl get.docker.io | sudo sh Has anybody tried this?
rolandtritsch 14y ago
GitHub: rolandtritsch

273 points by radimm 14y ago

andyl 14y ago

Looks cool. When is it gonna be open sourced?

niggler 14y ago
"Docker will be open source soon" Soon could be in tomorrow, a week, a month, a year, or longer ...
- shykes 14y ago
  ETA 2 weeks maximum.

asadjb 14y ago

I'm not familiar with any of the technologies used in this. Anybody care to comment on how strong the isolations would be security wise, compared to normal virtualization? If the security is almost at par and the isolation is good enough that one bad process can't bring the whole system down, might this be a good alternative to virtualization, since I imagine it would definitely use less resources.

scarmig 14y ago
Back in January I got a new laptop, installed Arch on it, got it all nicely set up. And I decided it was high time to start playing with LXC because container virtualization seems extremely promising to me. Created an Ubuntu container, seemed to work fine, and then used lxc-destroy, which took some time. It destroyed my entire file system. I have no clue how the hell it happened--it floors me that something like that would be possible--and I suspect it's probably simply the result of a newbie like myself somehow misusing userspace tools. But it was enough to turn me off of it for the time being.
zdw 14y ago
Barring kernel bugs, it should prevent against the mentioned resource monopolization issues. Normal virtualization is pretty resource wasteful, especially if the guests are not hypervisor aware. Getting away from huge per-VM block devices is a step in the right direction.
teraflop 14y ago
The Linux namespace stuff is evolving pretty fast, and I personally wouldn't trust it as the main line of defense for anything important. With virtualization, a buggy or malicious guest is still limited to its sandbox unless there's a flaw in the hypervisor itself. With containers/namespaces, the host and guest are just different sets of processes that see different "views" of the same kernel, so bugs are much more likely to be exploitable. Plus, if you enable user namespaces, some code paths (like on-demand filesystem module loading) that used to require root are now available to unprivileged users. There's already been at least one local root exploit that almost made it into 3.9: https://lkml.org/lkml/2013/3/13/361 https://lkml.org/lkml/2013/3/13/361
- derefr 14y ago
  > The Linux namespace stuff is evolving pretty fast, and I personally wouldn't trust it as the main line of defense for anything important. If I recall, Heroku uses cgroups (EDIT: and namespaces) exclusively for multitenant isolation (and by the looks of this, dotCloud does too), so that's two big votes in the "if it's good enough for them" category.
  - teraflop 14y ago
    Sure, but cgroups and namespaces are kind-of-orthogonal features that both happen to be useful for making container-like things. cgroups are for limiting resource usage; namespaces are for providing the illusion of root access while actually being in a sandboxed environment. And as far as I'm aware (speaking as an interested non-expert, so please correct me if I'm wrong) cgroups have no effect on permissions, whereas UID namespaces required a lot of very invasive changes to the kernel.
    - deleted 14y ago
      [deleted]
    - derefr 14y ago
      Oh, no, you're right--I meant to include both things. I should have just been lazy and said "all that tech LXC uses." ;)
wslh 14y ago
I am familiar with Microsoft App-V, VMWare ThinApp, and Symantec Workspace Virtualization. They can help you as a security sandbox but not as a full protection. A virtual machine will be much more secure (and theoretically very strong), although there are security bugs there that enable you to escape it. Those products work at two levels: using filtering drivers for registry and the filesystem, and hooking into the Windows operating system API.
- laumars 14y ago
  Virtual machines are not more secure. In fact there's been more documented attacks where root access on a guest VM has gained shell access on the host, than there's been against containers. This doesn't mean that containers are more secure than VMs either. Attacking VMs attracts more security researchers from what I've seen (but I may be wrong on that point). However whether your running a container or a virtual machine, you still need some shared processes (eg the 'ticks' of a system clock) and with any sufficiently complicated code WILL have bugs that can be potentially exploited. However the crux of the matter is regardless of whether you're running containers or full blown virtual machines, you cannot escape out of the sandbox without having elevated privileges on the guest to begin with. And if an attacker has that, then you've already lost - regardless of whether the attacker can or cannot escape the sandbox. Lastly, I'm not sure if you're aware of this or not, but this is a Linux solution and has nothing to do with Windows (I only say this because your post seemed tailored towards Windows-hosted virtualisation)
  - wslh 14y ago
    Are you saying that both approaches have the same level of security or probable insecurities? or that you can't currently estimate the difference? Even being aware that this is a Linux solution I mentioned the Windows technologies that I know technically.
    - laumars 14y ago
      > Are you saying that both approaches have the same level of security or probable insecurities? or that you can't currently estimate the difference? A bit of both, but mostly the former. In practical terms, they both have the same level of security. But -as with any software- something could be published tomorrow exposing some massive flaw that totally blows one or the other out of the water. However neither offer any technical advantage over the other from a security stand point and from a practical perspective, the real question of security is whether your guest OSs are locked down to begin with (eg it's no good arguing which home security system is the most effective if you leave the front door open to begin with). > Even being aware that this is a Linux solution I mentioned the Windows technologies that I know technically. That's fair enough and I had suspected that was the case. I just wanted to make sure that we were both talking about the same thing :)
trotsky 14y ago
Container based virtualization can provide an impressive amount of isolation while improving density dramatically on light duty loads over virtualization. Solaris zones are very well regarded and are used for multi-tenant by Joyent, and many many linux hosts provide multi-tenant solutions based on virtuozzo which predates linux containers by a good number of years. The main theoretical difference between hypervisor isolation and container isolation is one sits above the kernel, so a kernel level exploit only applies to a single virtual machine. With containers you're relying on the kernel to provide the isolation so you are still subject to (some) kernel level exploits. Practically linux containers (the mainline implementation) have only provided full isolation in recent patches and probably shouldn't be considered full shaken out for something like full in the wild root level multi-tenant access. They are super for application isolation for delivery of multiple single tenant workloads on one machine though - something people use hypervisors for quite a bit. The resources used can be a small fraction of what you're committing to with a hypervisor.
- bcantrill 14y ago
  As trotsky mentions, we at Joyent are fervent believers in OS-based virtualization -- to the point that in SmartOS, we run hardware virtualization within an OS container. There are many reasons to favor OS-based virtualization over hardware-based virtualization, but first among these (in my opinion) is DRAM utilization: with OS-based virtualization, all unused DRAM is available to the system at large, and in the SmartOS case is used as adaptive replacement cache (ARC) that benefits all tenants. Given that few tenants consume every byte of their allocated DRAM, this alone leads to huge efficiencies from both the perspective of the cloud operator and the cloud user -- a higher-performing, higher-margin service. By contrast, for hardware-based virtualization, unused DRAM remains with the guest and is simply wasted (kludges like kernel samepage mapping and memory ballooning notwithstanding). DRAM isn't the only win, of course: for every other resource in the system (CPU, network, disk), OS-based virtualization offers tremendous (and insurmountable) efficiency advantages over hardware-based virtualization -- and it's great to see others make the same realization! For more details on the relative performance of OS-based virtualization, hardware-based virtualization and para-virtualization, see my colleague Brendan Gregg's excellent blog post on the subject[1]. [1] http://dtrace.org/blogs/brendan/2013/01/11/virtualization-performance-zones-kvm-xen/ http://dtrace.org/blogs/brendan/2013/01/11/virtualization-pe...
bradrydzewski 14y ago
This is still technically a virtualization technique, known as "operating system-level virtualization". http://en.wikipedia.org/wiki/Operating_system-level_virtualization http://en.wikipedia.org/wiki/Operating_system-level_virtuali... Here are some of the technologies explained: cgroups: Linux kernel feature that allows resource limiting and metering, as well as process isolation. The process isolation, also called namespaces, is important because it prevents a process from seeing or terminating other running processes. lxc: this is a utility that glues together cgroups and chroots to provide virtualization. It helps you easily setup a guest OS by downloading your favorite distro and unpacking it (kind of like debootstrap). It can then "boot" the guest OS by starting it's "init" process. The init process runs in its own namespace, inside a chroot. This is why they call LXC a chroot on steroids. It does everything that chroot does, with full process isolation and metering. aufs: this is sometimes called a "stacked" file system. It allows you to mount one file system on top of another. Why is this important? Because if you are managing a large number of virtual machines, each one with 1GB+ OS, it uses a lot of disk space. Also, the slowest part of creating a new container is copying the distro (can take up to 30 seconds). Using something like AUFS gives you much better performance. So what about security? Well, like every (relatively) new technology LXC has its issues. If you use Ubuntu 12.04 they provide a set of Apparmor scripts to mitigate known security risks (like disabling reboot or shutdown commands inside containers, and write access to the /sys filesystem).

zdw 14y ago

Are the aufs filesystems easily navigable from the host OS, ala Solaris's zones feature?

zimbatm 14y ago
The aufs mount is (probably) only mounted on the guest's VFS so I'm not sure how you would access it from the host.
- shykes 14y ago
  Docker lets you visualize changes on any container's filesystem live, as they happen. It also lets you snapshot changes from any container into a new image, and immediately start running new containers from that image. No manual post-processing of the image, no configuration files to templatize. The whole thing is 2 commands and maybe half a second.
- jpetazzo 14y ago
  The AUFS mountpoint is also reachable from the host. However, each container uses its own `mnt` namespace, so further mounts (done within the container) will not automatically be visible.

iand675 14y ago

I'm not entirely sure that I understand what this does. Is it some sort of hybrid between provisioning automation and deployment automation?

derefr 14y ago
It's a white-labelling of dotCloud's implementation of Heroku's "slug" concept (https://devcenter.heroku.com/articles/slug-compiler https://devcenter.heroku.com/articles/slug-compiler): basically, a SquashFS image with a known SHA, storing a precompiled runtime+libraries+code artifact that will never change, able to be union-mounted atop a "base image" (a chroot filesystem, possibly also a known-SHA SquashFS image), then spun up as an ephemeral LXC container. I actually use the idea in my own ad-hoc deployment process; they're very convenient for ensuring repeatability. (As a side-note, this is an example of an interesting bit of game theory: in a niche, the Majority Player will tend to keep their tech proprietary to stay ahead, while the Second String will tend to release everything OSS in order to remove the Majority Player's advantages. This one is dotCloud taking a stab at Heroku, but you can also think of, for example, Atlassian--who runs Github-competitor Bitbucket--poking at Github by releasing a generic Git GUI client, whereas Github released a Github client.)
- shykes 14y ago
  This is mostly accurate :) I will add that our implementation predates Heroku's. Using a generic container layer early on (first OpenVZ-based prototypes in 2009) is what allowed us to launch multi-language support a year before any other paas. It's also how we operate both application servers and databases with the same underlying codebase, and the same ops team.
zdw 14y ago
Containerization has been around a long time. Probably the start would be chroot, which was developed further by BSD's jails, Solaris zones, and Linux containers. Basically, these are all ways a way of running programs with a partitioned subset of access to the OS - in some cases, it may even appear to be an entire installation under that subset. This (and many of the other mentioned solutions) adds to that by being able to capture and deploy an environment for running a specific app.

Harkins 14y ago

This sounds like a reimplementation of virtual machines at the os layer instead of hardware layer.

voidlogic 14y ago
It sounds like they are just building on cgroups etc, which are already part of the Linux kernel. I would argue that virtual machines at a hypervisor/hardware level were just a hack for OSs not living up to their isolation promises/obligations. Strong OS level isolation implementations (cgroups, namespaces etc) allow people to put isolation back where it belongs, the OS. The job of the OS is to control the hardware, wrapping the OS is software to emulate hardware is ridiculous and VMs generally have much more performance overhead than isolation containers.
laumars 14y ago
Containers have existed as long as virtual machines have. FreeBSD implemented "Jails" back in the late 90s / yr2000. Linux also has OpenVZ and Solaris has Zones. If you couple a container with a CoW file system that supports snapshotting (eg ZFS or BtrFS), then you can have most of the features you'd expect from virtualisation but without as heavy footprint. Containers are an underrated and often forgotten solution in my opinion.
batgaijin 14y ago
lxc leverages hvm I think... someone correct me? edit: it's too early, sorry this has nothing to do with your post... but I hope someone does correct me about hvm.

duked 14y ago

That sounds like a definitive improvement over LXC specially the isolation properties but I'm not sure what is the added value of Docker compared to OpenVZ ? Any ideas ?

zimbatm 14y ago
My guess is that the container specification is orthogonal to the sandboxing feature. In fact they're using LXC.
- shykes 14y ago
  Correct, Docker is currently based on lxc, but that is an implementation detail. In theory it could be ported to any process-level isolation tech with similar features: OpenVZ, Solaris Zones - you could also try using BSD jails although I don't know if they have all the required features. To answer the original question: Docker extends LXC with a higher-level API which operates at the process level. OpenVZ helps you create "mini-servers". Docker lets you forget about servers and manage processes.
  - zimbatm 14y ago
    Nice. Can't way to try it out. I've built a similar tool in Go for PandaStream to isolate or encoding processes.

allerratio 14y ago

Why should I use this when systemd offers the same (minus the AUFS root fs)?

shykes 14y ago
We get that question a lot... Until people start playing with it. Then they never ask it again :)

nwg 14y ago

Does anyone know, or can anyone speculate on exactly how the "heterogenous payloads" are specified?

shykes 14y ago
The payload is anything which can be recorded on the filesystem by running a unix command. For example: # Run command which adds your payload $ docker run base apt-get install curl 5b4a1ee8 # Commit the result to a new image $ docker commit 5b4a1ee8 nwg/base-with-curl # Run a command from the new image. Your payload is available! $ docker run nwg/base-with-curl curl http://www.google.com Docker doesn't care how the payload was added. apt-get, shell script, pip install, gem bundle... All the same. In the end it's just a change to the filesystem.

rlpb 14y ago

How is this different from what you can already do with lxc on, for example, Ubuntu Server?

derefr 14y ago
It's not that it's any different, it's that it's standardized. The idea is that a Docker container would be portable between different PaaS hosts (and from your own staging environment to those hosts!) without rebuilding, because they'd all be using the "Docker standard for deployment." A PaaS host saying they supported Docker would imply that they'd be using, for example, SquashFS for container format, AuFS instead of OverlayFS for union-mounts, LXC instead of OpenVZ/Xen/KVM for isolation, and any other set of things your container might subtly rely upon. The culmination of this, I imagine, would be a PaaS host allowing you to specify the "stuff" you want to run just by the URL of the container-image.
- shykes 14y ago
  Correct. Docker is the direct result of dotCloud's experience running hundreds of thousands of containers in production over the last 2 years. We tried very hard to put it in a form factor which makes it useful beyond the traditional PaaS. We think Docker's API is a fundamental building block for running any process on the server.
- batgaijin 14y ago
  What about virtsandbox from fedora? how does this project overlap with that? can this handle certain situations better?
- peterwwillis 14y ago
  Doesn't a standard involve, you know, standards? AFAIK a product name is not a standard. What if the namespace changes? What if AuFS changes? What if LXC changes? Independently or all together? ABI changes? Version changes? Feature changes? Are all the licenses compatible? Will it ever support platforms other than just certain versions of Linux? Or languages other than Go? I don't see a standard. I see marketing for a product and a mailing list to collect potential customers. But maybe i'm missing something.

shykes 14y ago

Wow! Did not expect this to show up on HN before actual release! (I work at dotCloud). We're still polishing a few rough edges. If you want early access add your github ID to this thread and we'll add you right away!

themckman 14y ago
Would REALLY love to take a look at this. Github: 198d
buster 14y ago
Yay, here too! Github: buster
GuiA 14y ago
There was a lightning talk about Docker at Pycon; I'd assume that's where OP got the info from :)
- ARothfusz 14y ago
  I've posted a video of the lightning talk here: https://plus.google.com/photos/115695491015706412558/albums/5856171619418221505/5856171619150829938 https://plus.google.com/photos/115695491015706412558/albums/... It should be public so you can view it even without a G+ account (I think)
- shykes 14y ago
  That's possible. When we submitted the lightning talk, we pictured a poorly lit backroom with maybe 15 fellow kernel geeks. Instead, we presented in the main room to at least 500 people... So much for discreetly asking for feedback :)
efnx 14y ago
Yes please and thank you! Github: schell
deleted 14y ago
[deleted]
14y ago

buster 14y ago

Wow, i need this! What would be the requirements for docker? I suppose it wouldn't run on linux 2.6.x? (CentOS 5.x)? Any other requirements?

shykes 14y ago
The main requirement is a modern kernel with the aufs module. We do most of our testing on Ubuntu 12.04 and 12.10. But any modern distro should be fine. There are a few people testing it on CentOS as I write this.

nicolast 14y ago

I recently changed our Jenkins CI infrastructure to use something like this: all jobs run in their own LXC container, using BTRFS and its snapshots/clones instead of AUFS. Works like a charm! The script is available at https://github.com/Incubaid/jenkins-lxc https://github.com/Incubaid/jenkins-lxc (no docs for now, and several improvements are possible). It expects CI job scripts to be contained in the job repository, e.g. https://github.com/Incubaid/arakoon/tree/1.6/jenkins https://github.com/Incubaid/arakoon/tree/1.6/jenkins

niall_ohiggins 14y ago
That's pretty awesome. At FrozenRidge.co, we work on our own CI server called Strider and have actually integrated Docker directly: You can read about it at: http://blog.frozenridge.co/next-generation-continuous-integration-deployment-with-dotclouds-docker-and-strider/ http://blog.frozenridge.co/next-generation-continuous-integr...
jordanthoms 14y ago
This is great, I've been wanting to do the same thing for our jenkins setup.

theatrus2 14y ago

Whats the interaction between systems based on Mesos, which can (and many do) use containers? Is this really designed for more multi-tenancy with lower trust over same-org clouds?

shykes 14y ago
Docker solves the problem of running things in a repeatable and infrastructure-agnostic way. Mesos solves the problem of telling many nodes what to run and when. In other words, Docker + Mesos is a killer combo. There is already experimentation underway to use Docker as an execution engine for Mesos.

pshc 14y ago

Will there be any possibility of running dev instances in OS X? Perhaps we'll be able to do brew install docker-compat at some point in the future and get a best-effort emulation layer even though the Linux APIs are missing. I hate messing around with virtualization.

niall_ohiggins 14y ago
I run a Docker host fine on OS X w/ Vagrant. The client runs on OS X but not Linux Containers.

zobzu 14y ago

Note that LXC DOES NOT PROVIDE SECURITY. It provides resource separation (to a point) and so on. Breaking out of a filesystem container is as easy as creating a root block device. Breaking out of a network container is as easy as creating a network device And in all cases, you can just inject memory, load lkms, etc. That's without mentioning the amount of weekly CVEs for Linux namespaces.

shykes 14y ago
(Copying my answer to a similar question) Yes, you probably don't want to run untrusted code with root privileges inside a container if anything valuable is running on the same host. However if that code is trusted, or if you're running it as an unprivileged user, or if nothing else of importance is sharing the same host, then I would not hesitate to use them. Containers are awesome because they represent a logical component in your software stack. You can also use them as a unit of hardware resource allocation and multi-tenancy, but you don't have to: you can map a container 1-to-1 to a physical box, for example. But the logical unit remains the same regardless of the underlying hardware and multi-tenancy setup, which is truly awesome. EDIT: details on multi-tenancy.
- zobzu 14y ago
  you're trying to justify the use of lxc for security, IMO. Your webpage does state "strong guarantees of isolation" if you're sharing nothing of importance on the host, then, you don't really need LXC, unless you don't know how to setup mysql with more than one database, nginx with more than one virtual host, yada yada. Here's the trick: you CAN use LXC and SUPPLEMENT it by something providing security such as SELinux.
  - duked 14y ago
    or use OpenVZ which has been designed with security in mind ;)
    - jpetazzo 14y ago

wereHamster 14y ago

I'm interested to see how that standardization works across linux distributions. I mean, different distros use different versions of various libraries (openssl for example). So if your app links with libssl.so.X but the host only provides libssl.so.Y, then your app won't work. Of course, you could bundle libssl with your app. But then the standardization is at the level of kernel/libc ABI. In which case the container is basically a full LXC guest. But then why standardize an image format if you can create a small script which builds the image with lxc-create + installs whatever else necessary for your app. That script will be much smaller than the full image, even a barebones ubuntu lxc guest (debootstrap quantal) is ~400MB.

wmf 14y ago
Their base layer needs to specify exact versions of libraries. Deploying from images should be much faster and less fragile (oops, is Github down?) than from scripts.
- shykes 14y ago
  Absolutely. Think of a Docker container as a universal build artifact. Your build step can be an arbitrary sequence of unix commands (download dependencies, build libssl, run shell scripts etc.). Docker can freeze the result of that build and guarantee that it will run in a repeatable and self-contained way, no matter where you run it. So you get clean separation of build and run, which is a hugely important part of reliable deployment.
  - wereHamster 14y ago
    So docker container ABI is basically the kernel? You just create a filesystem image and docker starts that as LXC guest. You could build a statically compiled binary and put that as /bin/init into the container image, right?
    - shykes 14y ago
      Yes! Exactly :) You wouldn't need to save it as /sbin/init. You would just type: $ docker run MYIMAGE /path/to/my/static/binary Here's the smallest image I've personally used to run a docker container: http://get.docker.io/images/busybox http://get.docker.io/images/busybox

NateDad 14y ago

Not to be a stick in the mud, but you're using a copyrighted image as your logo without any attribution or acknowledgement of the original owner of the copyright (the Lego Group). You should probably fix that.

shykes 14y ago
You're right! We put it up there when it was an internal project, it's probably time to take care of that. Taking it down until we found a correct way to do it. I'm a copyright noob: would simply acknowledging the copyright owner be enough and fall under fair use, or should we not use it unless we get written permission?
- NateDad 14y ago
  I am not a copyright expert, however I am familiar with the Lego Group (the company that produces Lego toys) ... the set is pretty old (1986), which means they're not worried about making money off it. If you just stick a disclaimer after it that it's copyright 1986 the Lego Group, that's probably fine, especially since this looks to be an open source project, so you're not going to be charging people for something with their picture on it. This page has what is very likely the original image: http://www.peeron.com/scans/7823-1/ http://www.peeron.com/scans/7823-1/ Peeron.com has special permission directly from Lego to display the images, so if you wanted to be extra careful you could email dan@peeron.com and ask for permission to deep link to their picture (they'd probably say yes, the admin is a linux geek too). But honestly, a simple copyright disclaimer is probably fine. Lego won't reach out and swat you even if they do decide they don't like it, they'll just ask you to take it down.
- binarycrusader 14y ago
  I would avoid using anyone else's images without their explicit permission. It's just safer that way.

undoware 14y ago

So what we have here is a presently-closed-source, linux-only implementation of closure copying, which is a tiny part of Nix, which is the platform-agnostic package manager behind the elegant NixOS (nixos.org). Or is there something I'm not getting? If it's more platform-dependent, less well-tested, and (ostensibly temporarily) less open, then why is this on the top of my page?

shykes 14y ago
As a fan of Nix, and someone who expected a lot from Conary/rPath back in the day, I can provide at least a partial answer: these distros are great as long as you use their packaging tools for everything. Which is simply not practical. There's a reason Nix and rPath never took off, and it's the same reason developers use virtualenv and rvm instead of system packages. They don't like being forced to use a single packaging tool for everything. Docker doesn't have an opinion about how you package things. It only cares about the resulting changes on the filesystem. So you are free to use the best tool for each job. By the way this means you can use Docker and Nix together. I would love to see that :)

epynonymous 14y ago

fyi, aufs which is the union mount filesystem required by docker (and cloud foundry warden), is not well supported on rhel, centos, and other linux distros, seems mainly support for ubuntu.

kalmar 14y ago
More than this, Ubuntu are trying to remove it from their distribution in favour of overlayfs [0]. As mentioned there, the reason is that aufs is not and will not be part of the mainline kernel. Precise 12.04 was going to be without AUFS, but some issues cropped up, keeping it around. It now looks like overlayfs will make it into the 3.10 mainline kernel [1], so it may be a better choice in the future. I think that the Under the hood stuff on docker's page are implementation details that can change, so a switch to overlayfs when that becomes more suitable could be possible. (Confirmation from the dotClouders present would be apprecaited.) [0] https://lists.ubuntu.com/archives/ubuntu-devel/2012-February/034850.html https://lists.ubuntu.com/archives/ubuntu-devel/2012-February... [1] http://lwn.net/Articles/542707/ http://lwn.net/Articles/542707/

felixr 14y ago

curl get.docker.io | sudo sh Has anybody tried this?

rolandtritsch 14y ago

GitHub: rolandtritsch