7 ms·

Docker joins the Linux Foundation

97 points by julien421 13y ago

batgaijin 13y ago
I'm curious why the fuck it took until 2013 to get something comparable to freebsd jails/solaris zones. I mean, congratulation guys, but why the hell didn't this happen earlier? I know I'm not the brightest bulb in the room but this has been a fundamentally dangerous feature to be missing. I don't get how anyone pretends that their Linux boxes are secure without some extremely personal contrived creations.
- strmpnk 13y ago
  Just a guess: Vendors certainly seemed more focussed on getting people to use complicated virtualization schemes with proprietary hypervisors. I wonder if it was more a matter of man-power at that point. In comparison, Sun was already selling systems so the value of selling virtualization separately so zones may have ended up being an elegant integrated solution. I'm not sure of the FreeBSD jail history but there may be a similar story to tell if you look at the source of the contributions.
- fotcorn 13y ago
  There is OpenVZ, LXC and Linux-VServer which are the same as FreeBSD jails but for the Linux kernel.
  - DASD 13y ago
    There was also another predcessor, UML(User Mode Linux). This was widely available and offered such as by Linode before they moved to Xen.
    - tmzt 13y ago
      UML is a little different, using a Linux kernel which relies on the system calls of a parent kernel. The other technologies you mentioned, and LXC which docker is built on, have a single unified kernel which interprets paths and other namespaces in system calls from different process trees differently.
- justincormack 13y ago
  Here is a 2006 article about it - there is no doubt more stuff on LWN if you dig around https://lwn.net/Articles/179361/ https://lwn.net/Articles/179361/ The gist is that there were a lot of out of tree solutions (openvz is still used eg by Travis CI) but they had conflicting requirements and a lot of the suggestions were just "we need BSD jails" which is a very monolithic single purpose solution, like having a web server in the kernel (yes there was one) rather than having the kernel APIs in place so that was not needed. What we have ended up with (mainly cgroups and namespacing basically) is much more flexible in that each piece is for a separate thing, eg you can just namespace networks without full isolation (which is really useful for eg testing network code without messing with the real config). It did take a very very long time though.
- chubot 13y ago
  Here is my conjecture from a few days ago: https://news.ycombinator.com/item?id=5901649 https://news.ycombinator.com/item?id=5901649 In summary, I think it's because containers, although open source, are a relatively commercial technology. Interested in any comments from people were there in the 90's when lots of hosting providers started using Linux.
  - jpetazzo 13y ago
    raises a hand In the early 2000s, I was in the hosting business, and we started using UML (user-mode-linux), then vserver and openvz. When Xen came around, we switched to that, because it made more sense for us (and for our customers): more control, more "this is your machine" feeling. "Yes, but containers are better because they are lightweight!" Of course, but this idea was pointless back then, because people who wanted really lightweight things went to shared hosting (and for massive PHP hosting, containers can't beat a single Apache instance serving tens of thousands of vhosts; at least not yet). Maybe (maybe!) things changed with the rise of stacks like Python/Ruby/Node.js/etc for which there are no cheap shared hosting solutions like there is for PHP. Then containers are obviously cheaper than running your own VPS, and easier to setup (and isolate) than shared hosting on a single box.
- e1ven 13y ago
  I've been running OpenVZ for years - It works very well. LXC is built into the kernel directly, rather than a patch, which is very nice. So far as I can see, Docker is just some nice wrapper scripts around LXC, nothing particularly revolutionary in and of itself.
  - shykes 13y ago
    Correct, docker is not an implementation of linux containers. The heavy lifting is done by lxc (and alternatives like openvz). What docker brings is a way for backend developers to package and distribute their app into portable, sandboxed containers (think "iphone apps for the server"). This is only possible now that linux has decent container support, because it means the vast majority of servers out there can potentially run a docker container unmodified.
- Andys 13y ago
  Linux has had containers for ages. Docker has simply made them easier to use as a service. You should be asking why Solaris/BSD never had something as useful as docker despite having zones/jails. And linux supports overlay filesystems (to clone containers cheaply) without the need for ZFS.
- tmzt 13y ago
  The facilities in the kernel for doing this have gotten better progressively over the years. Support for separate namespaces, and adding more and more of those types of namespaces have only been added in the last few years.
general_failure 13y ago
EDIT: I had previously asked what exactly Docker is. https://www.youtube.com/watch?feature=player_embedded&v=wW9CAH9nSLs https://www.youtube.com/watch?feature=player_embedded&v=wW9C... gives a super quick overview. Very interesting technology.
- zgohr 13y ago
  Basically. One thing to consider is the fact that decoupling applications and services they interact with is generally considered a good practice. What services does a web application need? Generally speaking, a web process to handle requests, a database, and maybe some in-memory storage such as memcache. What docker can do is script the creation of these services as containers that can be deployed "anywhere." So say you have your web app now, and your 3 services are on a single server and the load is getting heavy because of database queries. Docker makes it extremely easy to reliably move the services between servers to handle the load better. Albeit this is a very simplified vision, I don't think the complete use cases for docker have been hashed out quite yet. tl;dr dev ops is about to get magnitudes easier.
- shykes 13y ago
  Yes, that's exactly the goal of docker: "build once, actually run anywhere". Docker is a portable container engine. It gives you, the developer, a way to package your app and all its dependencies into a portable container which can be deployed on any modern Linux server: local VM, bare metal server, EC2 instance, doesn't matter. Containers are completely sandboxed and do not interfere with each other (think "iPhone apps for the server"). The specificities of each machine (location, distro, storage, networking) are abstracted away so you can focus on what your application requires (a known filesystem state, tcp ports to receive connections...) without having to worry about how those requirements will be met on each individual machine. This abstraction is based on common unix concepts - files, processes, tcp connections - and is therefore not language-specific, so it doesn't matter what programming language your application is written in. To do this, Docker relies on a recent feature of the linux kernel called lxc. That's the part which allows sandboxing of applications from one another. The concept itself is not new - what is new is that it is now part of the mainstream kernel, which means almost every server in the world will soon be capable of running docker containers - hence the exciting possibility of "run anywhere".
kbar13 13y ago
sweet! does this mean that AUFS will be included in tree?
- shykes 13y ago
  That would be fantastic. Unfortunately I don't have the answer right now.
  - bradrydzewski 13y ago
    Have you considered using OverlayFS in a future version of Docker, instead of AUFS? It comes bundled with Ubuntu 12.04 and higher, and my understanding is that it could possibly get merged w/ kernel 3.10
    - shykes 13y ago
      Overlayfs has a static limit of 1 ro layer, which makes it useless out of the box for docker. It's also less proven in production than aufs. Aufs works great and has no decent alternative. Merging aufs into mainline feels like the logical thing to do, but clearly someone in the chain disagrees. It's disappointing but it is what it is. Btrfs seems like a more likely alternative for docker.
sciurus 13y ago
I'm glad that dotCloud is contributing money to the Linux Foundation. I'm curious how involved they are with linux kernel and plumbing development. It seems like the changes coming to cgroups, for example, could be a big deal for Docker. https://lwn.net/Articles/555920/ https://lwn.net/Articles/555920/

97 points by julien421 13y ago

I'm curious why the fuck it took until 2013 to get something comparable to freebsd jails/solaris zones. I mean, congratulation guys, but why the hell didn't this happen earlier? I know I'm not the brightest bulb in the room but this has been a fundamentally dangerous feature to be missing. I don't get how anyone pretends that their Linux boxes are secure without some extremely personal contrived creations.

strmpnk 13y ago
Just a guess: Vendors certainly seemed more focussed on getting people to use complicated virtualization schemes with proprietary hypervisors. I wonder if it was more a matter of man-power at that point. In comparison, Sun was already selling systems so the value of selling virtualization separately so zones may have ended up being an elegant integrated solution. I'm not sure of the FreeBSD jail history but there may be a similar story to tell if you look at the source of the contributions.
fotcorn 13y ago
There is OpenVZ, LXC and Linux-VServer which are the same as FreeBSD jails but for the Linux kernel.
- DASD 13y ago
  There was also another predcessor, UML(User Mode Linux). This was widely available and offered such as by Linode before they moved to Xen.
  - tmzt 13y ago
    UML is a little different, using a Linux kernel which relies on the system calls of a parent kernel. The other technologies you mentioned, and LXC which docker is built on, have a single unified kernel which interprets paths and other namespaces in system calls from different process trees differently.
justincormack 13y ago
Here is a 2006 article about it - there is no doubt more stuff on LWN if you dig around https://lwn.net/Articles/179361/ https://lwn.net/Articles/179361/ The gist is that there were a lot of out of tree solutions (openvz is still used eg by Travis CI) but they had conflicting requirements and a lot of the suggestions were just "we need BSD jails" which is a very monolithic single purpose solution, like having a web server in the kernel (yes there was one) rather than having the kernel APIs in place so that was not needed. What we have ended up with (mainly cgroups and namespacing basically) is much more flexible in that each piece is for a separate thing, eg you can just namespace networks without full isolation (which is really useful for eg testing network code without messing with the real config). It did take a very very long time though.
chubot 13y ago
Here is my conjecture from a few days ago: https://news.ycombinator.com/item?id=5901649 https://news.ycombinator.com/item?id=5901649 In summary, I think it's because containers, although open source, are a relatively commercial technology. Interested in any comments from people were there in the 90's when lots of hosting providers started using Linux.
- jpetazzo 13y ago
  raises a hand In the early 2000s, I was in the hosting business, and we started using UML (user-mode-linux), then vserver and openvz. When Xen came around, we switched to that, because it made more sense for us (and for our customers): more control, more "this is your machine" feeling. "Yes, but containers are better because they are lightweight!" Of course, but this idea was pointless back then, because people who wanted really lightweight things went to shared hosting (and for massive PHP hosting, containers can't beat a single Apache instance serving tens of thousands of vhosts; at least not yet). Maybe (maybe!) things changed with the rise of stacks like Python/Ruby/Node.js/etc for which there are no cheap shared hosting solutions like there is for PHP. Then containers are obviously cheaper than running your own VPS, and easier to setup (and isolate) than shared hosting on a single box.
e1ven 13y ago
I've been running OpenVZ for years - It works very well. LXC is built into the kernel directly, rather than a patch, which is very nice. So far as I can see, Docker is just some nice wrapper scripts around LXC, nothing particularly revolutionary in and of itself.
- shykes 13y ago
  Correct, docker is not an implementation of linux containers. The heavy lifting is done by lxc (and alternatives like openvz). What docker brings is a way for backend developers to package and distribute their app into portable, sandboxed containers (think "iphone apps for the server"). This is only possible now that linux has decent container support, because it means the vast majority of servers out there can potentially run a docker container unmodified.
Andys 13y ago
Linux has had containers for ages. Docker has simply made them easier to use as a service. You should be asking why Solaris/BSD never had something as useful as docker despite having zones/jails. And linux supports overlay filesystems (to clone containers cheaply) without the need for ZFS.
tmzt 13y ago
The facilities in the kernel for doing this have gotten better progressively over the years. Support for separate namespaces, and adding more and more of those types of namespaces have only been added in the last few years.

general_failure 13y ago

EDIT: I had previously asked what exactly Docker is. https://www.youtube.com/watch?feature=player_embedded&v=wW9CAH9nSLs https://www.youtube.com/watch?feature=player_embedded&v=wW9C... gives a super quick overview. Very interesting technology.

zgohr 13y ago
Basically. One thing to consider is the fact that decoupling applications and services they interact with is generally considered a good practice. What services does a web application need? Generally speaking, a web process to handle requests, a database, and maybe some in-memory storage such as memcache. What docker can do is script the creation of these services as containers that can be deployed "anywhere." So say you have your web app now, and your 3 services are on a single server and the load is getting heavy because of database queries. Docker makes it extremely easy to reliably move the services between servers to handle the load better. Albeit this is a very simplified vision, I don't think the complete use cases for docker have been hashed out quite yet. tl;dr dev ops is about to get magnitudes easier.
shykes 13y ago
Yes, that's exactly the goal of docker: "build once, actually run anywhere". Docker is a portable container engine. It gives you, the developer, a way to package your app and all its dependencies into a portable container which can be deployed on any modern Linux server: local VM, bare metal server, EC2 instance, doesn't matter. Containers are completely sandboxed and do not interfere with each other (think "iPhone apps for the server"). The specificities of each machine (location, distro, storage, networking) are abstracted away so you can focus on what your application requires (a known filesystem state, tcp ports to receive connections...) without having to worry about how those requirements will be met on each individual machine. This abstraction is based on common unix concepts - files, processes, tcp connections - and is therefore not language-specific, so it doesn't matter what programming language your application is written in. To do this, Docker relies on a recent feature of the linux kernel called lxc. That's the part which allows sandboxing of applications from one another. The concept itself is not new - what is new is that it is now part of the mainstream kernel, which means almost every server in the world will soon be capable of running docker containers - hence the exciting possibility of "run anywhere".

kbar13 13y ago

sweet! does this mean that AUFS will be included in tree?

shykes 13y ago
That would be fantastic. Unfortunately I don't have the answer right now.
- bradrydzewski 13y ago
  Have you considered using OverlayFS in a future version of Docker, instead of AUFS? It comes bundled with Ubuntu 12.04 and higher, and my understanding is that it could possibly get merged w/ kernel 3.10
  - shykes 13y ago
    Overlayfs has a static limit of 1 ro layer, which makes it useless out of the box for docker. It's also less proven in production than aufs. Aufs works great and has no decent alternative. Merging aufs into mainline feels like the logical thing to do, but clearly someone in the chain disagrees. It's disappointing but it is what it is. Btrfs seems like a more likely alternative for docker.

sciurus 13y ago

I'm glad that dotCloud is contributing money to the Linux Foundation. I'm curious how involved they are with linux kernel and plumbing development. It seems like the changes coming to cgroups, for example, could be a big deal for Docker. https://lwn.net/Articles/555920/ https://lwn.net/Articles/555920/