7 ms·

Linux Container Internals

201 points by deepakkarki 10y ago

siegecraft 10y ago
This is from February, which is ancient in Docker years, but the container history and references are quite useful.
- so0k 10y ago
  yet it covers containerd ;)
Demiurge 10y ago
The url is correct, 'Docker-internals', but the title, and further equivocation of 'Linux Containers' and 'Docker', is a bit confusing. I'm a happy lxc/lxd user, and had to stop reading because of the cognitive dissonance.
- agumonkey 10y ago
  I also thought it was about LXC. Sad.
  - deleted 10y ago
    [deleted]
  - skeptic2718 10y ago
    Why is it sad?
    - dozzie 10y ago
      Because LXC on a technical level is much simpler, doesn't make mess in network and filesystem setup, and is easy to understand. Docker does some heavy magic for things to work for programmers (the ones who can't be bothered to learn how ethernet bridging or IP routing work), so the whole thing feels brittle. In short, LXC is vastly underappreciated.
- so0k 10y ago
  Good point, updated the title. Thanks for the comments (and yes it's almost a year ago I wrote this post)
  - Demiurge 10y ago
    Thanks, feels like the nitpick, but I gave it another go, and I see some other changes now? Seems like a great amount of information.
    - so0k 10y ago
      don't think there are other changes - https://github.com/docker-saigon/docker-saigon.github.io/commits/master https://github.com/docker-saigon/docker-saigon.github.io/com...
      - Demiurge 10y ago
        neatz, I guess I just need to be more patient and set my expectations :)
SEJeff 10y ago
Also relevant: https://0xax.gitbooks.io/linux-insides/content/Cgroups/cgroups1.html https://0xax.gitbooks.io/linux-insides/content/Cgroups/cgrou...
kragniz 10y ago
If you want to learn about linux container internals, I can recommend just trying to implement one yourself in some random language for fun. I wrote a basic runtime in python that can run docker images: https://github.com/kragniz/omochabako https://github.com/kragniz/omochabako Quick demo: https://asciinema.org/a/77296?speed=2&autoplay=1 https://asciinema.org/a/77296?speed=2&autoplay=1
- decebalus1 10y ago
  that sounds pretty cool. Sorry for ignorance (heavy Windows background) but where does one start with something like that? Is there a RFC type of thing for these images or runtime spec?
  - archevel 10y ago
    I started by watching https://www.youtube.com/watch?v=HPuvDm8IC-4 https://www.youtube.com/watch?v=HPuvDm8IC-4 and reading https://www.infoq.com/articles/build-a-container-golang https://www.infoq.com/articles/build-a-container-golang Also "man namespaces" on a linux machine will give some deeper documentation.
  - gens 10y ago
    https://lwn.net/Articles/531114/ https://lwn.net/Articles/531114/ and https://www.kernel.org/doc/Documentation/cgroup-v1/ https://www.kernel.org/doc/Documentation/cgroup-v1/ (or the newer https://www.kernel.org/doc/Documentation/cgroup-v2.txt https://www.kernel.org/doc/Documentation/cgroup-v2.txt ) Actually all that was said (and somewhat explained) in the article in question. Although i would recommend to try cgroups directly with the kernel.
- deleted 10y ago
  [deleted]
- archevel 10y ago
  I second this! I watched Liz Rice's Golang UK 2016 talk and was inspired to write something to create containers. I was also interested in x86_64 assembly so decided to go with that. In the end it is realy simple to create something to run containers even in assembly since it is just a few syscalls to set things up. Ended up with: https://github.com/archevel/quic https://github.com/archevel/quic The main difficulty was figuring out how to do the netns part.
peterwwillis 10y ago
I have some small critiques of some of the hyperbole in the article: "Package managers failed us due to shared libraries version differences causing dependency issues" Incorrect. The software administrators (read: The Users) failed to understand that installing duplicate incompatible software does not work, was never intended to happen, and shouldn't even be possible. But users are stubborn and will force a conflict if at all possible. Containers allow users to side-step package management. It doesn't replace it or help it at all, because it completely ignores all the work gone into the package. Imagine putting on tennis shoes, and then trying to put on snow boots. Containers give users a second pair of feet. And this is not a container innovation. Chroot environments have been providing the exact same functionality (installing side-by-side conflicting packaged software in a simple manner) for decades. You don't even need any extra software to use it. "Docker provides a self-contained image that is exactly that same image running on your laptop vs in the cloud while i.e. Puppet/Chef are procedural scripts that need to rerun to converge your cluster machines. This enables approaches also know as Immutable Infrastructure or Phoenix Deploys." Unless you designed your software to be immutable, it probably isn't. Software changes as it runs, and different hardware changes software differently, so at the best this claim is disingenuous. Different networks and systems interacting in different locations add complications. If you tested it on your laptop, do not expect it to run the same in production, period. "Before Docker, LXC would create a full copy of FileSystem when creating a container. This would be slow and take up a lot of space." Loop and COW filesystems (Unionfs, Aufs, Overlayfs, etc) on Linux pre-date Docker by a long time, and were used with containers and container alternatives. -- I thought i'd see more about linux container internals, not a description of how Docker works, but I guess the host name should have been a dead giveaway. Don't read this if you want to know about the kernel.
- so0k 10y ago
  Totally agree, the post was written almost a year ago across 3 days while preparing a talk (see intro) and while still learning about Docker. Thanks for the feedback :)
  - deleted 10y ago
    [deleted]
xorcist 10y ago
OpenVZ is dated 2005 in the document, which makes FreeBSD look a bit alone back in 2000, but that's not really accurate. OpenVZ was just a re-branding of Virtuozzo, which was released in 2000, in an effort to upstream it. Virtuzzo got really popular quickly in the cheap webhosting market, as a more secure and powerful alternative to shared hosting. I worked with it a lot in the early 2000s. I don't think they ever outgrew that market however, and when "real" virtualization came with VMware and friends, that's where all the money went. It's only fair to mention where it really started in the Linux world. (Also a bit funny to see the pendulum of tech swing back again. It's about time!)
- so0k 10y ago
  Cool! thanks for info. I've added a link back to these comments to the blog source (haven't re-generated the static html yet though)
ihaveahadron 10y ago
Can't Linux be fixed. Nothing's perfect. But can't we at least try to make it work as correctly as possible instead of not? Win ten and osx basically "work" flawlessly. Linux doesn't.
MaBu 10y ago
Site looks broken, because CSS is loaded over HTTP which is disabled if site is loaded over HTTPS.
mthoms 10y ago
I've been struggling with fully understanding containers. This article helps but it's a little too low level for me. A quick question for HN'ers: If you've got a machine running say 4 docker instances, does it help resource usage if all instances are running the same Linux distro? Or, since the kernel is the only thing shared between them does it even matter?

201 points by deepakkarki 10y ago

siegecraft 10y ago

This is from February, which is ancient in Docker years, but the container history and references are quite useful.

so0k 10y ago
yet it covers containerd ;)

Demiurge 10y ago

The url is correct, 'Docker-internals', but the title, and further equivocation of 'Linux Containers' and 'Docker', is a bit confusing. I'm a happy lxc/lxd user, and had to stop reading because of the cognitive dissonance.

agumonkey 10y ago
I also thought it was about LXC. Sad.
- deleted 10y ago
  [deleted]
- skeptic2718 10y ago
  Why is it sad?
  - dozzie 10y ago
    Because LXC on a technical level is much simpler, doesn't make mess in network and filesystem setup, and is easy to understand. Docker does some heavy magic for things to work for programmers (the ones who can't be bothered to learn how ethernet bridging or IP routing work), so the whole thing feels brittle. In short, LXC is vastly underappreciated.
so0k 10y ago
Good point, updated the title. Thanks for the comments (and yes it's almost a year ago I wrote this post)
- Demiurge 10y ago
  Thanks, feels like the nitpick, but I gave it another go, and I see some other changes now? Seems like a great amount of information.
  - so0k 10y ago
    don't think there are other changes - https://github.com/docker-saigon/docker-saigon.github.io/commits/master https://github.com/docker-saigon/docker-saigon.github.io/com...
    - Demiurge 10y ago
      neatz, I guess I just need to be more patient and set my expectations :)

SEJeff 10y ago

Also relevant: https://0xax.gitbooks.io/linux-insides/content/Cgroups/cgroups1.html https://0xax.gitbooks.io/linux-insides/content/Cgroups/cgrou...

kragniz 10y ago

If you want to learn about linux container internals, I can recommend just trying to implement one yourself in some random language for fun. I wrote a basic runtime in python that can run docker images: https://github.com/kragniz/omochabako https://github.com/kragniz/omochabako Quick demo: https://asciinema.org/a/77296?speed=2&autoplay=1 https://asciinema.org/a/77296?speed=2&autoplay=1

decebalus1 10y ago
that sounds pretty cool. Sorry for ignorance (heavy Windows background) but where does one start with something like that? Is there a RFC type of thing for these images or runtime spec?
- archevel 10y ago
  I started by watching https://www.youtube.com/watch?v=HPuvDm8IC-4 https://www.youtube.com/watch?v=HPuvDm8IC-4 and reading https://www.infoq.com/articles/build-a-container-golang https://www.infoq.com/articles/build-a-container-golang Also "man namespaces" on a linux machine will give some deeper documentation.
- gens 10y ago
  https://lwn.net/Articles/531114/ https://lwn.net/Articles/531114/ and https://www.kernel.org/doc/Documentation/cgroup-v1/ https://www.kernel.org/doc/Documentation/cgroup-v1/ (or the newer https://www.kernel.org/doc/Documentation/cgroup-v2.txt https://www.kernel.org/doc/Documentation/cgroup-v2.txt ) Actually all that was said (and somewhat explained) in the article in question. Although i would recommend to try cgroups directly with the kernel.
deleted 10y ago
[deleted]
archevel 10y ago
I second this! I watched Liz Rice's Golang UK 2016 talk and was inspired to write something to create containers. I was also interested in x86_64 assembly so decided to go with that. In the end it is realy simple to create something to run containers even in assembly since it is just a few syscalls to set things up. Ended up with: https://github.com/archevel/quic https://github.com/archevel/quic The main difficulty was figuring out how to do the netns part.

peterwwillis 10y ago

I have some small critiques of some of the hyperbole in the article: "Package managers failed us due to shared libraries version differences causing dependency issues" Incorrect. The software administrators (read: The Users) failed to understand that installing duplicate incompatible software does not work, was never intended to happen, and shouldn't even be possible. But users are stubborn and will force a conflict if at all possible. Containers allow users to side-step package management. It doesn't replace it or help it at all, because it completely ignores all the work gone into the package. Imagine putting on tennis shoes, and then trying to put on snow boots. Containers give users a second pair of feet. And this is not a container innovation. Chroot environments have been providing the exact same functionality (installing side-by-side conflicting packaged software in a simple manner) for decades. You don't even need any extra software to use it. "Docker provides a self-contained image that is exactly that same image running on your laptop vs in the cloud while i.e. Puppet/Chef are procedural scripts that need to rerun to converge your cluster machines. This enables approaches also know as Immutable Infrastructure or Phoenix Deploys." Unless you designed your software to be immutable, it probably isn't. Software changes as it runs, and different hardware changes software differently, so at the best this claim is disingenuous. Different networks and systems interacting in different locations add complications. If you tested it on your laptop, do not expect it to run the same in production, period. "Before Docker, LXC would create a full copy of FileSystem when creating a container. This would be slow and take up a lot of space." Loop and COW filesystems (Unionfs, Aufs, Overlayfs, etc) on Linux pre-date Docker by a long time, and were used with containers and container alternatives. -- I thought i'd see more about linux container internals, not a description of how Docker works, but I guess the host name should have been a dead giveaway. Don't read this if you want to know about the kernel.

so0k 10y ago
Totally agree, the post was written almost a year ago across 3 days while preparing a talk (see intro) and while still learning about Docker. Thanks for the feedback :)
- deleted 10y ago
  [deleted]

xorcist 10y ago

OpenVZ is dated 2005 in the document, which makes FreeBSD look a bit alone back in 2000, but that's not really accurate. OpenVZ was just a re-branding of Virtuozzo, which was released in 2000, in an effort to upstream it. Virtuzzo got really popular quickly in the cheap webhosting market, as a more secure and powerful alternative to shared hosting. I worked with it a lot in the early 2000s. I don't think they ever outgrew that market however, and when "real" virtualization came with VMware and friends, that's where all the money went. It's only fair to mention where it really started in the Linux world. (Also a bit funny to see the pendulum of tech swing back again. It's about time!)

so0k 10y ago
Cool! thanks for info. I've added a link back to these comments to the blog source (haven't re-generated the static html yet though)

ihaveahadron 10y ago

Can't Linux be fixed. Nothing's perfect. But can't we at least try to make it work as correctly as possible instead of not? Win ten and osx basically "work" flawlessly. Linux doesn't.

MaBu 10y ago

Site looks broken, because CSS is loaded over HTTP which is disabled if site is loaded over HTTPS.

mthoms 10y ago

I've been struggling with fully understanding containers. This article helps but it's a little too low level for me. A quick question for HN'ers: If you've got a machine running say 4 docker instances, does it help resource usage if all instances are running the same Linux distro? Or, since the kernel is the only thing shared between them does it even matter?