7 ms·

Life of a Container (2020)

73 points by luu 5y ago

deleted 5y ago
[deleted]
whalesalad 5y ago
I love seeing posts like this. The best way to understand things is to go back to basics. When you understand the fundamentals, all the "crazy sophisticated" stuff becomes a lot easier to reason about and internalize.
0x0 5y ago
Did the cgroup names change randomly in the middle of the examples? It changes from "child" to "demo" between # mkdir /sys/fs/cgroup/memory/child and # ls -lh /sys/fs/cgroup/memory/demo/
- singlow 5y ago
  evidence of a spliced speedrun...
jeffbee 5y ago
Small quibble: cgroup fs may be accessible at /sys/fs/cgroup, but it may also be mounted anywhere. Use `mount -t cgroup` to find out.
arpa 5y ago
check out bocker[1] for a tl;dr version (~100 lines of bash) without any unnecessary flavouring. 1. https://github.com/p8952/bocker https://github.com/p8952/bocker
Joker_vD 5y ago
Does anybody know if the functionality mentioned in the "One final note" — having all children, grandchildren, great-grandchildren, etc. of a designated "main" process reliably terminated when that "main" process terminates — is available via something more lightweight than the containers? Traditional process groups/sessions are laughably inappropriate for this purpose since breaking out of them is trivial and in fact, about half of applications that internally use worker (sub)processes do exactly that, in order to (re)implement their own job control.
- yrro 5y ago
  Interesting idea. A sort of PR_SET_CHILD_SUBREAPER + "if I die, send SIGKILL to all my children" option. That would be handy...
  - Joker_vD 5y ago
    Technically, it could be implemented if the subreaper process were getting notified somehow of new processes being reparented to it. Alas, subreapers don't get such notifications. Also, you need some protection against children who do "kill(-getpid(), SIGKILL)" before the exit.
- paulfurtado 5y ago
  If running as a privileged user, you can run the process in just a PID namespace with no other container features enabled using the unshare command. Systemd can also handle it by killing the processes by cgroup i believe, and for non services you could take advantage of that with systemd-run.
  - Joker_vD 5y ago
    Are these things properly "nested"? Imagine a scenario like this: A process is started with "unshare A" or "cgexec A" or whatever, then it creates children one of which does "unshare B" or "cgexec B", then the first process, "A", is killed, or its cgroup is signalled, whatever. Will the process "B", that was put into a new namespace/cgroup, be terminated too or not? With traditional pgroups/sessions the answer is "no", and I'd like it to be "yes", so no runaway processes should be possible. I personally think the only way to make a "detached" process should be by asking some process from an entirely different "group" (over RPC, presumably) to launch something for you: so that the newly created process would technically be a children of this other "launcher" process.
stuff4ben 5y ago
My default question I give to interviewees looking for a dev or devops position is to tell me what a container is. Usually I get the "well it's like a vm but not a vm" and then I ask the followup which is to ask what are the differences between a container and vm? A lot of hemming and hawing but in the 6 or so years I've been interviewing people who are going to be using Docker, I have yet to hear the answer that I give when I interview, which is, "a container is a process." I usually leave it at that and wait to see if I get a confused look on their face. Then I say, "with cgroups and namespaces". It's really not that complicated (to explain anyways). EDIT: I found this link at the bottom of the OP's blog that was even better at simplifying things, https://jvns.ca/blog/2016/10/10/what-even-is-a-container/ https://jvns.ca/blog/2016/10/10/what-even-is-a-container/
- mellosouls 5y ago
  I'm not sure a "A process with cgroups and namespaces" makes things much clearer though somebody with understanding of those might infer certain properties. "a kind of lightweight VM" seems a much more intuitive answer, with limitations. The link is useful tho, she's a good writer.
  - nomoreplease 5y ago
    Not OP but I also interview. I see CGroups and namespaces is a follow up to “how does it work or how is it usually isolated?” And I ask “why would I use it?” Or “when would I choose containers over vm?” Or vice versa to get to actual architectural decisions The wrong answer usually involves a lot of hemming and hawing
    - mellosouls 5y ago
      Yeah tbf it was offered as a follow on question, I'm just not sure it makes things clearer in describing functional and design differences.
  - stuff4ben 5y ago
    Except "a kind of lightweight VM" is an incorrect answer. I'm trying to gauge if the person knows what they're talking about because running Docker and Kubernetes in production is hard. I don't want someone thinking they can go into some vsphere console to reboot their VMs when a pod is having problems.
anyonecancode 5y ago
A container as just an isolated process makes enough sense to me, but where I get confused is how it allows me to use alpine or ubuntu or some other base image that differs from the actual OS I'm running on. That's what makes it feel more like a lightweight VM to me, and which doesn't seem explained by just cgroups and namespaces. Or if it is, there's something I'm still failing to understand here.
- infogulch 5y ago
  The layers typically go: hardware > hypervisor instance (vm) > linux+distro > process. I think it's accurate to say that containers shuffle around those last two so it looks more like: hw > vm > linux > distro+process. The complication is that the lone linux in the middle is still deployed with a distro, it's just that the distro part is abstracted away from the perspective of the container.
- tene 5y ago
  I'd guess that the gap you're missing is that one of the namespace types containers use is a mount namespace. During container setup, you get a new mount namespace so that any changes to mounted filesystems are only seen by processes sharing that new mount namespace. Then you mount the filesystem from the docker image, replacing the filesystems mounted in the root namespace. I don't know if it'll help, but that process is very similar to what happens during boot, where you've initially got a root filesystem mounted from the initrd, but you replace it with the root filesystem from a disk after you've loaded the right drivers from the initrd.
  - anyonecancode 5y ago
    Yes, that is definitely one of the gaps I had, thank you (also the other replies to my comment -- upvoted all of you). I'm generally on a Mac (which runs docker in a linux VM -- the reason why is much clearer now) and so my exposure to linux is mostly in the context of ssh-ing into a server. I think the fact that OS X is linux-like without actually being a linux distro also subtly led me astray, as I've been thinking of different distros in the same way I think of OS X as different from Windows, and not really thinking about the "userland" vs "kernel" distinction.

73 points by luu 5y ago

deleted 5y ago

[deleted]

I love seeing posts like this. The best way to understand things is to go back to basics. When you understand the fundamentals, all the "crazy sophisticated" stuff becomes a lot easier to reason about and internalize.

0x0 5y ago

Did the cgroup names change randomly in the middle of the examples? It changes from "child" to "demo" between # mkdir /sys/fs/cgroup/memory/child and # ls -lh /sys/fs/cgroup/memory/demo/

singlow 5y ago
evidence of a spliced speedrun...

jeffbee 5y ago

Small quibble: cgroup fs may be accessible at /sys/fs/cgroup, but it may also be mounted anywhere. Use `mount -t cgroup` to find out.

arpa 5y ago

check out bocker[1] for a tl;dr version (~100 lines of bash) without any unnecessary flavouring. 1. https://github.com/p8952/bocker https://github.com/p8952/bocker

Joker_vD 5y ago

Does anybody know if the functionality mentioned in the "One final note" — having all children, grandchildren, great-grandchildren, etc. of a designated "main" process reliably terminated when that "main" process terminates — is available via something more lightweight than the containers? Traditional process groups/sessions are laughably inappropriate for this purpose since breaking out of them is trivial and in fact, about half of applications that internally use worker (sub)processes do exactly that, in order to (re)implement their own job control.

yrro 5y ago
Interesting idea. A sort of PR_SET_CHILD_SUBREAPER + "if I die, send SIGKILL to all my children" option. That would be handy...
- Joker_vD 5y ago
  Technically, it could be implemented if the subreaper process were getting notified somehow of new processes being reparented to it. Alas, subreapers don't get such notifications. Also, you need some protection against children who do "kill(-getpid(), SIGKILL)" before the exit.
paulfurtado 5y ago
If running as a privileged user, you can run the process in just a PID namespace with no other container features enabled using the unshare command. Systemd can also handle it by killing the processes by cgroup i believe, and for non services you could take advantage of that with systemd-run.
- Joker_vD 5y ago
  Are these things properly "nested"? Imagine a scenario like this: A process is started with "unshare A" or "cgexec A" or whatever, then it creates children one of which does "unshare B" or "cgexec B", then the first process, "A", is killed, or its cgroup is signalled, whatever. Will the process "B", that was put into a new namespace/cgroup, be terminated too or not? With traditional pgroups/sessions the answer is "no", and I'd like it to be "yes", so no runaway processes should be possible. I personally think the only way to make a "detached" process should be by asking some process from an entirely different "group" (over RPC, presumably) to launch something for you: so that the newly created process would technically be a children of this other "launcher" process.

stuff4ben 5y ago

My default question I give to interviewees looking for a dev or devops position is to tell me what a container is. Usually I get the "well it's like a vm but not a vm" and then I ask the followup which is to ask what are the differences between a container and vm? A lot of hemming and hawing but in the 6 or so years I've been interviewing people who are going to be using Docker, I have yet to hear the answer that I give when I interview, which is, "a container is a process." I usually leave it at that and wait to see if I get a confused look on their face. Then I say, "with cgroups and namespaces". It's really not that complicated (to explain anyways). EDIT: I found this link at the bottom of the OP's blog that was even better at simplifying things, https://jvns.ca/blog/2016/10/10/what-even-is-a-container/ https://jvns.ca/blog/2016/10/10/what-even-is-a-container/

mellosouls 5y ago
I'm not sure a "A process with cgroups and namespaces" makes things much clearer though somebody with understanding of those might infer certain properties. "a kind of lightweight VM" seems a much more intuitive answer, with limitations. The link is useful tho, she's a good writer.
- nomoreplease 5y ago
  Not OP but I also interview. I see CGroups and namespaces is a follow up to “how does it work or how is it usually isolated?” And I ask “why would I use it?” Or “when would I choose containers over vm?” Or vice versa to get to actual architectural decisions The wrong answer usually involves a lot of hemming and hawing
  - mellosouls 5y ago
    Yeah tbf it was offered as a follow on question, I'm just not sure it makes things clearer in describing functional and design differences.
- stuff4ben 5y ago
  Except "a kind of lightweight VM" is an incorrect answer. I'm trying to gauge if the person knows what they're talking about because running Docker and Kubernetes in production is hard. I don't want someone thinking they can go into some vsphere console to reboot their VMs when a pod is having problems.

anyonecancode 5y ago

A container as just an isolated process makes enough sense to me, but where I get confused is how it allows me to use alpine or ubuntu or some other base image that differs from the actual OS I'm running on. That's what makes it feel more like a lightweight VM to me, and which doesn't seem explained by just cgroups and namespaces. Or if it is, there's something I'm still failing to understand here.

infogulch 5y ago
The layers typically go: hardware > hypervisor instance (vm) > linux+distro > process. I think it's accurate to say that containers shuffle around those last two so it looks more like: hw > vm > linux > distro+process. The complication is that the lone linux in the middle is still deployed with a distro, it's just that the distro part is abstracted away from the perspective of the container.
tene 5y ago
I'd guess that the gap you're missing is that one of the namespace types containers use is a mount namespace. During container setup, you get a new mount namespace so that any changes to mounted filesystems are only seen by processes sharing that new mount namespace. Then you mount the filesystem from the docker image, replacing the filesystems mounted in the root namespace. I don't know if it'll help, but that process is very similar to what happens during boot, where you've initially got a root filesystem mounted from the initrd, but you replace it with the root filesystem from a disk after you've loaded the right drivers from the initrd.
- anyonecancode 5y ago
  Yes, that is definitely one of the gaps I had, thank you (also the other replies to my comment -- upvoted all of you). I'm generally on a Mac (which runs docker in a linux VM -- the reason why is much clearer now) and so my exposure to linux is mostly in the context of ssh-ing into a server. I think the fact that OS X is linux-like without actually being a linux distro also subtly led me astray, as I've been thinking of different distros in the same way I think of OS X as different from Windows, and not really thinking about the "userland" vs "kernel" distinction.