6 ms·

The What, Why and How of Containers

226 points by ben_s 3y ago

pjmlp 3y ago
Misses HP-UX Vault, Solaris Zones, Aix LPAR, and whatever IBM was doing with System 360 and MVS.
simpaticoder 3y ago
Conceptually, I've come to think of containers as a kind of "known-good starting point", the origin of a coordinate system where "movement" is adding things. A set of Dockerfiles form a trie where each line of the Dockerfile is a node in that trie's branch. The great benefit of containers is that they allow you to reach any possible point in the space for a single process, without affecting any other. The other features of containers are, to me, secondary, things like container images, or even access or resource control. The main draw of the tool is giving the user a declarative way to move reliably and repeatedly through system-space, and to do so for any number of processes. (The main cost is the ~20% overhead such a system incurs).
disconnect3d 3y ago
It's a nice blog post but it still misses a few important building blocks without which it would be trivial to escape a container running as root. Apart from chroot, cgroups and namespaces, the containers are also build upon: 1) linux capabilities - that split the privileges of a root user into "capabilities" which allows limiting the actions a root user can do (see `man 7 capabilities`, `cat /proc/self/status | grep Cap` or `capsh --decode=a80425fb`) 2) seccomp - which is used to filter syscalls and their arguments that a process can execute. (fwiw Docker renders its seccomp policy based on the capabilities requested by the container) 3) AppArmor (or SELinux, though AppArmor is the default) - a LSM (Linux Security Module) used to limit access to certain paths on the system and syscalls 4) masked paths - container engines bind mounts certain sensitive paths so they can't be read or written to (like /proc/sysrq-trigger, /proc/irq, /proc/kcore etc.) 5) NoNewPrivs flag - while not enabled by default (e.g., in Docker) this prevents the user from gaining more privileges (e.g., suid binaries won't change the uid) If anyone is interested in reading more about those topics and security of containers, you may want to read a blog post [0] where I dissected a privileged docker escape technique (note: with --privileged, you could just mount the disk device and read/write to it) and slides from a talk [1] I have given which details the Docker container building blocks and shows how we can investigate them etc. [0] https://blog.trailofbits.com/2019/07/19/understanding-docker-container-escapes/ https://blog.trailofbits.com/2019/07/19/understanding-docker... [1] https://docs.google.com/presentation/d/1tCqmGSOJJzi6ZK7TNhbzVFsTekvjvQR8GGPoaYBrM1o/ https://docs.google.com/presentation/d/1tCqmGSOJJzi6ZK7TNhbz...
- nurple 3y ago
  Excellent info! I started head-deving a project similar to nix-snapshotter[0] and I was thinking "ok, I can probably just build CRI impl that builds a rootfs dir with nix and just shell out to bubblewrap to make a "container". But once I went through that mental exercise I started reading code in containerd and cri-o. Wow, these are _not_ simple projects; containerd itself having a full GRPC-based service registry for driving dynamic logic via config. One thing I was pretty disappointed about is how deeply ingrained OSI images are in the whole ecosystem. While you can replace almost all functional parts of runtime, but not really the concept of images. I think images are a poor solution to the problem they solve, and a big downside of this is a bunch of complexity in the runtimes trying to work around how images work (like remote snapshotters). [0] https://github.com/pdtpartners/nix-snapshotter https://github.com/pdtpartners/nix-snapshotter
ahepp 3y ago
If this struck your interest, but you want more nitty gritty examples and details, you may find the following article interesting: https://ericchiang.github.io/post/containers-from-scratch/ https://ericchiang.github.io/post/containers-from-scratch/ If I'm remembering correctly from when I ran through the instructions at home, it was written for the original cgroup sysfs interface rather than the more modern cgroup2 [0]. You can figure out which you're running with > mount | grep cgroup > cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) which turns the examples is a nice "check your understanding" [0]: https://docs.kernel.org/admin-guide/cgroup-v2.html#basic-operations https://docs.kernel.org/admin-guide/cgroup-v2.html#basic-ope...
zinodaur 3y ago
I'm still not sold on the "why" wrt kubernetes. I hate that my resource hog map reduce jobs run on the same kernel and contend for the same resources as my user facing live site service.
- EraYaN 3y ago
  But kubernetes has very good support for segmenting applications and long running processes, you don't even have to segment the nodes, you can just "let it happen" (although you should probably segment the nodes somewhat). You can set (anti) affinity for example to make applications not tolerate each other when scheduled etc. And there are quite a few more knobs the scheduler has that you can tune.
- kube-system 3y ago
  That is one of the reasons why. Containers share the kernel and system resources. When you want to start running a bunch of containers in a particular configuration, that's when you'd use a container orchestration tool like kubernetes to define how and where you want those containers to run across multiple systems. While you could schedule containers manually, or just run your application on VMs or hardware manually, something like kubernetes will let you define rules which it will dynamically evaluate against your infrastructure. You can instruct kubernetes to run your map reduce jobs on different nodes than your user-facing site... and you can give kubernetes an arbitrary number of nodes to work with, and it can scale your workloads for you automatically while also following your rules.
  - zinodaur 3y ago
    I guess I would prefer "kubernetes but with VMs instead of containers". The overhead of running in a VM is not very high, and a hypervisor can restrict resource usage much more effectively - so that we could still bin pack map reduce jobs on the same machines as live site services
    - kube-system 3y ago
      If your kubernetes nodes are VMs then you can do both at the same time for different parts of your application.
deleted 3y ago
[deleted]
HeyLaughingBoy 3y ago
As someone whose primary area of development is embedded systems and has never used a Container, I really like this ELI5 explanation.
sam2426679 3y ago
When I was first learning about containers, I found the below course to be very helpful, which has a similar didactic trajectory to this article. https://frontendmasters.com/courses/complete-intro-containers/ https://frontendmasters.com/courses/complete-intro-container...
mikewarot 3y ago
Containers are a bad take on a solved problem. The problem was encountered, studied[0] and solved, decades ago. During the Viet Nam conflict, the Air Force needed to plan missions with multiple levels of classified data. This couldn't be done with the systems of that era. This resulted in research and development of multi-level security, the Bell-LaPadula model[2], and capability based security[1]. Conceptually, it's elegant, and requires almost no changes in user behavior while solving entire classes of problems with minimal code changes. It's a matter of changing the default from all access to no access, all the way down to the kernel. [0] https://csrc.nist.rip/publications/history/ande72.pdf https://csrc.nist.rip/publications/history/ande72.pdf [1] https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security [2] https://en.wikipedia.org/wiki/Bell%E2%80%93LaPadula_model https://en.wikipedia.org/wiki/Bell%E2%80%93LaPadula_model
- remram 3y ago
  Containers are not a security mechanism, they are a deployment mechanism.
timetraveller26 3y ago
systemd-nspawn is really useful for the day to day to be able to have isolated systems in a more lightweight fashion that lxc and docker. It comes already with systemd (I know, cool, right?) More info: https://wiki.archlinux.org/title/systemd-nspawn https://wiki.archlinux.org/title/systemd-nspawn
adamgordonbell 3y ago
If you use chroot to run something, it's interesting how the dynamic libs you need to get in place grows until you are mirroring a whole linux in a subtree. It gives you a sense for how you end up with containers. One thing that is wild to me is how nix solves this problem, of things needing to be linked together. It doesn't solve it with containers, but by rewriting the location of the links in the executable to be in the nix store. You can run LDD and see it in action. To me, all that points at containers being in some way a solution to Dynamic linking. And maybe an over the top solution. Should we be doing more static linking? Not even depending on libc? What are the challenges with that?
- tutfbhuf 3y ago
  One issue with static linking is that your dependencies will likely have critical CVEs over time. If you keep all your libraries separate on the filesystem, you can just do a "apt update; apt upgrade", and you will have all the latest patches. This will patch security issues in e.g. libssl or libc for all your applications that are dynamically linked against this shared libraries, which can be quite a few. In static binaries, the version of the libraries is not obvious from the outside. If you have, for example, 100 fully static binaries, these can come in 100 different major/minor/patch level versions of their dependencies. You now have to patch each binary separately by upgrading and recompilation 100 times to patch all your static binaries, that requires much more time and energy.
  - adamgordonbell 3y ago
    That all makes sense. But when those 100 binaries end up as 100 OCI images, and then to patch them you need to update those 100 OCI images to have the new version, it does seem like we've gone in a circle a bit. I mean, there are some advantages, if they all share the same base layer, maybe they share those libs at least on disk via a shared layer. But practically, though you are maybe not back where you started, you are at a place that seems to share some similarities.
    - nurple 3y ago
lysecret 3y ago
Oh nice I really like this style of explanation.
Zambyte 3y ago
Unfortunately this skips over the history of microkernels, which solve the same problems in a much more elegant way than containers.
- mati365 3y ago
  Can you elaborate?
  - Zambyte 3y ago
    The point of containers is to run a process in an isolated environment. Microkernels by design allow isolating any process with very fine grain control, by allowing or disallowing certain IPC connections for a given process. Those connections can be enabled or disabled for a running process as well, which would essentially be like moving a process in and out of a container while it is running. Individual processes can also run entirely isolated stacks for things like networking, storage, etc. in an unprivileged way. The former can be particularly painful to deal with in Linux containers. Containers are basically monolithic kernels playing catching to the features designed into microkernel-based operating systems.
  - palata 3y ago
    I'd love to get more details, too. Sounds interesting!
    - FuriouslyAdrift 3y ago
      Probably referring to library operating systems and making a unikernel instead of a shared kernel container. https://www.sigarch.org/leave-your-os-at-home-the-rise-of-library-operating-systems/ https://www.sigarch.org/leave-your-os-at-home-the-rise-of-li...
    - FuriouslyAdrift 3y ago
      Another write up: https://phoenixnap.com/kb/unikernel-vs-container https://phoenixnap.com/kb/unikernel-vs-container
kqr 3y ago
I wish I had read this article a decade ago. For many years I have been wondering "why the heck would I use containers when I have chroot, cgroups and namespaces?" Turns out that's exactly what containers are a packaging of! And I only found out about two years ago. Although this article doesn't go into it, the benefits I've found of using containers rather than rolling isolation by hand is that a lot of semi-standardised monitoring, deployment, and workload management tooling expects things to come packaged as containers.
- otabdeveloper4 3y ago
  > Turns out that's exactly what containers are a packaging of! Well, no. When people say "containers", they always mean "Docker". And Docker also comes with a daemon with full root permissions and ridiculous security policies. (Like, for example, forcefully turning off your machine's firewall, #yolo. WTF!) P.S. I actually run systemd-nspawn in production, but I am probably the only person on earth to do so.
  - moreentropy 3y ago
    > P.S. I actually run systemd-nspawn in production, but I am probably the only person on earth to do so. You're not alone, systemd-nspawn is very much underrated. I have used it a lot for machine containers, though I'm using podman+quadlet+systemd more right now. systemd-nspawn with mkosi for generating workload images is still a nice & powerful ecosystem.
  - kachnuv_ocasek 3y ago
    Perhaps I belong to the minority, but I really don't think about containers as Docker. Actually, I don't remember the last time I used Docker for anything. For the past several years, I've been using either Podman or systemd-nspawn, as yourself.
  - EraYaN 3y ago
    Or you know containerd, which is I feel used much more often in production than docker itself.
  - 3y ago
the_duke 3y ago
Great high level explanation. Note: if you look into the details of how setting up namespaces and cgroups works you'll run away in horror. The APIs are very iteratively evolved piecework, not really a coherent(ly designed) abstraction.
cwillu 3y ago
Computing is an endless cycle of inventing ways to isolate code in a private machine, followed by inventing ways to make it easier for those machines to interoperate.
- begueradj 3y ago
  That's a wise statement.
- npteljes 3y ago
  Absolute. I feel like society goes through changes in a similar cyclic way. We, as humans, basically have a finite span of understanding and attention, and so, basically create cycles that are longer than that.
- nurple 3y ago
  This is a really interesting way to think about the progression. As a timeline I like to plot the ratio of users to isolated compute. We've moved along points like users per building, users per room, user per computer, computers per user, kernels per user, processes per user. Containers enabled the latest shift.
- cjk2 3y ago
  Don't forget an endless cycle of inventing ways to make debugging and problem solving harder by adding isolation boundaries and complexity :)
  - forgotmyinfo 3y ago
    Debugging Docker-anything makes me want to go into drywall, I swear.
    - rqtwteye 3y ago
      Better than going into brick walls I guess
- pjmlp 3y ago
  It is like the endless cycle of "microservices" since distributed computing was invented, after computer networks came to be.
dzonga 3y ago
something that is nice in the container world -- better than docker are lxc containers - but the steward of the project Canonical seem to have done a bad job with it. last time I played with lxc the ux was clunky. if you could have the automation / configuration of docker / podman for lxc that would have been nice.
- hunter2_ 3y ago
  I've used Proxmox to manage my LXC workloads for years and it's been great, although I'm unaware to what extent it meets your criteria of offering automation. I find its interface to do roughly what a VM host (VirtualBox, VMWare, etc.) can do, but for LXC containers (and QEMU VMs) instead of VMs.
jason2323 3y ago
Is there a guide around that teaches you to build a container from scratch with chroot, namespaces and cgroups?
- Izkata 3y ago
  It's not a tutorial, but "Docker implemented in around 100 lines of Bash" might help: https://news.ycombinator.com/item?id=33218094 https://news.ycombinator.com/item?id=33218094
- adamgordonbell 3y ago
  I did one with just the chroot part: https://earthly.dev/blog/chroot/ https://earthly.dev/blog/chroot/ Liz Rice has a good talk about the cgroups and namespaces. https://www.youtube.com/watch?v=_TsSmSu57Zo https://www.youtube.com/watch?v=_TsSmSu57Zo
- deleted 3y ago
  [deleted]
- ahepp 3y ago
  This might be what you're looking for? IIRC it was written for the older cgroup (v1) sysfs interface, so you may need to cross reference it with the cgroup2 documentation https://ericchiang.github.io/post/containers-from-scratch/ https://ericchiang.github.io/post/containers-from-scratch/

226 points by ben_s 3y ago

Misses HP-UX Vault, Solaris Zones, Aix LPAR, and whatever IBM was doing with System 360 and MVS.

Conceptually, I've come to think of containers as a kind of "known-good starting point", the origin of a coordinate system where "movement" is adding things. A set of Dockerfiles form a trie where each line of the Dockerfile is a node in that trie's branch. The great benefit of containers is that they allow you to reach any possible point in the space for a single process, without affecting any other. The other features of containers are, to me, secondary, things like container images, or even access or resource control. The main draw of the tool is giving the user a declarative way to move reliably and repeatedly through system-space, and to do so for any number of processes. (The main cost is the ~20% overhead such a system incurs).

disconnect3d 3y ago

It's a nice blog post but it still misses a few important building blocks without which it would be trivial to escape a container running as root. Apart from chroot, cgroups and namespaces, the containers are also build upon: 1) linux capabilities - that split the privileges of a root user into "capabilities" which allows limiting the actions a root user can do (see `man 7 capabilities`, `cat /proc/self/status | grep Cap` or `capsh --decode=a80425fb`) 2) seccomp - which is used to filter syscalls and their arguments that a process can execute. (fwiw Docker renders its seccomp policy based on the capabilities requested by the container) 3) AppArmor (or SELinux, though AppArmor is the default) - a LSM (Linux Security Module) used to limit access to certain paths on the system and syscalls 4) masked paths - container engines bind mounts certain sensitive paths so they can't be read or written to (like /proc/sysrq-trigger, /proc/irq, /proc/kcore etc.) 5) NoNewPrivs flag - while not enabled by default (e.g., in Docker) this prevents the user from gaining more privileges (e.g., suid binaries won't change the uid) If anyone is interested in reading more about those topics and security of containers, you may want to read a blog post [0] where I dissected a privileged docker escape technique (note: with --privileged, you could just mount the disk device and read/write to it) and slides from a talk [1] I have given which details the Docker container building blocks and shows how we can investigate them etc. [0] https://blog.trailofbits.com/2019/07/19/understanding-docker-container-escapes/ https://blog.trailofbits.com/2019/07/19/understanding-docker... [1] https://docs.google.com/presentation/d/1tCqmGSOJJzi6ZK7TNhbzVFsTekvjvQR8GGPoaYBrM1o/ https://docs.google.com/presentation/d/1tCqmGSOJJzi6ZK7TNhbz...

nurple 3y ago
Excellent info! I started head-deving a project similar to nix-snapshotter[0] and I was thinking "ok, I can probably just build CRI impl that builds a rootfs dir with nix and just shell out to bubblewrap to make a "container". But once I went through that mental exercise I started reading code in containerd and cri-o. Wow, these are _not_ simple projects; containerd itself having a full GRPC-based service registry for driving dynamic logic via config. One thing I was pretty disappointed about is how deeply ingrained OSI images are in the whole ecosystem. While you can replace almost all functional parts of runtime, but not really the concept of images. I think images are a poor solution to the problem they solve, and a big downside of this is a bunch of complexity in the runtimes trying to work around how images work (like remote snapshotters). [0] https://github.com/pdtpartners/nix-snapshotter https://github.com/pdtpartners/nix-snapshotter

ahepp 3y ago

If this struck your interest, but you want more nitty gritty examples and details, you may find the following article interesting: https://ericchiang.github.io/post/containers-from-scratch/ https://ericchiang.github.io/post/containers-from-scratch/ If I'm remembering correctly from when I ran through the instructions at home, it was written for the original cgroup sysfs interface rather than the more modern cgroup2 [0]. You can figure out which you're running with > mount | grep cgroup > cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) which turns the examples is a nice "check your understanding" [0]: https://docs.kernel.org/admin-guide/cgroup-v2.html#basic-operations https://docs.kernel.org/admin-guide/cgroup-v2.html#basic-ope...

zinodaur 3y ago

I'm still not sold on the "why" wrt kubernetes. I hate that my resource hog map reduce jobs run on the same kernel and contend for the same resources as my user facing live site service.

EraYaN 3y ago
But kubernetes has very good support for segmenting applications and long running processes, you don't even have to segment the nodes, you can just "let it happen" (although you should probably segment the nodes somewhat). You can set (anti) affinity for example to make applications not tolerate each other when scheduled etc. And there are quite a few more knobs the scheduler has that you can tune.
kube-system 3y ago
That is one of the reasons why. Containers share the kernel and system resources. When you want to start running a bunch of containers in a particular configuration, that's when you'd use a container orchestration tool like kubernetes to define how and where you want those containers to run across multiple systems. While you could schedule containers manually, or just run your application on VMs or hardware manually, something like kubernetes will let you define rules which it will dynamically evaluate against your infrastructure. You can instruct kubernetes to run your map reduce jobs on different nodes than your user-facing site... and you can give kubernetes an arbitrary number of nodes to work with, and it can scale your workloads for you automatically while also following your rules.
- zinodaur 3y ago
  I guess I would prefer "kubernetes but with VMs instead of containers". The overhead of running in a VM is not very high, and a hypervisor can restrict resource usage much more effectively - so that we could still bin pack map reduce jobs on the same machines as live site services
  - kube-system 3y ago
    If your kubernetes nodes are VMs then you can do both at the same time for different parts of your application.

deleted 3y ago

[deleted]

HeyLaughingBoy 3y ago

As someone whose primary area of development is embedded systems and has never used a Container, I really like this ELI5 explanation.

sam2426679 3y ago

When I was first learning about containers, I found the below course to be very helpful, which has a similar didactic trajectory to this article. https://frontendmasters.com/courses/complete-intro-containers/ https://frontendmasters.com/courses/complete-intro-container...

mikewarot 3y ago

Containers are a bad take on a solved problem. The problem was encountered, studied[0] and solved, decades ago. During the Viet Nam conflict, the Air Force needed to plan missions with multiple levels of classified data. This couldn't be done with the systems of that era. This resulted in research and development of multi-level security, the Bell-LaPadula model[2], and capability based security[1]. Conceptually, it's elegant, and requires almost no changes in user behavior while solving entire classes of problems with minimal code changes. It's a matter of changing the default from all access to no access, all the way down to the kernel. [0] https://csrc.nist.rip/publications/history/ande72.pdf https://csrc.nist.rip/publications/history/ande72.pdf [1] https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security [2] https://en.wikipedia.org/wiki/Bell%E2%80%93LaPadula_model https://en.wikipedia.org/wiki/Bell%E2%80%93LaPadula_model

remram 3y ago
Containers are not a security mechanism, they are a deployment mechanism.

timetraveller26 3y ago

systemd-nspawn is really useful for the day to day to be able to have isolated systems in a more lightweight fashion that lxc and docker. It comes already with systemd (I know, cool, right?) More info: https://wiki.archlinux.org/title/systemd-nspawn https://wiki.archlinux.org/title/systemd-nspawn

adamgordonbell 3y ago

If you use chroot to run something, it's interesting how the dynamic libs you need to get in place grows until you are mirroring a whole linux in a subtree. It gives you a sense for how you end up with containers. One thing that is wild to me is how nix solves this problem, of things needing to be linked together. It doesn't solve it with containers, but by rewriting the location of the links in the executable to be in the nix store. You can run LDD and see it in action. To me, all that points at containers being in some way a solution to Dynamic linking. And maybe an over the top solution. Should we be doing more static linking? Not even depending on libc? What are the challenges with that?

tutfbhuf 3y ago
One issue with static linking is that your dependencies will likely have critical CVEs over time. If you keep all your libraries separate on the filesystem, you can just do a "apt update; apt upgrade", and you will have all the latest patches. This will patch security issues in e.g. libssl or libc for all your applications that are dynamically linked against this shared libraries, which can be quite a few. In static binaries, the version of the libraries is not obvious from the outside. If you have, for example, 100 fully static binaries, these can come in 100 different major/minor/patch level versions of their dependencies. You now have to patch each binary separately by upgrading and recompilation 100 times to patch all your static binaries, that requires much more time and energy.
- adamgordonbell 3y ago
  That all makes sense. But when those 100 binaries end up as 100 OCI images, and then to patch them you need to update those 100 OCI images to have the new version, it does seem like we've gone in a circle a bit. I mean, there are some advantages, if they all share the same base layer, maybe they share those libs at least on disk via a shared layer. But practically, though you are maybe not back where you started, you are at a place that seems to share some similarities.
  - nurple 3y ago

lysecret 3y ago

Oh nice I really like this style of explanation.

Zambyte 3y ago

Unfortunately this skips over the history of microkernels, which solve the same problems in a much more elegant way than containers.

mati365 3y ago
Can you elaborate?
- Zambyte 3y ago
  The point of containers is to run a process in an isolated environment. Microkernels by design allow isolating any process with very fine grain control, by allowing or disallowing certain IPC connections for a given process. Those connections can be enabled or disabled for a running process as well, which would essentially be like moving a process in and out of a container while it is running. Individual processes can also run entirely isolated stacks for things like networking, storage, etc. in an unprivileged way. The former can be particularly painful to deal with in Linux containers. Containers are basically monolithic kernels playing catching to the features designed into microkernel-based operating systems.
- palata 3y ago
  I'd love to get more details, too. Sounds interesting!
  - FuriouslyAdrift 3y ago
    Probably referring to library operating systems and making a unikernel instead of a shared kernel container. https://www.sigarch.org/leave-your-os-at-home-the-rise-of-library-operating-systems/ https://www.sigarch.org/leave-your-os-at-home-the-rise-of-li...
  - FuriouslyAdrift 3y ago
    Another write up: https://phoenixnap.com/kb/unikernel-vs-container https://phoenixnap.com/kb/unikernel-vs-container

kqr 3y ago

I wish I had read this article a decade ago. For many years I have been wondering "why the heck would I use containers when I have chroot, cgroups and namespaces?" Turns out that's exactly what containers are a packaging of! And I only found out about two years ago. Although this article doesn't go into it, the benefits I've found of using containers rather than rolling isolation by hand is that a lot of semi-standardised monitoring, deployment, and workload management tooling expects things to come packaged as containers.

otabdeveloper4 3y ago
> Turns out that's exactly what containers are a packaging of! Well, no. When people say "containers", they always mean "Docker". And Docker also comes with a daemon with full root permissions and ridiculous security policies. (Like, for example, forcefully turning off your machine's firewall, #yolo. WTF!) P.S. I actually run systemd-nspawn in production, but I am probably the only person on earth to do so.
- moreentropy 3y ago
  > P.S. I actually run systemd-nspawn in production, but I am probably the only person on earth to do so. You're not alone, systemd-nspawn is very much underrated. I have used it a lot for machine containers, though I'm using podman+quadlet+systemd more right now. systemd-nspawn with mkosi for generating workload images is still a nice & powerful ecosystem.
- kachnuv_ocasek 3y ago
  Perhaps I belong to the minority, but I really don't think about containers as Docker. Actually, I don't remember the last time I used Docker for anything. For the past several years, I've been using either Podman or systemd-nspawn, as yourself.
- EraYaN 3y ago
  Or you know containerd, which is I feel used much more often in production than docker itself.
- 3y ago

the_duke 3y ago

Great high level explanation. Note: if you look into the details of how setting up namespaces and cgroups works you'll run away in horror. The APIs are very iteratively evolved piecework, not really a coherent(ly designed) abstraction.

cwillu 3y ago

Computing is an endless cycle of inventing ways to isolate code in a private machine, followed by inventing ways to make it easier for those machines to interoperate.

begueradj 3y ago
That's a wise statement.
npteljes 3y ago
Absolute. I feel like society goes through changes in a similar cyclic way. We, as humans, basically have a finite span of understanding and attention, and so, basically create cycles that are longer than that.
nurple 3y ago
This is a really interesting way to think about the progression. As a timeline I like to plot the ratio of users to isolated compute. We've moved along points like users per building, users per room, user per computer, computers per user, kernels per user, processes per user. Containers enabled the latest shift.
cjk2 3y ago
Don't forget an endless cycle of inventing ways to make debugging and problem solving harder by adding isolation boundaries and complexity :)
- forgotmyinfo 3y ago
  Debugging Docker-anything makes me want to go into drywall, I swear.
  - rqtwteye 3y ago
    Better than going into brick walls I guess
pjmlp 3y ago
It is like the endless cycle of "microservices" since distributed computing was invented, after computer networks came to be.

dzonga 3y ago

something that is nice in the container world -- better than docker are lxc containers - but the steward of the project Canonical seem to have done a bad job with it. last time I played with lxc the ux was clunky. if you could have the automation / configuration of docker / podman for lxc that would have been nice.

hunter2_ 3y ago
I've used Proxmox to manage my LXC workloads for years and it's been great, although I'm unaware to what extent it meets your criteria of offering automation. I find its interface to do roughly what a VM host (VirtualBox, VMWare, etc.) can do, but for LXC containers (and QEMU VMs) instead of VMs.

jason2323 3y ago

Is there a guide around that teaches you to build a container from scratch with chroot, namespaces and cgroups?

Izkata 3y ago
It's not a tutorial, but "Docker implemented in around 100 lines of Bash" might help: https://news.ycombinator.com/item?id=33218094 https://news.ycombinator.com/item?id=33218094
adamgordonbell 3y ago
I did one with just the chroot part: https://earthly.dev/blog/chroot/ https://earthly.dev/blog/chroot/ Liz Rice has a good talk about the cgroups and namespaces. https://www.youtube.com/watch?v=_TsSmSu57Zo https://www.youtube.com/watch?v=_TsSmSu57Zo
deleted 3y ago
[deleted]
ahepp 3y ago
This might be what you're looking for? IIRC it was written for the older cgroup (v1) sysfs interface, so you may need to cross reference it with the cgroup2 documentation https://ericchiang.github.io/post/containers-from-scratch/ https://ericchiang.github.io/post/containers-from-scratch/