14 ms·

What chroot taught me about containers

501 points by mmphosis 4y ago

cratermoon 4y ago
> for me, containers are just chrooted processes. Sure, they are more than that: Containers have a nice developer experience, an open-source foundation, and a whole ecosystem of cloud-native companies pushing them forward" "All right, but apart from the sanitation, the medicine, education, wine, public order, irrigation, roads, a fresh water system, and public health, what have the Romans ever done for us?"
lamontcg 4y ago
They're "RPMs" with some union mounted fanciness, a chroot, and some cgroups isolation to keep other people's fork bombs from taking out your service. Agreed that explainng containers should never start with VMs. I think it should start with package managers rather than chroots though.
- chriswarbo 4y ago
  I think .rpm/.deb is a bad analogy, since those (a) get installed into a global namespace (the / folder), (b) rely on the existence of other packages in that namespace (their dependencies) and (c) require global consistency between all the installed packages. You can avoid those global consistency problems by installing RPM/Deb packages into a chroot (or a "proper" container), but that ceases to be a good analogy of what containers are.
  - lamontcg 4y ago
    Yeah I've spent like a decade building rpms that packaged all their deps inside of them, so I very much don't find the dependency management between them to be fundamental, which might not be intuitively obvious to everyone else. Maybe it'd be better to call them a tarball, but there is the way that you can install it and run it and then nuke it cleanly which you don't get out of the box with a tarball, which is why I like the package metaphor slightly better. EDIT: the fact that it is in a chroot, though, I think sort of covers the way that it doesn't really interact with the external filesystem and installed packages. That implies that you have to ship all the deps with the package itself and those usually would be installed into something like /chroot/<whatever> and never into e.g. /usr/bin and the nature of the chroot would make everything in /usr/bin inaccessible. If you build an RPM which installs into a chroot you're really not going to be able to have any dependencies on anything else in the system. The tweaks to make that separation slightly nicer are fairly small tweaks overall. That might not be intuitively obvious though.
    - chriswarbo 4y ago
      > Maybe it'd be better to call them a tarball, but there is the way that you can install it and run it and then nuke it cleanly which you don't get out of the box with a tarball, which is why I like the package metaphor slightly better. Containers are tarballs, e.g. the rootfs for runc, the layers of an image, and images themselves are usually tarballs. The extra stuff "you don't get out of the box with a tarball" are precisely the tools which we call "containers" (from low-level stuff like runc, to high-level things like online image registries)
efficax 4y ago
containers are cgroups and namespaces and are much more than just chroots, plus a set of tools for making those work
- StopHammoTime 4y ago
  I think the point is that it's not magic, but a very clear and obvious system that's been created using native tools. I've run into a lot of people (including Senior SRE/DevOps people) who don't realise that Containers are just another process running on a system.
  - pushedx 4y ago
    A container is not a process. It’s an execution environment created by namespaces (mount, pid, network, others) and cgroups. A container can house many processes.
    - liquidgecka 4y ago
      ^-- this! Also, a container can house a container.. kind of. If you think of containers like file system inodes then there is a logical view of containers container1: (init -> dockerd) -> container2: (init -> dockerd) -> container3: (init -> dockerd) -> container4: (process) is perfectly feasible and from our view there is a hierarchy of containers, but in reality it would be possible assign a process in container4 to container1 and create all kinds of cool logical loops. This is effectively what a daemonset is in kubernetes. =)
      - lstodd 4y ago
        In reality there is no such thing as a 'container'. You can create an incredible self-intersecting mess of process namespaces. It is not what you want to do in production. Also, docker is an overengineered mess. Kubernetes only an couple of orders of magnitude more so.
    - G3rn0ti 4y ago
      On Windows and Macs Docker uses VMs ... I think the DEV community really needs a new definition of "containers" and one that is not backed by implementation details of Docker Linux.
akadempythag 4y ago
Fun deep dive. In a previous life I wrote web apps, and I liked to develop on Chromebooks to make sure that everything ran smoothly on low-end machines. I would set a price point of $2-300, which usually meant I was lucky to get a Rockchip. And this was before it was popular to build for ARM architectures. And yet, ChromeOS' chroot-based projects like crosh always managed to deliver. Surrounded by aluminum MacBooks and carbon-fiber Dells, I would run Rails apps and X GUIs and source builds on a rubberized hunk of plastic that had been designed for use in schools. I have always been surprised at how well that worked. If it was using the same fundamentals as Docker and Podman, I'm not surprised that the containerization movement has enjoyed such popularity.
- spiffytech 4y ago
  There was something really nice about developing on a Chromebook. It felt simplified in a very pleasant way. I ultimately discontinued the practice because beefy Chromebooks never become commonplace, some points of development friction got tiresome, and I stopped preferring Chrome. Yet I'm nostalgic for the experience.
- j0hnyl 4y ago
  Likewise it's amazing what one can do with Termux on an Android device.
bitwize 4y ago
Yeah, more or less, except containers also have a smoother UX and processes within a container are more successfully tricked into believing they're on a separate machine.
whalesalad 4y ago
I like containers, but I absolutely love chroot and debootstrap. It’s killer for when you need isolation (like building software locally without ruining the host OS) but don’t need the pomp and circumstance of containers.
- mikepurvis 4y ago
  I've long been a fan of basic debootstrap->chroot, though I will say in recent times systemd-nspawn and then buildah have definitely pretty much displaced chroot in my toolbox. They're equally pomp-free relative to Docker, but have a bunch of nice affordances in terms of a properly set up network including DNS and hosts, handling of filesystem permissions, and correctly presenting a read-only /proc.
  - whalesalad 4y ago
    I’ve been meaning to grok nspawn - for my recent use cases the filesystem was the only level of isolation required but I’m gonna check it out for sure. Glad to hear it’s as good as I’ve heard it is.
  - quickthrower2 4y ago
    If you are trying to tilt me over to always use linux instead of windows for all things you are doing a damn fine job. My big pain in any OS is messing it up by installing the wrong thing or the wrong way. But at the same time I want to play with different languages. I mean why use nvm just for node let alone anything else, when I can do this, it feels way cleaner.
exabrial 4y ago
I think the only correct way to do containers is… groan… how systemd does it.[ One should be able to launch a process and not let it touch anything else but it’s very process definition..] Thank you for your down vote. I look forward to being moderated out of existence.
- sargun 4y ago
  Perhaps a more productive comment (and I honestly am curious), would be -- what does systemd do right?
  - XorNot 4y ago
    Systemd is containers, is the thing. Its entirely built around using cgroups to manage process resources, it just starts from the normal "I'm running root processes" level because it was intended to init a whole system. The one missing component was the concept of union mounting layered file systems from a repository.
    - jsmith45 4y ago
      The systemd-nspawn system (which is how running more more typical containers is normally implemented) is a little unusual because it it geared towards running os containers, including running their whole init system (via --boot or the template unit file). There is a reason why machinectl calls the input machine images, and both accept raw disk images. This is also reflected in the systemd-nspawn requirement of having an os-release file in the container. Of course, it is entirely capable of running application containers (hence features like --as-pid2), but some of the defaults like registering with systemd-machined don't make as much sense in those scenarios. Docker is nominally geared towards application containers, although it mostly works with OS containers. Indeed it seems like container images that contain a full OS (minus kernel) are probably more common than not.
  - xorcist 4y ago
    It is much more straightforward to use and with less surprises. The limits actually do what they say the do. It also takes care of process lifecycle. And logs by default. It's preferable in every way. What it doesn't do is manipulate and transfer file system images, that's outside the scope. Which is probably the right design. The whole thing of having Docker manipulate images for you was a strange decision and makes CI/CD unnecessarily complicated.
herodoturtle 4y ago
This article was fun to read. It gives a nice history and overview of chroot. Irrespective of whether one agrees with the claim in the title, it’s an informative read. If you’re scanning through these comments, and know very little about chroot, I’d recommend you check this out. [Edit] The previous linkbaity title was (thankfully) changed by dang.
Someone1234 4y ago
Wouldn't it be more apt to say they're "chroot delivered via a package management system?" My big gripe with containers is how insecure many of them are, and how ignored the problem is because it is "someone else's problem." But even I won't pretend that taking a powerful tool like chroot (and or cgroups, etc) and then packaging them for rapid deployment/rollbacks/etc isn't key to their popularity.
klyrs 4y ago
> I pronounce chroot as change root. The real hot take
- psd1 4y ago
  Ain' dat de chroot
  - em-bee 4y ago
    "I am chroot"
- tambourine_man 4y ago
  Actually, it’s pronounced “charuto”, for cigar in Portuguese.
  - pie_flavor 4y ago
    Ch as in chord. 'Kroot'.
    - addaon 4y ago
      But "ot" as in depot. "Kruō".
  - kirubakaran 4y ago
    Originally from Tamil https://en.wikipedia.org/wiki/Cheroot https://en.wikipedia.org/wiki/Cheroot
- dools 4y ago
  I pronounce it like Dwight’s last name from the office
  - deleted 4y ago
    [deleted]
- BiteCode_dev 4y ago
liquidgecka 4y ago
Uhh.. No. "Containers" are a mix of two major pieces of technology. One is cgroups, the other is namespaces. Chroot is analogous (kind of) to mount namespaces is all. Containers need not use all of the features of all the cgroups and namespaces available. For example, from the linux command line I can use the "unshare" command line took to create namespaces to simulate some of the components without the other fanfare. for example, with namespaces I can remove network access to any process without changing anything else about the system. In this example the process run (ping) has no network devices at all. I love using this to ensure that a script doesn't attempt to download anything unexpected. $ sudo unshare -n ping 127.0.0.1 ping: connect: Network is unreachable I can also do things like remove files mounted under another mount. Make a directory called "foo", touch a file in that directory, then mount a ramdisk on top of it and the file is no longer accessible. I can use unshare -m to create a process with a new mount namespace, umount foo and the file becomes accessible again, all while other processes on the system can still see the contents of foo. A real world example of this is when the mysql database got copied onto the root disk before the mysql mount was created. You can leave mysql running while removing the over-mounted files without production risk. Cgroups are also nifty, you can restrict resources without changing the root file system. So I can ensure that no children of mine can consume more than 1G of RAM without fiddling with file systems at all. Systemd does this with its limits configurations and there are a ton of useful limits that can be applied. One of the coolest aspects of this is that you can do this per thread, meaning that you can do something like run your render thread in a network namespace without internet access, this preventing possible RCE on just that thread, same for mount namespaces and such, making it possible to chroot effectively a single thread in your application. So no, containers are not just fancy chroot, they are so very much more.
- TheDong 4y ago
  > "Containers" are a mix of two major pieces of technology. One is cgroups, the other is namespaces. I respectfully disagree with that definition of what a "container" is. cgroups, namespaces, and so on are an implementation detail of how containers are commonly implemented on linux these days. OpenVZ (https://en.wikipedia.org/wiki/OpenVZ https://en.wikipedia.org/wiki/OpenVZ) and Solaris Containers (aka zones) both predate the current linux implementation and did not use namespaces or cgroups. All three of those ways of running containers are containers. unshare/namespaces/cgroups is just one possible implementation. There are also, certainly, other ways of implementing containers, so it feels wrong to define it in terms of just one specific, admittedly the most popular, implementation.
asicsp 4y ago
See also: * https://jvns.ca/blog/2016/10/10/what-even-is-a-container/ https://jvns.ca/blog/2016/10/10/what-even-is-a-container/ * https://jvns.ca/blog/2020/04/27/new-zine-how-containers-work/ https://jvns.ca/blog/2020/04/27/new-zine-how-containers-work...
- mitnk 4y ago
  Also: - https://github.com/saschagrunert/demystifying-containers https://github.com/saschagrunert/demystifying-containers - https://ericchiang.github.io/post/containers-from-scratch/ https://ericchiang.github.io/post/containers-from-scratch/
its-summertime 4y ago
chroot's only use is to make / point somewhere else, it doesn't restrict file system access except during perfect weather conditions, and even then. https://man7.org/linux/man-pages/man2/chroot.2.html https://man7.org/linux/man-pages/man2/chroot.2.html > This call changes an ingredient in the pathname resolution process and does nothing else. In particular, it is not intended to be used for any kind of security purpose, neither to fully sandbox a process nor to restrict filesystem system calls. Actual restrictions are done with other methods, chroot is just to make things look nice inside the container
- cpuguy83 4y ago
  Runtimes typically use pivot_root and not chroot as well.
tamrix 4y ago
Or freebsd jails.
liquid153 4y ago
Can someone link me to the I think google vid of what a container is
latenightcoding 4y ago
Related: https://github.com/kazuho/jailing https://github.com/kazuho/jailing
keithnz 4y ago
I quite enjoyed Liz Rices video on containers... though I'm not quite sure which one I originally watched and she seems to have done the talk quite a few times ( and updated it ). But here's one https://www.youtube.com/watch?v=oSlheqvaRso https://www.youtube.com/watch?v=oSlheqvaRso
patrick451 4y ago
So I was trying to follow along with this and kept getting an error /# chroot /testroot /hello chroot: failed to run command ‘/hello’: No such file or directory If anybody was wondering, the trick is to statically link gcc hello.c -o hello -static -static-libgcc Or, use go for the hello executable
- BeefWellington 4y ago
  Or dynamically link and use ldd to identify and copy libraries the program is linking: ldd hello
  - chriswarbo 4y ago
    This is why I build containers using Nix (plus the reproducibility, caching, etc.): since everything has a path containing an unguessable hash, all dependencies must be referred to by their absolute path (either directly, or indirectly via an env var, etc.). That makes it trivial to find all the dependencies of a file (whether they're dynamic libraries, interpreter packages, image files, etc.), transitively, and tar them all up into a self-contained container :)
- adamgordonbell 4y ago
  Author here. Sorry, I should have added more instruction there. What I was expecting someone to do is use the 'pull' command made later to extract a statically linked hello from the hello:latest image, but I didn't explain how until later. So yes, thanks for including this, static linked is the way to go here, or ldd and copying other things in.
alpb 4y ago
tl;dr the author uses "docker export" to squash a container image layers into a rootfs, and uses chroot to make it the root of that filesystem and runs a binary there. This practically has nothing to do with the term "containers". Author says "Others will tell you containers are about namespaces [...] But for me, containers are just chrooted processes. " , however containers are very much about isolation, resource controls, and various network features like port mappings for many people. So you really didn't build anything even remotely resembling a container runtime, you redefined the term and built something that does "docker export | tar zxvf && cd /dir && chroot /dir && /prog". :)
- nunez 4y ago
  tbf, he could have also built the rootfs from source, though that takes a lot longer
blt 4y ago
tangent: in github.com/dspinellis/unix-history-repo linked from the article, the default branch is shown to have "∞ commits". Any ideas where this bug comes from? Something related to Unix epoch?
- evilc00kie 4y ago
  > Something related to Unix epoch? Nope, I just tested it. I'd guess there are too many branches with too many commits (e.g. all those FreeBSD ones with 10k-200k commits). Although you can click on branch and see the count of commits per branch, there might be an issue showing this information on the branch view due to some "DB-query-edge-case". You know, it doesn't seem to be that hard to solve this problem in the first place, but backends with their DBs and services can be a wild jungle of data correlation.
  - blt 4y ago
    What was your test? Do you mean the problem is in GitHub, not git? I suppose GitHub caches a lot of git's output in DBs to improve response time, sounds like fertile ground for bugs.
bjarneh 4y ago
> StarStruck is just there to check if you are paying attention. It’s not a container runtime at all, but Joe Exotic’s 2nd studio album.
ge96 4y ago
Oh that's cool he does the Corecursive podcast.
nunez 4y ago
Yes, containers are more than chroot, until you: - Want to give them their own IP addresses or networks, or - Put upper bounds on their resources, or - Get tired of dealing with chroot and unshare and seccomp and probably other tools I'm forgetting, or - You want to run an arm64 container on an x86 host with minimal configuration, or... this is a really fun (and important) exercise for anyone working with containers seriously to undergo but let's not trivialize how insanely easy Docked made creating containers become.
- simplotek 4y ago
  > (...) but let's not trivialize how insanely easy Docked made creating containers become. So much this. Docker doesn't get the respect it deserves. I'd also mention packaging/installing/deploying applications, which Docker made so trivial and trouble-free. Docker at it's core is a masterclass of UX.
  - nine_k 4y ago
    Its UX used to be... slightly suboptimal here and there. Nevertheless, it took the world by storm. IMHO it's because Docker made the few core things absurdly easy, not just low-friction, but zero-friction. That was key. The rest was important but not critical, and was built out eventually.
- nine_k 4y ago
  Frankly, you don't need much more for a proof of concept. Beside chroot, you basically need a namespace control utility, an iptables control utility, and a volume control utility. The "Docker in 100 lines of bash" [1] does just that. It skips the trouble of managing volumes by demanding btrfs; it could instead use stuff like tar / dd and mount to a comparable effect. This is, of course, about as fiddly, slow, and unreliable as, say, building an amplifier with discrete transistors, or building a clock with nixie tubes and 74xx chips, etc. The point is not in producing a sleek and reliable solution (though this is not ruled out), but in seeing and understanding how a thing works from inside. [1]: https://news.ycombinator.com/item?id=9925896 https://news.ycombinator.com/item?id=9925896
kramerger 4y ago
Didn't someone do this with the real container "API" and the code was not that much longer?. IIRC, she also used Go and it was more of a live coding talk than a blog post.
- aabbccde 4y ago
  Are you talking about this talk from Liz Rice titled "Containers from Scratch"[1] [1]: https://www.youtube.com/watch?v=oSlheqvaRso https://www.youtube.com/watch?v=oSlheqvaRso
  - kramerger 4y ago
    That's the one, thanks!
moss2 4y ago
Wow this explains so well what Docker is. Fancy building a $1 billion company around chroot.
- palata 4y ago
  Don't check what Slack (or WhatsApp before) is, then.
noufalibrahim 4y ago
Back before the birth of docker when I was consulting, a friends startup wanted a way to install their multi component application (web service, database etc.) inside a corporate environment. They built it on a Debian setup and generally apt-getted (apt-got?) all their requirements, used supervisord to manage the backend process (written in Python) and had it running. To do this on prem, they needed an installer. I cooked up this idea of creating a dummy file (dd), formatting it into a file system, mounting it as a loopback device and deb-bootstrapping it. Then I chrooted inside it, installed all the dependencies needed, added all the code and things we needed for the application, left the chroot and use makeself to create a self extracting zip file of the whole application along with an initial config dialogue. We wrapped up the process using a Python script so that people create such "installers". Once you ran the file, it would unzip itself into /opt/{something} and start up all the processes. Inside the chroot was Debian with a db and several other things running. Outside the chroot was Redhat or whatever. It ran surprisingly well and was used for a couple of years for all their customers even after Docker came out. I rejoined them as a contractor after they got a decent amount of funding and the same thing was still in use . I wrote about this as an answer to a Stack Overflow question back then https://stackoverflow.com/questions/5661385/packaging-and-shipping-a-python-library-and-scripts-the-professional-way/5731719#5731719 https://stackoverflow.com/questions/5661385/packaging-and-sh... It was a proto docker without proper process isolation. Just file system isolation using chroot. Definitely brought back memories. The real value docker added was standardization, a central container registry, tooling etc. It mainstreamed these kinds of arcane old school sysad tricks into a generally usable product.
- ilyt 4y ago
  We did that for some web hosting client it worked but managing this wasn't exactly pleasant. Then again there isn't much managing if instead of upgrading the system you rebuild it from scratch every time.
  - noufalibrahim 4y ago
    Yup. That's the reason why we did this.
- 4y ago
notpushkin 4y ago
Surprised no one has mentioned Bocker yet – “Docker implemented in around 100 lines of bash”. [1, 2] [1]: https://github.com/p8952/bocker/ https://github.com/p8952/bocker/ [2]: https://news.ycombinator.com/item?id=33218094 https://news.ycombinator.com/item?id=33218094 (116 comments)
- john-tells-all 4y ago
  People tend to think of Docker as big and complex and magic. Bocker lets us take a step back and not take Docker too seriously :)
crooksey 4y ago
I have been using chroot since approx 04 when I found it during my first Gentoo install, what docker did for me was make it so much easier to deploy various environments, due to how much exposure they have. It took a good amount of effort to make chroot both "cool" and widely available to both sys admins and developers. Sadly what it has done, is make users less aware of the core Linux/Unix principles that allow us to have cool software that just does it all for us.
froh 4y ago
meanwhile the chroot command line utility has been dropped from posix 2017: https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xcu_chap04.html https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xcu...
aaaaaalexas312 4y ago
1
zoobab 4y ago
1979 is the best year! chroot was written by one of the founders of Sun Microsystems, a cool guy: https://en.wikipedia.org/wiki/Bill_Joy https://en.wikipedia.org/wiki/Bill_Joy https://myhero.com/B_Joy_cvhs_cl_US_2016_ul https://myhero.com/B_Joy_cvhs_cl_US_2016_ul
- adamgordonbell 4y ago
  Wikipedia does say Bill Joy wrote chroot, but the unix history in the article says that it was DMR or Ken Thompson depending on which file you look at and chroot is listed in the V7 manual. So I think it predates the BSD Kernel.
  - jlokier 4y ago
    The BSD kernel is actually older then Unix V7, so it could plausibly have been added to BSD first.
artemonster 4y ago
There is junest that does all this nicely
tambourine_man 4y ago
I've been dreaming of a way of isolating and standardizing installations across Linux and macOS using chroot for a few years. Ideally Windows as well, but I'm not at all familiar with it, although if it's POSIX, it should be possible. Docker is cool on Linux, but having to virtualize an entire OS everywhere else is just too needlessly slow for situations where you just need to make sure you're running the same stack, not needing to ship the entire machine.
- adamgordonbell 4y ago
  I'm sure you know this, but OS X has chroot, so you could make a mac native chroot container for something by following the general direction the footnote of this article is pointing at. It would only work on macOS, but it would work.
  - tambourine_man 4y ago
    Yes, I know, that's what I was trying to convey
witnesser2 4y ago
This is not my area. Just audited Distributed System. This Docker thing, from beginning, sounds like try to implement a middleware using software. I don't recall but obscurely remember the distribute system guys has something about implementing a middleware. If the kernel is shared, it is not a container. It is like everything laid on the floor on crowded ferry. I would like to know if there is a specifiction of Docker. Particularly its Qos -- quality of service. That is the major consideration of any engineering. -- opinion
- witnesser2 4y ago
  I give a QOS example like a modem 24.4 kbps. Then for docker as middleware, eg throttle ticks per container per time frame etc.
john-tells-all 4y ago
Another take: DevBox provides 60% of the features of Docker while being faster and simpler. https://github.com/jetpack-io/devbox https://github.com/jetpack-io/devbox It provides filesystem isolation -- it uses Nix under the covers. This is great if you're a dev and don't want to mess with routing ports across a hidden VM to the host, just run a command and your service appears directly on the host. DevBox allows very rapid development. And for production, it exports to Docker. You get the best of all worlds!
- john-tells-all 4y ago
  Note DevBox is portable -- you don't need Linux! You don't need a VM, which would increase overhead and complexity. DevBox is limited, but very powerful in the features it provides.

501 points by mmphosis 4y ago

> for me, containers are just chrooted processes. Sure, they are more than that: Containers have a nice developer experience, an open-source foundation, and a whole ecosystem of cloud-native companies pushing them forward" "All right, but apart from the sanitation, the medicine, education, wine, public order, irrigation, roads, a fresh water system, and public health, what have the Romans ever done for us?"

lamontcg 4y ago

They're "RPMs" with some union mounted fanciness, a chroot, and some cgroups isolation to keep other people's fork bombs from taking out your service. Agreed that explainng containers should never start with VMs. I think it should start with package managers rather than chroots though.

chriswarbo 4y ago
I think .rpm/.deb is a bad analogy, since those (a) get installed into a global namespace (the / folder), (b) rely on the existence of other packages in that namespace (their dependencies) and (c) require global consistency between all the installed packages. You can avoid those global consistency problems by installing RPM/Deb packages into a chroot (or a "proper" container), but that ceases to be a good analogy of what containers are.
- lamontcg 4y ago
  Yeah I've spent like a decade building rpms that packaged all their deps inside of them, so I very much don't find the dependency management between them to be fundamental, which might not be intuitively obvious to everyone else. Maybe it'd be better to call them a tarball, but there is the way that you can install it and run it and then nuke it cleanly which you don't get out of the box with a tarball, which is why I like the package metaphor slightly better. EDIT: the fact that it is in a chroot, though, I think sort of covers the way that it doesn't really interact with the external filesystem and installed packages. That implies that you have to ship all the deps with the package itself and those usually would be installed into something like /chroot/<whatever> and never into e.g. /usr/bin and the nature of the chroot would make everything in /usr/bin inaccessible. If you build an RPM which installs into a chroot you're really not going to be able to have any dependencies on anything else in the system. The tweaks to make that separation slightly nicer are fairly small tweaks overall. That might not be intuitively obvious though.
  - chriswarbo 4y ago
    > Maybe it'd be better to call them a tarball, but there is the way that you can install it and run it and then nuke it cleanly which you don't get out of the box with a tarball, which is why I like the package metaphor slightly better. Containers are tarballs, e.g. the rootfs for runc, the layers of an image, and images themselves are usually tarballs. The extra stuff "you don't get out of the box with a tarball" are precisely the tools which we call "containers" (from low-level stuff like runc, to high-level things like online image registries)

efficax 4y ago

containers are cgroups and namespaces and are much more than just chroots, plus a set of tools for making those work

StopHammoTime 4y ago
I think the point is that it's not magic, but a very clear and obvious system that's been created using native tools. I've run into a lot of people (including Senior SRE/DevOps people) who don't realise that Containers are just another process running on a system.
- pushedx 4y ago
  A container is not a process. It’s an execution environment created by namespaces (mount, pid, network, others) and cgroups. A container can house many processes.
  - liquidgecka 4y ago
    ^-- this! Also, a container can house a container.. kind of. If you think of containers like file system inodes then there is a logical view of containers container1: (init -> dockerd) -> container2: (init -> dockerd) -> container3: (init -> dockerd) -> container4: (process) is perfectly feasible and from our view there is a hierarchy of containers, but in reality it would be possible assign a process in container4 to container1 and create all kinds of cool logical loops. This is effectively what a daemonset is in kubernetes. =)
    - lstodd 4y ago
      In reality there is no such thing as a 'container'. You can create an incredible self-intersecting mess of process namespaces. It is not what you want to do in production. Also, docker is an overengineered mess. Kubernetes only an couple of orders of magnitude more so.
  - G3rn0ti 4y ago
    On Windows and Macs Docker uses VMs ... I think the DEV community really needs a new definition of "containers" and one that is not backed by implementation details of Docker Linux.

akadempythag 4y ago

Fun deep dive. In a previous life I wrote web apps, and I liked to develop on Chromebooks to make sure that everything ran smoothly on low-end machines. I would set a price point of $2-300, which usually meant I was lucky to get a Rockchip. And this was before it was popular to build for ARM architectures. And yet, ChromeOS' chroot-based projects like crosh always managed to deliver. Surrounded by aluminum MacBooks and carbon-fiber Dells, I would run Rails apps and X GUIs and source builds on a rubberized hunk of plastic that had been designed for use in schools. I have always been surprised at how well that worked. If it was using the same fundamentals as Docker and Podman, I'm not surprised that the containerization movement has enjoyed such popularity.

spiffytech 4y ago
There was something really nice about developing on a Chromebook. It felt simplified in a very pleasant way. I ultimately discontinued the practice because beefy Chromebooks never become commonplace, some points of development friction got tiresome, and I stopped preferring Chrome. Yet I'm nostalgic for the experience.
j0hnyl 4y ago
Likewise it's amazing what one can do with Termux on an Android device.

bitwize 4y ago

Yeah, more or less, except containers also have a smoother UX and processes within a container are more successfully tricked into believing they're on a separate machine.

whalesalad 4y ago

I like containers, but I absolutely love chroot and debootstrap. It’s killer for when you need isolation (like building software locally without ruining the host OS) but don’t need the pomp and circumstance of containers.

mikepurvis 4y ago
I've long been a fan of basic debootstrap->chroot, though I will say in recent times systemd-nspawn and then buildah have definitely pretty much displaced chroot in my toolbox. They're equally pomp-free relative to Docker, but have a bunch of nice affordances in terms of a properly set up network including DNS and hosts, handling of filesystem permissions, and correctly presenting a read-only /proc.
- whalesalad 4y ago
  I’ve been meaning to grok nspawn - for my recent use cases the filesystem was the only level of isolation required but I’m gonna check it out for sure. Glad to hear it’s as good as I’ve heard it is.
- quickthrower2 4y ago
  If you are trying to tilt me over to always use linux instead of windows for all things you are doing a damn fine job. My big pain in any OS is messing it up by installing the wrong thing or the wrong way. But at the same time I want to play with different languages. I mean why use nvm just for node let alone anything else, when I can do this, it feels way cleaner.

exabrial 4y ago

I think the only correct way to do containers is… groan… how systemd does it.[ One should be able to launch a process and not let it touch anything else but it’s very process definition..] Thank you for your down vote. I look forward to being moderated out of existence.

sargun 4y ago
Perhaps a more productive comment (and I honestly am curious), would be -- what does systemd do right?
- XorNot 4y ago
  Systemd is containers, is the thing. Its entirely built around using cgroups to manage process resources, it just starts from the normal "I'm running root processes" level because it was intended to init a whole system. The one missing component was the concept of union mounting layered file systems from a repository.
  - jsmith45 4y ago
    The systemd-nspawn system (which is how running more more typical containers is normally implemented) is a little unusual because it it geared towards running os containers, including running their whole init system (via --boot or the template unit file). There is a reason why machinectl calls the input machine images, and both accept raw disk images. This is also reflected in the systemd-nspawn requirement of having an os-release file in the container. Of course, it is entirely capable of running application containers (hence features like --as-pid2), but some of the defaults like registering with systemd-machined don't make as much sense in those scenarios. Docker is nominally geared towards application containers, although it mostly works with OS containers. Indeed it seems like container images that contain a full OS (minus kernel) are probably more common than not.
- xorcist 4y ago
  It is much more straightforward to use and with less surprises. The limits actually do what they say the do. It also takes care of process lifecycle. And logs by default. It's preferable in every way. What it doesn't do is manipulate and transfer file system images, that's outside the scope. Which is probably the right design. The whole thing of having Docker manipulate images for you was a strange decision and makes CI/CD unnecessarily complicated.

herodoturtle 4y ago

This article was fun to read. It gives a nice history and overview of chroot. Irrespective of whether one agrees with the claim in the title, it’s an informative read. If you’re scanning through these comments, and know very little about chroot, I’d recommend you check this out. [Edit] The previous linkbaity title was (thankfully) changed by dang.

Someone1234 4y ago

Wouldn't it be more apt to say they're "chroot delivered via a package management system?" My big gripe with containers is how insecure many of them are, and how ignored the problem is because it is "someone else's problem." But even I won't pretend that taking a powerful tool like chroot (and or cgroups, etc) and then packaging them for rapid deployment/rollbacks/etc isn't key to their popularity.

klyrs 4y ago

> I pronounce chroot as change root. The real hot take

psd1 4y ago
Ain' dat de chroot
- em-bee 4y ago
  "I am chroot"
tambourine_man 4y ago
Actually, it’s pronounced “charuto”, for cigar in Portuguese.
- pie_flavor 4y ago
  Ch as in chord. 'Kroot'.
  - addaon 4y ago
    But "ot" as in depot. "Kruō".
- kirubakaran 4y ago
  Originally from Tamil https://en.wikipedia.org/wiki/Cheroot https://en.wikipedia.org/wiki/Cheroot
dools 4y ago
I pronounce it like Dwight’s last name from the office
- deleted 4y ago
  [deleted]
BiteCode_dev 4y ago

liquidgecka 4y ago

Uhh.. No. "Containers" are a mix of two major pieces of technology. One is cgroups, the other is namespaces. Chroot is analogous (kind of) to mount namespaces is all. Containers need not use all of the features of all the cgroups and namespaces available. For example, from the linux command line I can use the "unshare" command line took to create namespaces to simulate some of the components without the other fanfare. for example, with namespaces I can remove network access to any process without changing anything else about the system. In this example the process run (ping) has no network devices at all. I love using this to ensure that a script doesn't attempt to download anything unexpected. $ sudo unshare -n ping 127.0.0.1 ping: connect: Network is unreachable I can also do things like remove files mounted under another mount. Make a directory called "foo", touch a file in that directory, then mount a ramdisk on top of it and the file is no longer accessible. I can use unshare -m to create a process with a new mount namespace, umount foo and the file becomes accessible again, all while other processes on the system can still see the contents of foo. A real world example of this is when the mysql database got copied onto the root disk before the mysql mount was created. You can leave mysql running while removing the over-mounted files without production risk. Cgroups are also nifty, you can restrict resources without changing the root file system. So I can ensure that no children of mine can consume more than 1G of RAM without fiddling with file systems at all. Systemd does this with its limits configurations and there are a ton of useful limits that can be applied. One of the coolest aspects of this is that you can do this per thread, meaning that you can do something like run your render thread in a network namespace without internet access, this preventing possible RCE on just that thread, same for mount namespaces and such, making it possible to chroot effectively a single thread in your application. So no, containers are not just fancy chroot, they are so very much more.

TheDong 4y ago
> "Containers" are a mix of two major pieces of technology. One is cgroups, the other is namespaces. I respectfully disagree with that definition of what a "container" is. cgroups, namespaces, and so on are an implementation detail of how containers are commonly implemented on linux these days. OpenVZ (https://en.wikipedia.org/wiki/OpenVZ https://en.wikipedia.org/wiki/OpenVZ) and Solaris Containers (aka zones) both predate the current linux implementation and did not use namespaces or cgroups. All three of those ways of running containers are containers. unshare/namespaces/cgroups is just one possible implementation. There are also, certainly, other ways of implementing containers, so it feels wrong to define it in terms of just one specific, admittedly the most popular, implementation.

asicsp 4y ago

See also: * https://jvns.ca/blog/2016/10/10/what-even-is-a-container/ https://jvns.ca/blog/2016/10/10/what-even-is-a-container/ * https://jvns.ca/blog/2020/04/27/new-zine-how-containers-work/ https://jvns.ca/blog/2020/04/27/new-zine-how-containers-work...

mitnk 4y ago
Also: - https://github.com/saschagrunert/demystifying-containers https://github.com/saschagrunert/demystifying-containers - https://ericchiang.github.io/post/containers-from-scratch/ https://ericchiang.github.io/post/containers-from-scratch/

its-summertime 4y ago

chroot's only use is to make / point somewhere else, it doesn't restrict file system access except during perfect weather conditions, and even then. https://man7.org/linux/man-pages/man2/chroot.2.html https://man7.org/linux/man-pages/man2/chroot.2.html > This call changes an ingredient in the pathname resolution process and does nothing else. In particular, it is not intended to be used for any kind of security purpose, neither to fully sandbox a process nor to restrict filesystem system calls. Actual restrictions are done with other methods, chroot is just to make things look nice inside the container

cpuguy83 4y ago
Runtimes typically use pivot_root and not chroot as well.

tamrix 4y ago

Or freebsd jails.

liquid153 4y ago

Can someone link me to the I think google vid of what a container is

latenightcoding 4y ago

Related: https://github.com/kazuho/jailing https://github.com/kazuho/jailing

keithnz 4y ago

I quite enjoyed Liz Rices video on containers... though I'm not quite sure which one I originally watched and she seems to have done the talk quite a few times ( and updated it ). But here's one https://www.youtube.com/watch?v=oSlheqvaRso https://www.youtube.com/watch?v=oSlheqvaRso

patrick451 4y ago

So I was trying to follow along with this and kept getting an error /# chroot /testroot /hello chroot: failed to run command ‘/hello’: No such file or directory If anybody was wondering, the trick is to statically link gcc hello.c -o hello -static -static-libgcc Or, use go for the hello executable

BeefWellington 4y ago
Or dynamically link and use ldd to identify and copy libraries the program is linking: ldd hello
- chriswarbo 4y ago
  This is why I build containers using Nix (plus the reproducibility, caching, etc.): since everything has a path containing an unguessable hash, all dependencies must be referred to by their absolute path (either directly, or indirectly via an env var, etc.). That makes it trivial to find all the dependencies of a file (whether they're dynamic libraries, interpreter packages, image files, etc.), transitively, and tar them all up into a self-contained container :)
adamgordonbell 4y ago
Author here. Sorry, I should have added more instruction there. What I was expecting someone to do is use the 'pull' command made later to extract a statically linked hello from the hello:latest image, but I didn't explain how until later. So yes, thanks for including this, static linked is the way to go here, or ldd and copying other things in.

alpb 4y ago

tl;dr the author uses "docker export" to squash a container image layers into a rootfs, and uses chroot to make it the root of that filesystem and runs a binary there. This practically has nothing to do with the term "containers". Author says "Others will tell you containers are about namespaces [...] But for me, containers are just chrooted processes. " , however containers are very much about isolation, resource controls, and various network features like port mappings for many people. So you really didn't build anything even remotely resembling a container runtime, you redefined the term and built something that does "docker export | tar zxvf && cd /dir && chroot /dir && /prog". :)

nunez 4y ago
tbf, he could have also built the rootfs from source, though that takes a lot longer

blt 4y ago

tangent: in github.com/dspinellis/unix-history-repo linked from the article, the default branch is shown to have "∞ commits". Any ideas where this bug comes from? Something related to Unix epoch?

evilc00kie 4y ago
> Something related to Unix epoch? Nope, I just tested it. I'd guess there are too many branches with too many commits (e.g. all those FreeBSD ones with 10k-200k commits). Although you can click on branch and see the count of commits per branch, there might be an issue showing this information on the branch view due to some "DB-query-edge-case". You know, it doesn't seem to be that hard to solve this problem in the first place, but backends with their DBs and services can be a wild jungle of data correlation.
- blt 4y ago
  What was your test? Do you mean the problem is in GitHub, not git? I suppose GitHub caches a lot of git's output in DBs to improve response time, sounds like fertile ground for bugs.

bjarneh 4y ago

> StarStruck is just there to check if you are paying attention. It’s not a container runtime at all, but Joe Exotic’s 2nd studio album.

ge96 4y ago

Oh that's cool he does the Corecursive podcast.

nunez 4y ago

Yes, containers are more than chroot, until you: - Want to give them their own IP addresses or networks, or - Put upper bounds on their resources, or - Get tired of dealing with chroot and unshare and seccomp and probably other tools I'm forgetting, or - You want to run an arm64 container on an x86 host with minimal configuration, or... this is a really fun (and important) exercise for anyone working with containers seriously to undergo but let's not trivialize how insanely easy Docked made creating containers become.

simplotek 4y ago
> (...) but let's not trivialize how insanely easy Docked made creating containers become. So much this. Docker doesn't get the respect it deserves. I'd also mention packaging/installing/deploying applications, which Docker made so trivial and trouble-free. Docker at it's core is a masterclass of UX.
- nine_k 4y ago
  Its UX used to be... slightly suboptimal here and there. Nevertheless, it took the world by storm. IMHO it's because Docker made the few core things absurdly easy, not just low-friction, but zero-friction. That was key. The rest was important but not critical, and was built out eventually.
nine_k 4y ago
Frankly, you don't need much more for a proof of concept. Beside chroot, you basically need a namespace control utility, an iptables control utility, and a volume control utility. The "Docker in 100 lines of bash" [1] does just that. It skips the trouble of managing volumes by demanding btrfs; it could instead use stuff like tar / dd and mount to a comparable effect. This is, of course, about as fiddly, slow, and unreliable as, say, building an amplifier with discrete transistors, or building a clock with nixie tubes and 74xx chips, etc. The point is not in producing a sleek and reliable solution (though this is not ruled out), but in seeing and understanding how a thing works from inside. [1]: https://news.ycombinator.com/item?id=9925896 https://news.ycombinator.com/item?id=9925896

kramerger 4y ago

Didn't someone do this with the real container "API" and the code was not that much longer?. IIRC, she also used Go and it was more of a live coding talk than a blog post.

aabbccde 4y ago
Are you talking about this talk from Liz Rice titled "Containers from Scratch"[1] [1]: https://www.youtube.com/watch?v=oSlheqvaRso https://www.youtube.com/watch?v=oSlheqvaRso
- kramerger 4y ago
  That's the one, thanks!

moss2 4y ago

Wow this explains so well what Docker is. Fancy building a $1 billion company around chroot.

palata 4y ago
Don't check what Slack (or WhatsApp before) is, then.

noufalibrahim 4y ago

Back before the birth of docker when I was consulting, a friends startup wanted a way to install their multi component application (web service, database etc.) inside a corporate environment. They built it on a Debian setup and generally apt-getted (apt-got?) all their requirements, used supervisord to manage the backend process (written in Python) and had it running. To do this on prem, they needed an installer. I cooked up this idea of creating a dummy file (dd), formatting it into a file system, mounting it as a loopback device and deb-bootstrapping it. Then I chrooted inside it, installed all the dependencies needed, added all the code and things we needed for the application, left the chroot and use makeself to create a self extracting zip file of the whole application along with an initial config dialogue. We wrapped up the process using a Python script so that people create such "installers". Once you ran the file, it would unzip itself into /opt/{something} and start up all the processes. Inside the chroot was Debian with a db and several other things running. Outside the chroot was Redhat or whatever. It ran surprisingly well and was used for a couple of years for all their customers even after Docker came out. I rejoined them as a contractor after they got a decent amount of funding and the same thing was still in use . I wrote about this as an answer to a Stack Overflow question back then https://stackoverflow.com/questions/5661385/packaging-and-shipping-a-python-library-and-scripts-the-professional-way/5731719#5731719 https://stackoverflow.com/questions/5661385/packaging-and-sh... It was a proto docker without proper process isolation. Just file system isolation using chroot. Definitely brought back memories. The real value docker added was standardization, a central container registry, tooling etc. It mainstreamed these kinds of arcane old school sysad tricks into a generally usable product.

ilyt 4y ago
We did that for some web hosting client it worked but managing this wasn't exactly pleasant. Then again there isn't much managing if instead of upgrading the system you rebuild it from scratch every time.
- noufalibrahim 4y ago
  Yup. That's the reason why we did this.
4y ago

notpushkin 4y ago

Surprised no one has mentioned Bocker yet – “Docker implemented in around 100 lines of bash”. [1, 2] [1]: https://github.com/p8952/bocker/ https://github.com/p8952/bocker/ [2]: https://news.ycombinator.com/item?id=33218094 https://news.ycombinator.com/item?id=33218094 (116 comments)

john-tells-all 4y ago
People tend to think of Docker as big and complex and magic. Bocker lets us take a step back and not take Docker too seriously :)

crooksey 4y ago

I have been using chroot since approx 04 when I found it during my first Gentoo install, what docker did for me was make it so much easier to deploy various environments, due to how much exposure they have. It took a good amount of effort to make chroot both "cool" and widely available to both sys admins and developers. Sadly what it has done, is make users less aware of the core Linux/Unix principles that allow us to have cool software that just does it all for us.

froh 4y ago

meanwhile the chroot command line utility has been dropped from posix 2017: https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xcu_chap04.html https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xcu...

aaaaaalexas312 4y ago

zoobab 4y ago

1979 is the best year! chroot was written by one of the founders of Sun Microsystems, a cool guy: https://en.wikipedia.org/wiki/Bill_Joy https://en.wikipedia.org/wiki/Bill_Joy https://myhero.com/B_Joy_cvhs_cl_US_2016_ul https://myhero.com/B_Joy_cvhs_cl_US_2016_ul

adamgordonbell 4y ago
Wikipedia does say Bill Joy wrote chroot, but the unix history in the article says that it was DMR or Ken Thompson depending on which file you look at and chroot is listed in the V7 manual. So I think it predates the BSD Kernel.
- jlokier 4y ago
  The BSD kernel is actually older then Unix V7, so it could plausibly have been added to BSD first.

artemonster 4y ago

There is junest that does all this nicely

tambourine_man 4y ago

I've been dreaming of a way of isolating and standardizing installations across Linux and macOS using chroot for a few years. Ideally Windows as well, but I'm not at all familiar with it, although if it's POSIX, it should be possible. Docker is cool on Linux, but having to virtualize an entire OS everywhere else is just too needlessly slow for situations where you just need to make sure you're running the same stack, not needing to ship the entire machine.

adamgordonbell 4y ago
I'm sure you know this, but OS X has chroot, so you could make a mac native chroot container for something by following the general direction the footnote of this article is pointing at. It would only work on macOS, but it would work.
- tambourine_man 4y ago
  Yes, I know, that's what I was trying to convey

witnesser2 4y ago

This is not my area. Just audited Distributed System. This Docker thing, from beginning, sounds like try to implement a middleware using software. I don't recall but obscurely remember the distribute system guys has something about implementing a middleware. If the kernel is shared, it is not a container. It is like everything laid on the floor on crowded ferry. I would like to know if there is a specifiction of Docker. Particularly its Qos -- quality of service. That is the major consideration of any engineering. -- opinion

witnesser2 4y ago
I give a QOS example like a modem 24.4 kbps. Then for docker as middleware, eg throttle ticks per container per time frame etc.

john-tells-all 4y ago

Another take: DevBox provides 60% of the features of Docker while being faster and simpler. https://github.com/jetpack-io/devbox https://github.com/jetpack-io/devbox It provides filesystem isolation -- it uses Nix under the covers. This is great if you're a dev and don't want to mess with routing ports across a hidden VM to the host, just run a command and your service appears directly on the host. DevBox allows very rapid development. And for production, it exports to Docker. You get the best of all worlds!

john-tells-all 4y ago
Note DevBox is portable -- you don't need Linux! You don't need a VM, which would increase overhead and complexity. DevBox is limited, but very powerful in the features it provides.