9 ms·

The fact that containers are such a simple technology always makes me think this: Why is it still necessary to have whole, full-blown OS filesystems inside of

The fact that containers are such a simple technology always makes me think this: Why is it still necessary to have whole, full-blown OS filesystems inside of our containers, if their purpose is running a single binary? Dependencies/dynamic libraries are decent reason, sure. But wouldn't it make more sense to do things "bottom-up"? i.e. starting from an empty filesystem, and then progressively adding the files that are absolutely necessary for the binary to work, instead of the "top-down" approach, which starts from a complete OS filesystem and then starts removing the things that are not needed?

compsciphd 3y ago
it isnt and very few people actually think that. plenty of "from scratch" images. ergo any image that is a single go binary
- silisili 3y ago
  Never made a ton of sense to me. Go crosscompiles easily, ship a binary. Because the second you want to do anything involving https, you need certificates, and that's where having a minimal but existent base image starts shining, and mostly goes up from there...
  - speedgoose 3y ago
    Once you have a hammer… One advantage of software containers is to have this unique interface that is the same whatever is inside the container. Like shipping containers. If it's one single golang binary, or a weird python container running only on specific version of Debian compiled during a blue moon, or some 8GB java enterprise bloatware, it's the same.
    - claytongulick 3y ago
      See, this is actually my problem with containers. "or a weird python container running only on specific version of Debian compiled during a blue moon" This. I guess I've been fortunate that I'm able to reject software like this from my stack. I know that everyone isn't so lucky. I mostly do nodejs, and have zero need for containers when a simple npm install gets all deps. Or if I need performance, a single go binary. I tried doing the container thing just to understand how it all works and what the hype is about. It seemed needlessly complex and hard to develop/debug. For more complex situations where you need a bunch of interacting programs and services, I prefer stuff like Ansible and VMs, or just manually setting up a base image. I guess it's a "get off my lawn" kind of thing. It seems like containers are used a lot by folks who don't want to learn ops, like how to install and configure postgres, redis, etc... I think that's a mistake, and just pushes the problem onto others who have to support the software in production.
  - maccard 3y ago
    Containers are more than "just" running a binary now though. If your "deploy" script is: docker build -t my-app . && docker push my-app Then all of a sudden it's a reproducible, reusable deployment script that works for any language, any application, and that any other dev on your team can run (as opposed to playing "which flavor of coreutils did they use when they wrote this?"). It's provider agnostic - you can run it on DO, AWS, GCP, whatver. You get free rolling/blue green/canary/whatever you prefer deployments, a "basic" cross compilation out of the box. They're not magic, but they are an excellent abstraction, despite the warts.
shepherdjerred 3y ago
It is a bit nice to have a "full" operating system in your container, especially for debugging. There doesn't seem to be a _huge_ drawback to running an Ubuntu image vs Alpine, aside from the image size.
- AnonCoward42 3y ago
  Ironically Ubuntu image is worse for debugging than Alpine by default even if it is substantially bigger. `vi` and `less` are missing from Ubuntu for example and are part of Alpine.
  - gjvc 3y ago
    So? All you have to do is "docker exec -it container /bin/bash" and run "apt update" and "apt install vim", then remake the image with fixes or re-run the container once you've figured it out.
    - AnonCoward42 3y ago
      You can always smuggle the debugging tools in even `FROM scratch`, but my point was that "by default" these debugging tools were missing on Ubuntu.
  - shepherdjerred 3y ago
    The benefit of Ubuntu, at least for me, is that I know it pretty well. Alpine isn't bad, I'm just not familiar with it.
    - tracker1 3y ago
      Try debian-slim, should mostly be the same, but save you 100mb+ off of your final size most of the time.
- KronisLV 3y ago
  > There doesn't seem to be a _huge_ drawback to running an Ubuntu image vs Alpine, aside from the image size. I switched from using Alpine for my own images to using Ubuntu instead a while ago: https://blog.kronis.dev/articles/using-ubuntu-as-the-base-for-all-of-my-containers https://blog.kronis.dev/articles/using-ubuntu-as-the-base-fo... So far, it's surprisingly nice. Ubuntu based images take up more space, but thanks to layer reuse and storage being affordable, this isn't a big issue in practice (at my scale). There are no surprises in regards to performance or package management that I have to deal with. Actually, I can use Ubuntu/Linux Mint locally and have the same install instructions for tools/dependencies as well (if I want to test things outside of containers, running on the system directly). It's also really nice to be able to build my own base image with whatever tools I want available (e.g. nano and some debugging stuff) and have them be available in all of the language/stack-specific images that I build later. The EOL is also pretty long and I don't think that there's any shame in having a "good enough" and somewhat boring distro for most dev stuff, only occasionally looking at PPAs for newer stuff. That said, Alpine and Debian are both fine as well! I still run Bitnami images for most complex software like databases, which I think used Debian as a base: https://bitnami.com/stacks/containers https://bitnami.com/stacks/containers
kbenson 3y ago
As a sibling noted, people do that. But disk space is cheap, and you aren't really losing much other than disk space for unused portions of the OS that are in a container, but what you gain is a normalized environment that's easy to develop for, test, and add to as needed.
- takeda 3y ago
  If your container is big it also takes long time to fetch and start it
  - esperent 3y ago
    Alpine docker container starts at 5mb.
    - takeda 3y ago
      OP was arguing that today the big size doesn't matter, because disk space is cheap. I was providing argument that it still does.
  - notpushkin 3y ago
    Base image layers are deduplicated though. Now if only we could pick a single base image...
    - chaxor 3y ago
      Alpine or Debian. Those are the only base images you ever need.
      - tracker1 3y ago
        Specifically, Debian-slim. But even then, the versions are varied.
    - takeda 3y ago
      Theoretically that would work, in practice it doesn't. Making distroless containers is the way to go.
sim7c00 3y ago
this is what u are supposed to do, but people generally are lazy and for complex binaries or programs this can be a bit of a fiddley task honestly. you can write scripts which dump out all you need if u want. when people only had 'jails' still i think this was more common. rapid development kills all :D
IshKebab 3y ago
Because the main point of Docker is to allow you to run programs in their own OS. It became popular as a software distribution method for typical Linux software that is not self-contained - requires a ton of system dependencies and spews itself all over the filesystem. If your program doesn't do that - e.g. most Rust and Go programs which are a single statically linked binary - then there's very little reason to use Docker in the first place. People will say "but you still want to containerise things!" or "what about orchestration?" and sure those are some incidental benefits now, but they only really happened because everyone was more or less forced to use Docker to actually run apps. Docker is a workaround for shitty software.
- chaxor 3y ago
  "Docker is a workaround for shitty software" This just needs to be repeated for the pythonistas sitting in the back.
- claytongulick 3y ago
  Or, as another HN user put it a few years ago: "Docker is static linking for millennials" :D
mindwok 3y ago
This isn’t too far from how most containers are built nowadays. Usually you just use Alpine or slim Debian images. The benefit of using those distro images is that they provide a lot of utilities you might want in a container, like a shell. As an aside, the approach you described is how Nix works, and you can use Nix to produce OCI images.
- iso1631 3y ago
  > The benefit of using those distro images is that they provide a lot of utilities you might want in a container, like a shell. Why would you want a shell in a container? Surely the idea of a container is to run a program, not act as interactive environment.
  - kroolik 3y ago
    Troubleshooting inside the container or simple scripting.
  - rolisz 3y ago
    To debug stuff.
  - leodag 3y ago
    It's not necessarily interactive, it's also pretty common to have a shell script for initialization/setup.
  - the-alchemist 3y ago
    Like others have said, for debugging, although the proper-but-rarely-done solution is to use PID namespaces to attach another container containing the shell, rather than including it in the raw Docker image.
  - giobox 3y ago
    "docker exec -it" debugging/troubleshooting. It's generally harder to debug/troubleshoot random issues if there is no shell in the container, and you can get a working shell in a 10mb container image so not like it adds some dramatic inconvenience or size. Sure its not 100 percent necessary, but it's enormously convenient at times and I personally wouldn't build my own container images without a shell I can drop into usually, unless there are technical or security reasons to avoid doing it for the given application. The shell can also be useful for certain scripting steps during image build, not all containers are just going to copy in a statically compiled binary etc. In fact, I'd say the latter is significantly rarer than the former, at least in my experience. Being able to do commands like: $ docker exec -it mycontainer sh is generally very helpful during development. > https://docs.docker.com/engine/reference/commandline/exec/ https://docs.docker.com/engine/reference/commandline/exec/
raesene9 3y ago
There's plenty of ways to build small container images that don't start with a "full" distribution, but whether they work for specific use cases will depend on what the applications need. FROM Scratch - starts with a totally blank image, this is the smallest option but your application must work with no supporting files (e.g. statically compiled binaries). Distroless - has a small number of standard OS support files but no package manager, so works where you don't need to install many OS packages. Wolfi - Newer than the others, they're building an ecosystem of minimal images for specific purposes.
- RandomBK 3y ago
  And let's not forget Alpine, which is a fairly common minimalist distro clocking in a ~5MB
takeda 3y ago
If you use Nix that's pretty much how the container will be build. You specify that your app should be there and Nix, being aware of all dependencies will place inside everything needed.
- iopq 3y ago
  Came to say this, in the case of sane things* Nix works perfectly * pip is not guaranteed to get you to a working state even if you run the same commands, so don't even think about it
- aliasxneo 3y ago
  > will place inside everything needed And more. In my experience, depending on the quality of the packages you happen to be consuming, you may end up with a container twice as large as a comparable (say, Alpine) container. For example, I once tried to bring git into my Nix container and was surprised to see over 300MB in increased size. After having built Nix containers exclusively for a year now, I wouldn't really recommend it to others unless you're willing to invest a lot of additional time cleaning up community packages.
  - takeda 3y ago
    That's because by default Nix uses glibc (which is huge), while Alpine uses musl. You can create musl binaries, but you need to know Nix a little more and know how to cross compile. Another great thing is that you can can easily modify existing package to remove some of their dependencies that you don't use. This is shown here with packaging redis into a container here and making it 3x smaller than the alpine version: https://nixos.org/#asciinema-demo-cover https://nixos.org/#asciinema-demo-cover There's of course possibility that someone specified build dependency as a runtime dependency, but if that was done that's a bug.
nigma1337 3y ago
Lots of examples without the entire OS as other comments mention, an example would be Googles distroless[0] [0]: https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless
snickerer 3y ago
Because building a sane filesystem with only the needed libraries needs knowledge and some effort, whereas containers exist to keep people from using their brains. If you know what you are doing you have several alternatives: 1. Just link your binary statically. 2. Set the path to the picked out special libs as an environment variable only for the binary. 3. Re-think what's wrong with your system that leads to the pain of you wanting other lib versions for that binary.
- tracker1 3y ago
  1. Statically linking your application, then distributing that application, such as via Docker Hub to others will then require your application to be GPL licensed in many, many cases within the Linux ecosystem. This may not be desired. 2. You can absolutely limit the system libraries that are included to only those needed. Some platforms make this easier or harder than others. 3. That you have other software that you want to run in parallel and don't want to spin up full virtualization or manage various chroot structures to organize. I'd also add that just because a tool makes it easy to do something, and means you can bloat your file system doesn't make it inherently bad. Not everyone is trying to run a large database on a potato.
dlahoda 3y ago
nix does incremental file system. i use process-compose to build and manage process hierarchy.
tauroid 3y ago
sibling comments about Nix are not acknowledging that the base image size is around 600MB vs 5MB for alpine
- adisbladis 3y ago
  This is only true if use a Docker based workflow using `FROM nixos/nix`. This image exists mainly as a way for people to try out Nix with, not to build production images on top of. We ship many things which bloat the image size but makes it nicer for interactive usage. Using dockerTools from nixpkgs is much better and gives you much smaller images closer to Alpine size.
  - tauroid 3y ago
    I might have confused download volume with image size but the tar.gz for dockerTools.buildLayeredImage with just node and mariadb in the contents is still 220MB (just checked) Edit: with nothing in the contents it's 144M, which is getting reasonable but still nearly 30x alpine base
pravus 3y ago
> Why is it still necessary to have whole, full-blown OS filesystems inside of our containers, if their purpose is running a single binary? It's not. This is just the path of least resistance for most container builds since many applications require various supporting files to function properly. Most of my Go containers only contain 2 files: the go binary and ca certificates. > Dependencies/dynamic libraries are decent reason, sure. But wouldn't it make more sense to do things "bottom-up"? In theory, yes. With dynamic loading and support for loading plugins and runtime this is virtually impossible to do. That's why starting from a base image that already includes everything you need is generally the chosen method. BSD jails/chroot operate this way and it's an absolute pain to setup for complicated applications.
asimpletune 3y ago
On a related note. I wonder whatever happened to unikernels and why they haven't become the norm.
- nubinetwork 3y ago
  https://en.m.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_debate https://en.m.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_d...
  - reegnz 3y ago
    unikernel is not the same as microkernel. I've found these after some quick googling: https://unikraft.org/ https://unikraft.org/ https://hermitcore.org/ https://hermitcore.org/ https://nanos.org/ https://nanos.org/ Seems to be a living concept still, just not in the mainstream.
    - flaminHotSpeedo 3y ago
      There are mainstream use cases, particularly for cloud providers who want lightweight runtimes with hypervisor managed isolation. The only "example" I can think of off hand is Firecracker, but I'm not 100% confident that Firecracker is technically a unikernels. My guess is that there's a number of unikernel implementations behind closed doors that see heavy use
intelVISA 3y ago
The extent people will go to reinvent a worse, slower and more insecure ver of static bins + BSD jails is absurd...
db48x 3y ago
It’s not. You you can use systemd-nspawn to create a container that uses your own root filesystem by specifying --directory=/ --volatile=yes. This mounts a tmpfs into the container’s root, and then mounts your /usr into the container’s /usr in read–only mode. This allows the container to run all the software installed on your machine, while redirecting writes to the tmpfs. Alternatively, instead of --directory=/ you could specify some other directory that contains an OS image (such as --directory=/var/lib/machines/debian-bookworm or --directory=/var/lib/machines/fedora-38). Multiple containers can transparently share the same image, since all the writes go to a per–container tmpfs. https://0pointer.net/blog/running-an-container-off-the-host-usr.html https://0pointer.net/blog/running-an-container-off-the-host-...
- jeroenhd 3y ago
  That's an interesting way to use systemd-nspawn! But, if it mounts everything, wouldn't that also make container escapes very easy?
  - db48x 3y ago
    When it is volatile, it really only mounts the /usr inside the specified directory, rather than the directory itself. In particular, /dev and /etc will be empty.
mschuster91 3y ago
> But wouldn't it make more sense to do things "bottom-up"? i.e. starting from an empty filesystem, and then progressively adding the files that are absolutely necessary for the binary to work Unless you're using Go or a C/C++ stack that does static linking and thus needs no dependencies including libc, that's yak shaving to an extreme degree.
adamgordonbell 3y ago
It's fun the play around with going that way. I did via chroot, and using ldd to see what a program like redis needs copied in[1]. But eventually you might want to get a shell into the little environment, and then page thru logs from inside it, and so eventually, unless you are going pure from-scratch, statically linked, you might want something like alpine in there. But its funny how as a solution to dynamic linking is shipping a whole OS. [1] https://www.youtube.com/watch?v=JOsWB50LmwQ https://www.youtube.com/watch?v=JOsWB50LmwQ
imiric 3y ago
> The fact that containers are such a simple technology But they're really not. This is a common misconception because of the user friendliness of Docker, but underneath all that there are many moving parts that the Linux kernel exposes, which are tricky to manage manually, and additional features of Docker itself (Dockerfile, layered images, distribution, etc.). Sure, you can write a shell script that does a tiny fraction of what container tools do, but then you'd have a half-baked solution that reinvents the wheel because... it's lighterweight? What TFA is doing is fine if you want to learn how containers work and impress your colleagues, but for real world usage, stick to the established tooling. Like others mentioned, there are ways to do what you propose, and you can always create your own container tool based on that shell script that automates this :). Though you might be interested in unikernels, which is an extreme version of that approach using VMs.
kodah 3y ago
Docker came about when lxc was the container interface on Linux. It's hardly user friendly. Doing everything yourself is possible, but again is really only user friendly for very simple operations. For instance, most users would probably find network name spacing or even uid/gid mapping pretty confusing. These days the images produced are quite different in that Docker now produces OCI compatible images, which makes them run anywhere. lxc on the other hand will produce images that run only on Linux. lxd eventually came around which is a lot more analogous to Docker as an application and outputs OCI compatible images.
- ianlevesque 3y ago
  I’m confused. If OCI images run anywhere, why does Docker Desktop for windows and mac run virtual machines with a Linux kernel behind the scenes?
  - jeroenhd 3y ago
    Most container images contain binaries that were made to use Linux syscalls for things like I/O and memory management. Windows had an implementation of the Linux kernel interface for this exact purpose (WSL 1) but because of performance and compatibility challenges, they switched to plain virtual machines. These days, virtual machine run with almost 0 overhead anyway, so if you want to run a Linux binary they're a much simpler option than implementing a whole operating system.
    - tracker1 3y ago
      True enough... When I have been under Windows, a lot of things still run faster under WSL-Ubuntu than in Windows. Hyper-V is pretty nice, generally speaking, and the WSL tooling over it is pretty great. Makes having Windows as a host OS mostly tolerable. I used to joke that "WSL" was my favorite linux distro. I switched at home FT when I saw ads in my start menu search results that first time. The Edge nags weren't enough, the pre-installed games, etc. But an ad before what I was looking for on my local system, that was too far. I realize it was only a "test" feature, but even that someone wanted to test such a thing sickens me. I've still had to sometimes run under Windows at work. I prefer the M1/M2 macs now, just for silence and battery life, but they don't run Docker nearly as well, at least most of the x86_64 container issues are resolved, and most of what I touch has aarch64 targets.
goodpoint 3y ago
> Why is it still necessary to have whole, full-blown OS filesystems inside of our containers, if their purpose is running a single binary? It's not and Docker carries plenty of unnecessary complexity.
hardware2win 3y ago
Even better Get rid of those bloated OSes and run apps directly on HW/vHW
- claytongulick 3y ago
  That's what I do. I understand that there are some good use cases for containers, but I still haven't encountered a need for them. I run everything from a /home/app folder, with the app user and restricted permissions. I avoid projects that have container-only deployment (supabase, for example). Whenever I peek behind the curtain, I end up horrified at the complexity. Honestly, I just can't imagine why someone would want to run postgres, for example, in a container. Seems like a nightmare for maintenance and production support.
  - t0astbread 3y ago
    I think they meant "run apps directly on hardware / virtual hardware". As in building the app with a unikernel, bypassing the need for a (large) operating system.
eldridgea 3y ago
It's a challenge but seems to be becoming moire prevalent for folks focused on supply chain attacks. Chainguard in particular has a bunch of standard containers (e.g. nginx) that reduce the container footprint as much as possible. https://www.chainguard.dev/chainguard-images https://www.chainguard.dev/chainguard-images
m463 3y ago
It's for portability. Your alpine binary can run in an alpine container. But folks run that alpine container on ubuntu 18.04 or debian 11 or macos or wsl2. also, docker brings a lot of usefulness to this mix - dockerfile "recipe" that builds on other recipes, layered filesystem sort of like version control, global namespace, etc

compsciphd 3y ago

it isnt and very few people actually think that. plenty of "from scratch" images. ergo any image that is a single go binary

silisili 3y ago
Never made a ton of sense to me. Go crosscompiles easily, ship a binary. Because the second you want to do anything involving https, you need certificates, and that's where having a minimal but existent base image starts shining, and mostly goes up from there...
- speedgoose 3y ago
  Once you have a hammer… One advantage of software containers is to have this unique interface that is the same whatever is inside the container. Like shipping containers. If it's one single golang binary, or a weird python container running only on specific version of Debian compiled during a blue moon, or some 8GB java enterprise bloatware, it's the same.
  - claytongulick 3y ago
    See, this is actually my problem with containers. "or a weird python container running only on specific version of Debian compiled during a blue moon" This. I guess I've been fortunate that I'm able to reject software like this from my stack. I know that everyone isn't so lucky. I mostly do nodejs, and have zero need for containers when a simple npm install gets all deps. Or if I need performance, a single go binary. I tried doing the container thing just to understand how it all works and what the hype is about. It seemed needlessly complex and hard to develop/debug. For more complex situations where you need a bunch of interacting programs and services, I prefer stuff like Ansible and VMs, or just manually setting up a base image. I guess it's a "get off my lawn" kind of thing. It seems like containers are used a lot by folks who don't want to learn ops, like how to install and configure postgres, redis, etc... I think that's a mistake, and just pushes the problem onto others who have to support the software in production.
- maccard 3y ago
  Containers are more than "just" running a binary now though. If your "deploy" script is: docker build -t my-app . && docker push my-app Then all of a sudden it's a reproducible, reusable deployment script that works for any language, any application, and that any other dev on your team can run (as opposed to playing "which flavor of coreutils did they use when they wrote this?"). It's provider agnostic - you can run it on DO, AWS, GCP, whatver. You get free rolling/blue green/canary/whatever you prefer deployments, a "basic" cross compilation out of the box. They're not magic, but they are an excellent abstraction, despite the warts.

shepherdjerred 3y ago

It is a bit nice to have a "full" operating system in your container, especially for debugging. There doesn't seem to be a _huge_ drawback to running an Ubuntu image vs Alpine, aside from the image size.

AnonCoward42 3y ago
Ironically Ubuntu image is worse for debugging than Alpine by default even if it is substantially bigger. `vi` and `less` are missing from Ubuntu for example and are part of Alpine.
- gjvc 3y ago
  So? All you have to do is "docker exec -it container /bin/bash" and run "apt update" and "apt install vim", then remake the image with fixes or re-run the container once you've figured it out.
  - AnonCoward42 3y ago
    You can always smuggle the debugging tools in even `FROM scratch`, but my point was that "by default" these debugging tools were missing on Ubuntu.
- shepherdjerred 3y ago
  The benefit of Ubuntu, at least for me, is that I know it pretty well. Alpine isn't bad, I'm just not familiar with it.
  - tracker1 3y ago
    Try debian-slim, should mostly be the same, but save you 100mb+ off of your final size most of the time.
KronisLV 3y ago
> There doesn't seem to be a _huge_ drawback to running an Ubuntu image vs Alpine, aside from the image size. I switched from using Alpine for my own images to using Ubuntu instead a while ago: https://blog.kronis.dev/articles/using-ubuntu-as-the-base-for-all-of-my-containers https://blog.kronis.dev/articles/using-ubuntu-as-the-base-fo... So far, it's surprisingly nice. Ubuntu based images take up more space, but thanks to layer reuse and storage being affordable, this isn't a big issue in practice (at my scale). There are no surprises in regards to performance or package management that I have to deal with. Actually, I can use Ubuntu/Linux Mint locally and have the same install instructions for tools/dependencies as well (if I want to test things outside of containers, running on the system directly). It's also really nice to be able to build my own base image with whatever tools I want available (e.g. nano and some debugging stuff) and have them be available in all of the language/stack-specific images that I build later. The EOL is also pretty long and I don't think that there's any shame in having a "good enough" and somewhat boring distro for most dev stuff, only occasionally looking at PPAs for newer stuff. That said, Alpine and Debian are both fine as well! I still run Bitnami images for most complex software like databases, which I think used Debian as a base: https://bitnami.com/stacks/containers https://bitnami.com/stacks/containers

kbenson 3y ago

As a sibling noted, people do that. But disk space is cheap, and you aren't really losing much other than disk space for unused portions of the OS that are in a container, but what you gain is a normalized environment that's easy to develop for, test, and add to as needed.

takeda 3y ago
If your container is big it also takes long time to fetch and start it
- esperent 3y ago
  Alpine docker container starts at 5mb.
  - takeda 3y ago
    OP was arguing that today the big size doesn't matter, because disk space is cheap. I was providing argument that it still does.
- notpushkin 3y ago
  Base image layers are deduplicated though. Now if only we could pick a single base image...
  - chaxor 3y ago
    Alpine or Debian. Those are the only base images you ever need.
    - tracker1 3y ago
      Specifically, Debian-slim. But even then, the versions are varied.
  - takeda 3y ago
    Theoretically that would work, in practice it doesn't. Making distroless containers is the way to go.

sim7c00 3y ago

this is what u are supposed to do, but people generally are lazy and for complex binaries or programs this can be a bit of a fiddley task honestly. you can write scripts which dump out all you need if u want. when people only had 'jails' still i think this was more common. rapid development kills all :D

IshKebab 3y ago

Because the main point of Docker is to allow you to run programs in their own OS. It became popular as a software distribution method for typical Linux software that is not self-contained - requires a ton of system dependencies and spews itself all over the filesystem. If your program doesn't do that - e.g. most Rust and Go programs which are a single statically linked binary - then there's very little reason to use Docker in the first place. People will say "but you still want to containerise things!" or "what about orchestration?" and sure those are some incidental benefits now, but they only really happened because everyone was more or less forced to use Docker to actually run apps. Docker is a workaround for shitty software.

chaxor 3y ago
"Docker is a workaround for shitty software" This just needs to be repeated for the pythonistas sitting in the back.
claytongulick 3y ago
Or, as another HN user put it a few years ago: "Docker is static linking for millennials" :D

mindwok 3y ago

This isn’t too far from how most containers are built nowadays. Usually you just use Alpine or slim Debian images. The benefit of using those distro images is that they provide a lot of utilities you might want in a container, like a shell. As an aside, the approach you described is how Nix works, and you can use Nix to produce OCI images.

iso1631 3y ago
> The benefit of using those distro images is that they provide a lot of utilities you might want in a container, like a shell. Why would you want a shell in a container? Surely the idea of a container is to run a program, not act as interactive environment.
- kroolik 3y ago
  Troubleshooting inside the container or simple scripting.
- rolisz 3y ago
  To debug stuff.
- leodag 3y ago
  It's not necessarily interactive, it's also pretty common to have a shell script for initialization/setup.
- the-alchemist 3y ago
  Like others have said, for debugging, although the proper-but-rarely-done solution is to use PID namespaces to attach another container containing the shell, rather than including it in the raw Docker image.
- giobox 3y ago
  "docker exec -it" debugging/troubleshooting. It's generally harder to debug/troubleshoot random issues if there is no shell in the container, and you can get a working shell in a 10mb container image so not like it adds some dramatic inconvenience or size. Sure its not 100 percent necessary, but it's enormously convenient at times and I personally wouldn't build my own container images without a shell I can drop into usually, unless there are technical or security reasons to avoid doing it for the given application. The shell can also be useful for certain scripting steps during image build, not all containers are just going to copy in a statically compiled binary etc. In fact, I'd say the latter is significantly rarer than the former, at least in my experience. Being able to do commands like: $ docker exec -it mycontainer sh is generally very helpful during development. > https://docs.docker.com/engine/reference/commandline/exec/ https://docs.docker.com/engine/reference/commandline/exec/

raesene9 3y ago

There's plenty of ways to build small container images that don't start with a "full" distribution, but whether they work for specific use cases will depend on what the applications need. FROM Scratch - starts with a totally blank image, this is the smallest option but your application must work with no supporting files (e.g. statically compiled binaries). Distroless - has a small number of standard OS support files but no package manager, so works where you don't need to install many OS packages. Wolfi - Newer than the others, they're building an ecosystem of minimal images for specific purposes.

RandomBK 3y ago
And let's not forget Alpine, which is a fairly common minimalist distro clocking in a ~5MB

takeda 3y ago

If you use Nix that's pretty much how the container will be build. You specify that your app should be there and Nix, being aware of all dependencies will place inside everything needed.

iopq 3y ago
Came to say this, in the case of sane things* Nix works perfectly * pip is not guaranteed to get you to a working state even if you run the same commands, so don't even think about it
aliasxneo 3y ago
> will place inside everything needed And more. In my experience, depending on the quality of the packages you happen to be consuming, you may end up with a container twice as large as a comparable (say, Alpine) container. For example, I once tried to bring git into my Nix container and was surprised to see over 300MB in increased size. After having built Nix containers exclusively for a year now, I wouldn't really recommend it to others unless you're willing to invest a lot of additional time cleaning up community packages.
- takeda 3y ago
  That's because by default Nix uses glibc (which is huge), while Alpine uses musl. You can create musl binaries, but you need to know Nix a little more and know how to cross compile. Another great thing is that you can can easily modify existing package to remove some of their dependencies that you don't use. This is shown here with packaging redis into a container here and making it 3x smaller than the alpine version: https://nixos.org/#asciinema-demo-cover https://nixos.org/#asciinema-demo-cover There's of course possibility that someone specified build dependency as a runtime dependency, but if that was done that's a bug.

nigma1337 3y ago

Lots of examples without the entire OS as other comments mention, an example would be Googles distroless[0] [0]: https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless

snickerer 3y ago

Because building a sane filesystem with only the needed libraries needs knowledge and some effort, whereas containers exist to keep people from using their brains. If you know what you are doing you have several alternatives: 1. Just link your binary statically. 2. Set the path to the picked out special libs as an environment variable only for the binary. 3. Re-think what's wrong with your system that leads to the pain of you wanting other lib versions for that binary.

tracker1 3y ago
1. Statically linking your application, then distributing that application, such as via Docker Hub to others will then require your application to be GPL licensed in many, many cases within the Linux ecosystem. This may not be desired. 2. You can absolutely limit the system libraries that are included to only those needed. Some platforms make this easier or harder than others. 3. That you have other software that you want to run in parallel and don't want to spin up full virtualization or manage various chroot structures to organize. I'd also add that just because a tool makes it easy to do something, and means you can bloat your file system doesn't make it inherently bad. Not everyone is trying to run a large database on a potato.

dlahoda 3y ago

nix does incremental file system. i use process-compose to build and manage process hierarchy.

tauroid 3y ago

sibling comments about Nix are not acknowledging that the base image size is around 600MB vs 5MB for alpine

adisbladis 3y ago
This is only true if use a Docker based workflow using `FROM nixos/nix`. This image exists mainly as a way for people to try out Nix with, not to build production images on top of. We ship many things which bloat the image size but makes it nicer for interactive usage. Using dockerTools from nixpkgs is much better and gives you much smaller images closer to Alpine size.
- tauroid 3y ago
  I might have confused download volume with image size but the tar.gz for dockerTools.buildLayeredImage with just node and mariadb in the contents is still 220MB (just checked) Edit: with nothing in the contents it's 144M, which is getting reasonable but still nearly 30x alpine base

pravus 3y ago

> Why is it still necessary to have whole, full-blown OS filesystems inside of our containers, if their purpose is running a single binary? It's not. This is just the path of least resistance for most container builds since many applications require various supporting files to function properly. Most of my Go containers only contain 2 files: the go binary and ca certificates. > Dependencies/dynamic libraries are decent reason, sure. But wouldn't it make more sense to do things "bottom-up"? In theory, yes. With dynamic loading and support for loading plugins and runtime this is virtually impossible to do. That's why starting from a base image that already includes everything you need is generally the chosen method. BSD jails/chroot operate this way and it's an absolute pain to setup for complicated applications.

asimpletune 3y ago

On a related note. I wonder whatever happened to unikernels and why they haven't become the norm.

nubinetwork 3y ago
https://en.m.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_debate https://en.m.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_d...
- reegnz 3y ago
  unikernel is not the same as microkernel. I've found these after some quick googling: https://unikraft.org/ https://unikraft.org/ https://hermitcore.org/ https://hermitcore.org/ https://nanos.org/ https://nanos.org/ Seems to be a living concept still, just not in the mainstream.
  - flaminHotSpeedo 3y ago
    There are mainstream use cases, particularly for cloud providers who want lightweight runtimes with hypervisor managed isolation. The only "example" I can think of off hand is Firecracker, but I'm not 100% confident that Firecracker is technically a unikernels. My guess is that there's a number of unikernel implementations behind closed doors that see heavy use

intelVISA 3y ago

The extent people will go to reinvent a worse, slower and more insecure ver of static bins + BSD jails is absurd...

db48x 3y ago

It’s not. You you can use systemd-nspawn to create a container that uses your own root filesystem by specifying --directory=/ --volatile=yes. This mounts a tmpfs into the container’s root, and then mounts your /usr into the container’s /usr in read–only mode. This allows the container to run all the software installed on your machine, while redirecting writes to the tmpfs. Alternatively, instead of --directory=/ you could specify some other directory that contains an OS image (such as --directory=/var/lib/machines/debian-bookworm or --directory=/var/lib/machines/fedora-38). Multiple containers can transparently share the same image, since all the writes go to a per–container tmpfs. https://0pointer.net/blog/running-an-container-off-the-host-usr.html https://0pointer.net/blog/running-an-container-off-the-host-...

jeroenhd 3y ago
That's an interesting way to use systemd-nspawn! But, if it mounts everything, wouldn't that also make container escapes very easy?
- db48x 3y ago
  When it is volatile, it really only mounts the /usr inside the specified directory, rather than the directory itself. In particular, /dev and /etc will be empty.

mschuster91 3y ago

> But wouldn't it make more sense to do things "bottom-up"? i.e. starting from an empty filesystem, and then progressively adding the files that are absolutely necessary for the binary to work Unless you're using Go or a C/C++ stack that does static linking and thus needs no dependencies including libc, that's yak shaving to an extreme degree.

adamgordonbell 3y ago

It's fun the play around with going that way. I did via chroot, and using ldd to see what a program like redis needs copied in[1]. But eventually you might want to get a shell into the little environment, and then page thru logs from inside it, and so eventually, unless you are going pure from-scratch, statically linked, you might want something like alpine in there. But its funny how as a solution to dynamic linking is shipping a whole OS. [1] https://www.youtube.com/watch?v=JOsWB50LmwQ https://www.youtube.com/watch?v=JOsWB50LmwQ

imiric 3y ago

> The fact that containers are such a simple technology But they're really not. This is a common misconception because of the user friendliness of Docker, but underneath all that there are many moving parts that the Linux kernel exposes, which are tricky to manage manually, and additional features of Docker itself (Dockerfile, layered images, distribution, etc.). Sure, you can write a shell script that does a tiny fraction of what container tools do, but then you'd have a half-baked solution that reinvents the wheel because... it's lighterweight? What TFA is doing is fine if you want to learn how containers work and impress your colleagues, but for real world usage, stick to the established tooling. Like others mentioned, there are ways to do what you propose, and you can always create your own container tool based on that shell script that automates this :). Though you might be interested in unikernels, which is an extreme version of that approach using VMs.

kodah 3y ago

Docker came about when lxc was the container interface on Linux. It's hardly user friendly. Doing everything yourself is possible, but again is really only user friendly for very simple operations. For instance, most users would probably find network name spacing or even uid/gid mapping pretty confusing. These days the images produced are quite different in that Docker now produces OCI compatible images, which makes them run anywhere. lxc on the other hand will produce images that run only on Linux. lxd eventually came around which is a lot more analogous to Docker as an application and outputs OCI compatible images.

ianlevesque 3y ago
I’m confused. If OCI images run anywhere, why does Docker Desktop for windows and mac run virtual machines with a Linux kernel behind the scenes?
- jeroenhd 3y ago
  Most container images contain binaries that were made to use Linux syscalls for things like I/O and memory management. Windows had an implementation of the Linux kernel interface for this exact purpose (WSL 1) but because of performance and compatibility challenges, they switched to plain virtual machines. These days, virtual machine run with almost 0 overhead anyway, so if you want to run a Linux binary they're a much simpler option than implementing a whole operating system.
  - tracker1 3y ago
    True enough... When I have been under Windows, a lot of things still run faster under WSL-Ubuntu than in Windows. Hyper-V is pretty nice, generally speaking, and the WSL tooling over it is pretty great. Makes having Windows as a host OS mostly tolerable. I used to joke that "WSL" was my favorite linux distro. I switched at home FT when I saw ads in my start menu search results that first time. The Edge nags weren't enough, the pre-installed games, etc. But an ad before what I was looking for on my local system, that was too far. I realize it was only a "test" feature, but even that someone wanted to test such a thing sickens me. I've still had to sometimes run under Windows at work. I prefer the M1/M2 macs now, just for silence and battery life, but they don't run Docker nearly as well, at least most of the x86_64 container issues are resolved, and most of what I touch has aarch64 targets.

goodpoint 3y ago

> Why is it still necessary to have whole, full-blown OS filesystems inside of our containers, if their purpose is running a single binary? It's not and Docker carries plenty of unnecessary complexity.

hardware2win 3y ago

Even better Get rid of those bloated OSes and run apps directly on HW/vHW

claytongulick 3y ago
That's what I do. I understand that there are some good use cases for containers, but I still haven't encountered a need for them. I run everything from a /home/app folder, with the app user and restricted permissions. I avoid projects that have container-only deployment (supabase, for example). Whenever I peek behind the curtain, I end up horrified at the complexity. Honestly, I just can't imagine why someone would want to run postgres, for example, in a container. Seems like a nightmare for maintenance and production support.
- t0astbread 3y ago
  I think they meant "run apps directly on hardware / virtual hardware". As in building the app with a unikernel, bypassing the need for a (large) operating system.

eldridgea 3y ago

It's a challenge but seems to be becoming moire prevalent for folks focused on supply chain attacks. Chainguard in particular has a bunch of standard containers (e.g. nginx) that reduce the container footprint as much as possible. https://www.chainguard.dev/chainguard-images https://www.chainguard.dev/chainguard-images

m463 3y ago

It's for portability. Your alpine binary can run in an alpine container. But folks run that alpine container on ubuntu 18.04 or debian 11 or macos or wsl2. also, docker brings a lot of usefulness to this mix - dockerfile "recipe" that builds on other recipes, layered filesystem sort of like version control, global namespace, etc