7 ms·

I think containers are often misunderstood: The main benefit is not isolation and security, it's defined and reproducible environments and builds. If there is

by andix 2y ago

I think containers are often misunderstood: The main benefit is not isolation and security, it's defined and reproducible environments and builds. If there is some additional isolation required, just run the container in a VM.

brabel 2y ago
But if you can get isolation, security AND reproducible environments using a VM, specially one that's nearly as fast as a OS process, the case for using containers instead pretty much disappears. I don't know this LiteVM thing but I will definitely investigate that, specially given that on my Mac I need to use a VM anyway to run containers!
- hmottestad 2y ago
  I wouldn't discount the massive user base of containers among developers.
- alickz 2y ago
  i think at that point it's kind of just semantics i don't think devs care if they use containers or VMs, as long as it's easy and they don't have to worry about which version of Python the host is running
  - jackcviers3 2y ago
    This. It's why vagrant was popular before the container revolution. The killer app of Docker isn't the container, it's the depth and uniformity of the UX surrounding the container system. When that is broken by something on the host (non x86 cpu was a major pain for a while before popular images were x-built) and emulation gets in the way and is not as easy, or just mildly different (windows behind corporate firewalls that assign ips used by the docker engine for example), the ease of use falls away for non-power users and it's all painful again. Tech like Docker for windows and Rancher Desktop and lima has largely matured at this point, but somebody could make a new machine and then the process of gradual improvement starts all over again.
- tpetry 2y ago
  But then also monitoring gets harder. With containers you could see (and monitor) all those processes running in container easy on the host. With VMs? You now need a way to get data from within the VM which in most lightweight VM implementations just not possible.
  - fhuici 2y ago
    On kraft.cloud we have unikernels (specialized VMs) with Prometheus exporters that can be scraped, and other monitoring facilities.
- Timshel 2y ago
  At the time of publication of the article the tool used to create the minimalistic VM Tinyx was not released and as far as I can see was never released.
  - fhuici 2y ago
    Correct, we never did release Tinyx, mostly because it was in a very unclean/researchy state = not ready for public consumption. In retrospect, we probably should have either (a) made it available in whatever state it was in or (b) put more cycles into it.
- fhuici 2y ago
  Check out kraft.cloud and the accompanying LF OSS project www.unikraft.org :) (disclaimer: I'm one of the authors of the paper and one of the people behind that cloud offering). On KraftCloud we use Dockerfiles so users can conveniently specify the app/filesystem, and then at deploy time transparently convert that to a unikernel (specialized VMs). With this in place, NGINX cold starts in 20 millis, and even heavier apps/frameworks like Spring Boot in < 300 millis (and we have a number of tech to bring these numbers even further down).
  - internetter 2y ago
    Is the entire kraft stack open source? Can I run this on my own hardware?
    - Melatonic 2y ago
      Also was my question.
  - brabel 2y ago
    Looks very cool. For anyone else wondering how heavy this is on a MacOS, I ran the install script and it just delegated to brew... brew listed the following packages being installed: ==> Fetching dependencies for unikraft/cli/kraftkit: aarch64-elf-binutils, gmp, mpfr, aarch64-elf-gcc, coreutils, gettext, readline, gawk, gnu-sed, pcre2, grep, make, capstone, dtc, mpdecimal, ca-certificates, openssl@3, sqlite, python@3.12, glib, libunistring, libidn2, p11-kit, libnghttp2, unbound, gnutls, jpeg-turbo, libslirp, libssh, libusb, ncurses, snappy, vde, qemu, socat, wget, x86_64-elf-binutils and x86_64-elf-gcc Most should already exist on your mac if you do development... it seems to rely on qemu, unsurprisingly... openjdk as well (probably to support Java out-of-the-box?), imagegick etc. Took a few minutes to finish installing... the CLI seems to be based on the Docker commands (build, clean, run, 'net create', inspect etc.), some package-manager like commands ('pkg info', 'pkg pull', 'pkg list' etc.), a bunch of "cloud" commands (I suppose that's the non-free part) and "compose" commands just like docker-compose. Interesting stuff. Note for the parent commenter: the Lua link in the landing page is broken: https://github.com/unikraft/catalog/tree/main/examples/http-lua5.4 https://github.com/unikraft/catalog/tree/main/examples/http-... I tried to run the C hello world example... I get an error, it wants to run Docker?!?! I thought the whole point was to avoid Docker (and containers)?? Here's the log: i creating ephemeral buildkit container W could not connect to BuildKit client '' is BuildKit running? W W By default, KraftKit will look for a native install which W is located at /run/buildkit/buildkit.sock. Alternatively, you W can run BuildKit in a container (recommended for macOS users) W which you can do by running: W W docker run --rm -d --name buildkit --privileged moby/buildkit:latest W export KRAFTKIT_BUILDKIT_HOST=docker-container://buildkit W W For more usage instructions visit: https://unikraft.org/buildkit https://unikraft.org/buildkit W E creating buildkit container: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?: failed to create container PS. running the hello-world pre-built "image" worked: > kraft run unikraft.org/helloworld:latest EDIT: A lot of stuff looks broken on MacOS. For example, `kraft menu` doesn't work (error "no choices provided", even though the docs show it working fine without "choices"?)... `kraft run --elfloader loaders.unikraft.org/strace:latest ./my_binary` also doesn't work (the docs show it working). Error: "unknown flag: --elfloader". Seems like the product is still in alpha?!
- bluGill 2y ago
  VMs in general use more CPU power as you have two OSes each doing things like updating their real time clock... There are VM aware OSes that will not do this, but it needs special code and CPU support which means you are often lagging behind the latest (to be fair this is rarely important) A container will normally be slightly faster than a VM never slower (assuming a reasonable OS - I can write an exception if I was malicious) and so there is a lot of interest if they are good enough.
- byteknight 2y ago
  No it didn't. I want to rebuild my image using a different base. Docker? One line possibly. VM? Afternoon (unless I want to write ansible too)
  - pwagland 2y ago
    In this sort of scenario, then you can also use something like cloud-init: https://cloudinit.readthedocs.io/en/latest/ https://cloudinit.readthedocs.io/en/latest/
    - byteknight 2y ago
      Or I could use the tool I currently am. Your reply doesn't solve my use case it just says use yet another tool.
- bartekrutkowski 2y ago
  You don't need a VM on Mac to run containers, check out OrbStack, they provide a Docker compatible engine that is using native MacOS capabilities for running containers without the hidden Linux VM.
  - pxc 2y ago
    I don't know where you got that idea. OrbStack absolutely runs a Linux VM. That Linux VM then uses Linux containerization technologies (namely LXD) for each separate OrbStack 'machine' you set up, which is how you get such fast startup times for your OrbStack 'machines'. For Docker, OrbStack does the same thing as Docker Desktop, Podman Desktop, Rancher Desktop, etc., which is set up a Linux VM running Docker and then present a native socket interface on macOS which relays everything it receives to the Docker socket inside the VM. macOS doesn't have native capabilities for running containers, which is why the nearest thing you can get to containerd on it requires you to disable SIP so it can use a custom filesystem to emulate bind mounts/null mounts: https://darwin-containers.github.io/ https://darwin-containers.github.io/ If you read the PRs where the principal author of the Darwin Containers implementation is trying to upstream bits of his work, you'll see containerd comparing his approaches to others and complimenting them by calling them 'the most containerish' because real capabilities aren't there. (I believe I've read rumors here on HN that Apple has those features internally, fwiw. But they've evidently never released them in a public copy of macOS.) Another clue in all this is to just run uname in any of your Docker containers in OrbStack; you'll see they're Linux machines. Some operating systems have Linux syscall emulation layers (WSL1, FreeBSD's Linux emulation, Illumos' LX Zones) that could perhaps be used to run Linux containers without hardware emulation or paravirtualization in combination with some native containerization capabilities. Afaik Illumos' LX Zones is the only implementation where that's a supported, intended use case but maybe FreeBSD can do it. At any rate, macOS has never had that kind of syscall compatibility layer for Linux, either. So when you run `uname` in a 'macOS container' and see 'Linux', you can be certain that there's a VM in that stack. PS: Aside from the fact that it's proprietary, I really do quite like OrbStack. It's the nicest-to-use implementation of something like this that I've tried, including WSL2 and Lima. The fact that it makes the VM machinery so invisible is very much to its credit from a UX perspective!
- usrusr 2y ago
  Certainly depends a lot on what the term "VM" actually means in the context. If it's something as specialized as the JVM, or a native virtualization with an extremely trimmed down guest, then at some point you'll find yourself in need of something more heterogenous, e.g. running a tool on the side that does not fit the VM. Then you're back at square one, only this time with containers (or back in some irreproducible ad-hoc setup). Going with containers from the start, containers that may or may not contain a VM, and that may or may not actually do more than what a VM could supply, that's much less hassle than changing horses at a later point.
- andix 2y ago
  Where is the tooling to build and distribute lightweight vms like containers? How can I copy one html file into an nginx VM, built this vm image with multiple architectures (I have arm and x64 servers), publish it, pull it, and run it multiple times? Once again: Containers are not about isolation or security, they are a package format for shipping applications. The packages are easy to build, distribute, multiarch, ... And requiring a Linux-VM on macOS to run Linux containers, is not particularly surprising.
- beeboobaa3 2y ago
  > specially given that on my Mac I need to use a VM anyway to run containers Take it up with Apple.
- ravenstine 2y ago
  I agree in principle, but this is not so easy when shared disk IO is introduced, in my experience.
dweekly 2y ago
If this were true, then wouldn't folks just need an application binary that statically links all of its required libraries and resources into a giant, say, ELF? Why even bother with a container?
- dailykoder 2y ago
  Because static libraries ain't a big thing anymore. Maybe they will become popular again. This would make it easier to have reproduceable build without a container. But I think containers are the new static libs now
  - indymike 2y ago
    Interpreted languages and their associated dependencies are more of an issue than static linking with compiled languages.
- shawabawa3 2y ago
  people would absolutely love to do that, but it's difficult, and the UX sucks so people use containers instead (even if the container literally only contains a single statically linked binary)
- Cthulhu_ 2y ago
  That's exactly the summary of this. That said, it makes sense from a developer POV; if, during development, you don't need the isolation you can run multiple containers (with on paper fast boot times and minimal overhead) on your development box. There's plenty of cases to imagine where you need the containerization but not necessarily the isolation.
  - jbverschoor 2y ago
    That what I do. I use https://github.com/jrz/container-shell https://github.com/jrz/container-shell so my projects are contained, and I'm a little bit protected against supply chain attacks
nailer 2y ago
> If there is some additional isolation required, just run the container in a VM. No. Running a container in a VM gets you no additional isolation. Containers share kernel space and as such have limited isolation to VMs, which have isolated kennels. In exchange for this Lack of additional isolation, you’ve added a Bunch of extra Complexity. Pardon the extra caps I am using iOS voice dictation.
- kevincox 2y ago
  I think they mean run a VM with one container inside. So you do get strong isolation. This is similar to how managed container IaaS works. They launch a VM and run your container in it. It is extra complexity but has a few advantages. 1. People already have a convenient workflow for building container images. 2. The base OS can manage hardware, networking and whatever other low-level needs so that the container doesn't need to have these configurations. 3. If you want to trade of isolation for efficiency you can do this. For example running two instances of a container in the same VM. The container doesn't need any changes to support this setup.
  - fhuici 2y ago
    The model of a single container within a VM just adds overhead. The ideal case would be to remove the container layer and have the application(s) within the container run directly in the VM (which hopefully only includes the libs and OS modules needed for the app to run, and nothing more). This is the approach we take at kraft.cloud (based on the LF Unikraft project): use Dockerfiles to specify app/filesystem, and at deploy automatically convert to a lightweight VM (unikernel) without the container runtime/layer.
  - nailer 2y ago
    > So you do get strong isolation. No, you don’t. There is no benefit the container is providing, because The only feature of the container is isolating you from the zero other containers running on the VM.
    - kevincox 2y ago
nderjung 2y ago
Containers are perfect for build environments and for creating the root filesystem. The issue is that the kernel these days are super bulky and are intended for multi-user, multi-process environments. Running a container runtime on top just makes it worse when you're looking for "isolation". This paper argues that when you build a extremely minimal kernel (i.e. ditch Linux entirely) and link your application against necessary bits of code to execute _as_ a VM, then you'll get better performance than a container and you'll get that isolation. This is in fact true based on performance studies, the follow up paper to this shows so: https://arxiv.org/pdf/2104.12721 https://arxiv.org/pdf/2104.12721 (Disclosure, co-author of the linked paper.) We ended up taking this to real workloads if you want to see it in action: https://unikraft.io/ https://unikraft.io/
- melenaboija 2y ago
  > Running a container runtime on top just makes it worse when you're looking for "isolation". The point of the poster was pretty clear: “The main benefit is not isolation and security”
- posix_monad 2y ago
  > build a extremely minimal kernel (i.e. ditch Linux entirely) and link your application against necessary bits of code It would be nice, but this is really hard to do when modern software has so many layers of crud. Good luck getting say, a PyTorch app, to work doing this without some serious time investment.
  - bluGill 2y ago
    Most of that effort should be sharable. if you know you will only have one python process you can get rid of a lot of cruft. If you know you will be running in a VM then you only need the driver for the network interface the VM provides not every network interface every designed (often including ones that your hardware doesn't even physically support). So while there is serious time investment it isn't nearly as much as it would be to write a competitor to linux.
    - 2y ago
fhuici 2y ago
[disclaimer: I'm one of the authors of the paper] I 100% agree, containers are an amazing dev env/reprodicble env tool! In fact, we think they're the perfect marriage to the unikernels (specialized VMs) we used in the paper; on kraft.cloud , a cloud platform we built, we use Dockerfiles to specify apps/filesystems, and transparently convert them to unikernels for deployment. The end result is the convenience of containers with the power of unikernels (eg, millisecond cold starts, scale to zero and autoscale, reduced TCB, etc).
andrewpolidori 2y ago
While reproducible build envs are a nice feature of using containers, they aren't the primary benefit. The primary benefit is resource usage and orchestration. Rather than duplicating entire aspects of an OS stack (which might ne considered wasteful) they allow for workloads to share aspects of the system they run on while maintaining a kind of logical isolation. This allows for more densely packed workloads and more effective use of resources. This is a reason why the tech was developed and pushed by google and adopted by hyperscalers.
benreesman 2y ago
Containers have their place, if you’re racking and running your own gear life gets a lot better than screwing around with IPMI. But reproducible infrastructure as code is just orthogonal to that: everything from Salt to Nix is credible in that role. Containerizing on top of a Xen hypervisor never made sense to me.
- oceanplexian 2y ago
  All the younger engineers I talk to think you would need to be Albert Einstein to bootstrap a bare metal server. As someone who made a living doing this at scale, where we would build a new datacenter every 2-4 weeks using 100% open source or off the shelf tools, I completely disagree. I think PXE booting some servers and running a binary on them is 90% easier than most container orchestration engines, Kubernetes control plane, and all the other problems engineers seem to have invented for themselves. I also think it’s almost always much more performant. Engineers don’t have an intuition to realize that their XXLarge-SuperDuper instance is actually a 5 year old Xeon they’re sharing with 4 other customers. Cloud Prociders obfuscate this as much as possible, and charge a King’s ransom if you want modern, dedicated hardware.
  - benreesman 2y ago
    I’ll urge that we don’t compare Albert Einstein to a k9s expert. No one will ever be considered for a Nobel over Kubernetes.
kkfx 2y ago
NixOS and Guix System offer a far lighter and more reproducible approach, who also not push many running images build by unknown on the internet direct in production, full of outdated deps, wasting in the meantime storage and cpu resources...
- oneplane 2y ago
  Yet it doesn't even come close to a fraction of the adoption scale of containers, no matter how good it is. Ecosystems matter more than individual quality.
  - kkfx 2y ago
    That's because some interested parties have advertised containers, because they are good to sell as pre-built stuff, nice to sell VPS and alike etc, while pure IaC is useful for anyone and invite NOT to be dependent on third party platforms. It's not a technical matter, it's a human, economical matter and actually... Most people are poor, following the largest scale means following poverty not a good thing.
  - bigstrat2003 2y ago
    I think that's a gross overstatement. Ecosystems matter, yes. But they don't matter more than quality.
    - oneplane 2y ago
      I think Betamax might have something to say about that.
bluGill 2y ago
There is no such thing as a reproducible build environment anymore. You can get a temporary reproducible build environment, but any sane security policy will have certificates that expire and that in turn means that in a couple years your build environment won't be reproducible anymore.
- andix 2y ago
  If you run multiple instances of a container image, you get a reproducible environment. If you run a docker build multiple times, and copy a few files into the container, you get a reproducible container image. It is not a hash perfect duplicate, but functionally equivalent. If builds of your favourite programming language are reproducible or not, is not really related to VM vs. Container.
- cesarb 2y ago
  > but any sane security policy will have certificates that expire and that in turn means that in a couple years your build environment won't be reproducible anymore. "Reproducible" is usually defined as "identical output except for the cryptographic signature at the end" (and that should be the only use for a certificate in your build environment, a high-quality build environment should be self-contained and have no network access). That is, once you remove the signature, the built artifacts should be bit-by-bit identical.
  - bluGill 2y ago
    I said environment not build.
belter 2y ago
You are absolutely correct, and the creators of Docker did mention that was the core reason. Unfortunately your comment comes 10 years too late for many.
nimbius 2y ago
we arguably already had this with things like python venv. the articles main point still remains, containers are a slow and bloated answer to this problem. I concede youll need containers for Kubernetes, and Kubernetes on the surface is a very good idea, but this level of infrastructure automation exists already in things like foreman and openstack. designs like shift-on-stack trade simplicity of traditional hardware for ever byzantine levels of brittle versioned complexity...so ultimately instead of fixing the problem we invoke the god of immutability, destroy and rebuild, and hope the problem fixes itself somehow...its really quite comical. baremetal rust/python/go with good architecture and CI will absolutely crush container workloads in a fraction of disk, CPU, RAM, and personal frustration.
- jchw 2y ago
  Python venv is language specific, doesn't handle the interpreter version and doesn't handle C libraries. I really don't understand why people do this: I get having a distaste for containers but some people, seeing the massive success of OCI images, mainly seem content on trying to figure out how to discredit its popularity, rather than trying to understand why it's popular. The former may be good for contrarian Internet forums, but the latter is more practically useful and interesting. I say this with some level of understanding as I also have a distaste for containers and Docker is not my preferred way to do "hermetic" or "reproducible" (I am a huge Nix proponent.) I want to get past the "actually it was clearly useless from the start" because it wasn't...
- jbverschoor 2y ago
  Not really, you'd still need a proper chroot / etc. Check out https://github.com/jrz/container-shell https://github.com/jrz/container-shell
cduzz 2y ago
I tell people "An OCI container is a way to turn any random runtime into a statically linked binary." It is very useful for managing dependency hell, or at least moving it into "API dependencies" not "Library dependencies", it is handy for pickling a CI/CD release engineering infrastructure. It's not a security boundary. (I'm 100% agreeing with parent, in case I sound contentious)
- scarby2 2y ago
  > It's not a security boundary. It is a security boundary, just not necessarily the best one.
  - andix 2y ago
    I would even claim it's a pretty good security boundary, good enough for most applications.
  - bayindirh 2y ago
    It's an incidental security boundary because CGroups happen to isolate the process fairly well.
    - vilunov 2y ago
      Yeah, but that's not an incidental property of *namespaces* (of which cgroups is only one isolation axis), that was the requirement when namespaces were designed.
      - bayindirh 2y ago
        Yeah, I know. Namespaces are pretty cool outside containers too. My comment was more of a soft jab against using containers as the ultimate "thing" for anything and everything. I prefer to use them as "statically linked binaries" for short lived processes (like document building, etc.). But, whenever someone abuses containers (like adding an HTTPs fronting container in front of anything which can handle HTTPS on its own) I'm displeased. Relevant XKCD: https://xkcd.com/1988/ https://xkcd.com/1988/
BobbyTables2 2y ago
I agree with you, but the world seems to think otherwise!
flanked-evergl 2y ago
Here are some projects that run docker containers on top of micro/lightweight VMs: - https://github.com/firecracker-microvm/firecracker-containerd https://github.com/firecracker-microvm/firecracker-container... - https://github.com/kuasar-io/kuasar https://github.com/kuasar-io/kuasar - https://github.com/kata-containers/kata-containers https://github.com/kata-containers/kata-containers - https://github.com/QuarkContainer/Quark https://github.com/QuarkContainer/Quark - https://github.com/google/gvisor https://github.com/google/gvisor - https://github.com/containers/libkrun https://github.com/containers/libkrun
lttlrck 2y ago
The main advantage in my use case is in fact isolation (network and volumes) and a well defined API enabling management of those containers in production (not k8s, a tiny subnet of that perhaps). The isolation could be achieved using namespaces directly. But the API, tooling and registry add a lot of value that would otherwise require a lot of development. Also last time I looked hypervisors aren't possible on all cloud vendors, unless you have a bare metal server. This matters in my case. Maybe it has changed in the past 3 years. When docker fits it's great. Same can be said of k8s, where there are a whole bunch of additional benefits. Swings and roundabouts.
deleted 2y ago
[deleted]

by andix 2y ago

brabel 2y ago

But if you can get isolation, security AND reproducible environments using a VM, specially one that's nearly as fast as a OS process, the case for using containers instead pretty much disappears. I don't know this LiteVM thing but I will definitely investigate that, specially given that on my Mac I need to use a VM anyway to run containers!

hmottestad 2y ago
I wouldn't discount the massive user base of containers among developers.
alickz 2y ago
i think at that point it's kind of just semantics i don't think devs care if they use containers or VMs, as long as it's easy and they don't have to worry about which version of Python the host is running
- jackcviers3 2y ago
  This. It's why vagrant was popular before the container revolution. The killer app of Docker isn't the container, it's the depth and uniformity of the UX surrounding the container system. When that is broken by something on the host (non x86 cpu was a major pain for a while before popular images were x-built) and emulation gets in the way and is not as easy, or just mildly different (windows behind corporate firewalls that assign ips used by the docker engine for example), the ease of use falls away for non-power users and it's all painful again. Tech like Docker for windows and Rancher Desktop and lima has largely matured at this point, but somebody could make a new machine and then the process of gradual improvement starts all over again.
tpetry 2y ago
But then also monitoring gets harder. With containers you could see (and monitor) all those processes running in container easy on the host. With VMs? You now need a way to get data from within the VM which in most lightweight VM implementations just not possible.
- fhuici 2y ago
  On kraft.cloud we have unikernels (specialized VMs) with Prometheus exporters that can be scraped, and other monitoring facilities.
Timshel 2y ago
At the time of publication of the article the tool used to create the minimalistic VM Tinyx was not released and as far as I can see was never released.
- fhuici 2y ago
  Correct, we never did release Tinyx, mostly because it was in a very unclean/researchy state = not ready for public consumption. In retrospect, we probably should have either (a) made it available in whatever state it was in or (b) put more cycles into it.
fhuici 2y ago
Check out kraft.cloud and the accompanying LF OSS project www.unikraft.org :) (disclaimer: I'm one of the authors of the paper and one of the people behind that cloud offering). On KraftCloud we use Dockerfiles so users can conveniently specify the app/filesystem, and then at deploy time transparently convert that to a unikernel (specialized VMs). With this in place, NGINX cold starts in 20 millis, and even heavier apps/frameworks like Spring Boot in < 300 millis (and we have a number of tech to bring these numbers even further down).
- internetter 2y ago
  Is the entire kraft stack open source? Can I run this on my own hardware?
  - Melatonic 2y ago
    Also was my question.
- brabel 2y ago
  Looks very cool. For anyone else wondering how heavy this is on a MacOS, I ran the install script and it just delegated to brew... brew listed the following packages being installed: ==> Fetching dependencies for unikraft/cli/kraftkit: aarch64-elf-binutils, gmp, mpfr, aarch64-elf-gcc, coreutils, gettext, readline, gawk, gnu-sed, pcre2, grep, make, capstone, dtc, mpdecimal, ca-certificates, openssl@3, sqlite, python@3.12, glib, libunistring, libidn2, p11-kit, libnghttp2, unbound, gnutls, jpeg-turbo, libslirp, libssh, libusb, ncurses, snappy, vde, qemu, socat, wget, x86_64-elf-binutils and x86_64-elf-gcc Most should already exist on your mac if you do development... it seems to rely on qemu, unsurprisingly... openjdk as well (probably to support Java out-of-the-box?), imagegick etc. Took a few minutes to finish installing... the CLI seems to be based on the Docker commands (build, clean, run, 'net create', inspect etc.), some package-manager like commands ('pkg info', 'pkg pull', 'pkg list' etc.), a bunch of "cloud" commands (I suppose that's the non-free part) and "compose" commands just like docker-compose. Interesting stuff. Note for the parent commenter: the Lua link in the landing page is broken: https://github.com/unikraft/catalog/tree/main/examples/http-lua5.4 https://github.com/unikraft/catalog/tree/main/examples/http-... I tried to run the C hello world example... I get an error, it wants to run Docker?!?! I thought the whole point was to avoid Docker (and containers)?? Here's the log: i creating ephemeral buildkit container W could not connect to BuildKit client '' is BuildKit running? W W By default, KraftKit will look for a native install which W is located at /run/buildkit/buildkit.sock. Alternatively, you W can run BuildKit in a container (recommended for macOS users) W which you can do by running: W W docker run --rm -d --name buildkit --privileged moby/buildkit:latest W export KRAFTKIT_BUILDKIT_HOST=docker-container://buildkit W W For more usage instructions visit: https://unikraft.org/buildkit https://unikraft.org/buildkit W E creating buildkit container: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?: failed to create container PS. running the hello-world pre-built "image" worked: > kraft run unikraft.org/helloworld:latest EDIT: A lot of stuff looks broken on MacOS. For example, `kraft menu` doesn't work (error "no choices provided", even though the docs show it working fine without "choices"?)... `kraft run --elfloader loaders.unikraft.org/strace:latest ./my_binary` also doesn't work (the docs show it working). Error: "unknown flag: --elfloader". Seems like the product is still in alpha?!
bluGill 2y ago
VMs in general use more CPU power as you have two OSes each doing things like updating their real time clock... There are VM aware OSes that will not do this, but it needs special code and CPU support which means you are often lagging behind the latest (to be fair this is rarely important) A container will normally be slightly faster than a VM never slower (assuming a reasonable OS - I can write an exception if I was malicious) and so there is a lot of interest if they are good enough.
byteknight 2y ago
No it didn't. I want to rebuild my image using a different base. Docker? One line possibly. VM? Afternoon (unless I want to write ansible too)
- pwagland 2y ago
  In this sort of scenario, then you can also use something like cloud-init: https://cloudinit.readthedocs.io/en/latest/ https://cloudinit.readthedocs.io/en/latest/
  - byteknight 2y ago
    Or I could use the tool I currently am. Your reply doesn't solve my use case it just says use yet another tool.
bartekrutkowski 2y ago
You don't need a VM on Mac to run containers, check out OrbStack, they provide a Docker compatible engine that is using native MacOS capabilities for running containers without the hidden Linux VM.
- pxc 2y ago
  I don't know where you got that idea. OrbStack absolutely runs a Linux VM. That Linux VM then uses Linux containerization technologies (namely LXD) for each separate OrbStack 'machine' you set up, which is how you get such fast startup times for your OrbStack 'machines'. For Docker, OrbStack does the same thing as Docker Desktop, Podman Desktop, Rancher Desktop, etc., which is set up a Linux VM running Docker and then present a native socket interface on macOS which relays everything it receives to the Docker socket inside the VM. macOS doesn't have native capabilities for running containers, which is why the nearest thing you can get to containerd on it requires you to disable SIP so it can use a custom filesystem to emulate bind mounts/null mounts: https://darwin-containers.github.io/ https://darwin-containers.github.io/ If you read the PRs where the principal author of the Darwin Containers implementation is trying to upstream bits of his work, you'll see containerd comparing his approaches to others and complimenting them by calling them 'the most containerish' because real capabilities aren't there. (I believe I've read rumors here on HN that Apple has those features internally, fwiw. But they've evidently never released them in a public copy of macOS.) Another clue in all this is to just run uname in any of your Docker containers in OrbStack; you'll see they're Linux machines. Some operating systems have Linux syscall emulation layers (WSL1, FreeBSD's Linux emulation, Illumos' LX Zones) that could perhaps be used to run Linux containers without hardware emulation or paravirtualization in combination with some native containerization capabilities. Afaik Illumos' LX Zones is the only implementation where that's a supported, intended use case but maybe FreeBSD can do it. At any rate, macOS has never had that kind of syscall compatibility layer for Linux, either. So when you run `uname` in a 'macOS container' and see 'Linux', you can be certain that there's a VM in that stack. PS: Aside from the fact that it's proprietary, I really do quite like OrbStack. It's the nicest-to-use implementation of something like this that I've tried, including WSL2 and Lima. The fact that it makes the VM machinery so invisible is very much to its credit from a UX perspective!
usrusr 2y ago
Certainly depends a lot on what the term "VM" actually means in the context. If it's something as specialized as the JVM, or a native virtualization with an extremely trimmed down guest, then at some point you'll find yourself in need of something more heterogenous, e.g. running a tool on the side that does not fit the VM. Then you're back at square one, only this time with containers (or back in some irreproducible ad-hoc setup). Going with containers from the start, containers that may or may not contain a VM, and that may or may not actually do more than what a VM could supply, that's much less hassle than changing horses at a later point.
andix 2y ago
Where is the tooling to build and distribute lightweight vms like containers? How can I copy one html file into an nginx VM, built this vm image with multiple architectures (I have arm and x64 servers), publish it, pull it, and run it multiple times? Once again: Containers are not about isolation or security, they are a package format for shipping applications. The packages are easy to build, distribute, multiarch, ... And requiring a Linux-VM on macOS to run Linux containers, is not particularly surprising.
beeboobaa3 2y ago
> specially given that on my Mac I need to use a VM anyway to run containers Take it up with Apple.
ravenstine 2y ago
I agree in principle, but this is not so easy when shared disk IO is introduced, in my experience.

dweekly 2y ago

If this were true, then wouldn't folks just need an application binary that statically links all of its required libraries and resources into a giant, say, ELF? Why even bother with a container?

dailykoder 2y ago
Because static libraries ain't a big thing anymore. Maybe they will become popular again. This would make it easier to have reproduceable build without a container. But I think containers are the new static libs now
- indymike 2y ago
  Interpreted languages and their associated dependencies are more of an issue than static linking with compiled languages.
shawabawa3 2y ago
people would absolutely love to do that, but it's difficult, and the UX sucks so people use containers instead (even if the container literally only contains a single statically linked binary)
Cthulhu_ 2y ago
That's exactly the summary of this. That said, it makes sense from a developer POV; if, during development, you don't need the isolation you can run multiple containers (with on paper fast boot times and minimal overhead) on your development box. There's plenty of cases to imagine where you need the containerization but not necessarily the isolation.
- jbverschoor 2y ago
  That what I do. I use https://github.com/jrz/container-shell https://github.com/jrz/container-shell so my projects are contained, and I'm a little bit protected against supply chain attacks

nailer 2y ago

> If there is some additional isolation required, just run the container in a VM. No. Running a container in a VM gets you no additional isolation. Containers share kernel space and as such have limited isolation to VMs, which have isolated kennels. In exchange for this Lack of additional isolation, you’ve added a Bunch of extra Complexity. Pardon the extra caps I am using iOS voice dictation.

kevincox 2y ago
I think they mean run a VM with one container inside. So you do get strong isolation. This is similar to how managed container IaaS works. They launch a VM and run your container in it. It is extra complexity but has a few advantages. 1. People already have a convenient workflow for building container images. 2. The base OS can manage hardware, networking and whatever other low-level needs so that the container doesn't need to have these configurations. 3. If you want to trade of isolation for efficiency you can do this. For example running two instances of a container in the same VM. The container doesn't need any changes to support this setup.
- fhuici 2y ago
  The model of a single container within a VM just adds overhead. The ideal case would be to remove the container layer and have the application(s) within the container run directly in the VM (which hopefully only includes the libs and OS modules needed for the app to run, and nothing more). This is the approach we take at kraft.cloud (based on the LF Unikraft project): use Dockerfiles to specify app/filesystem, and at deploy automatically convert to a lightweight VM (unikernel) without the container runtime/layer.
- nailer 2y ago
  > So you do get strong isolation. No, you don’t. There is no benefit the container is providing, because The only feature of the container is isolating you from the zero other containers running on the VM.
  - kevincox 2y ago

nderjung 2y ago

Containers are perfect for build environments and for creating the root filesystem. The issue is that the kernel these days are super bulky and are intended for multi-user, multi-process environments. Running a container runtime on top just makes it worse when you're looking for "isolation". This paper argues that when you build a extremely minimal kernel (i.e. ditch Linux entirely) and link your application against necessary bits of code to execute _as_ a VM, then you'll get better performance than a container and you'll get that isolation. This is in fact true based on performance studies, the follow up paper to this shows so: https://arxiv.org/pdf/2104.12721 https://arxiv.org/pdf/2104.12721 (Disclosure, co-author of the linked paper.) We ended up taking this to real workloads if you want to see it in action: https://unikraft.io/ https://unikraft.io/

melenaboija 2y ago
> Running a container runtime on top just makes it worse when you're looking for "isolation". The point of the poster was pretty clear: “The main benefit is not isolation and security”
posix_monad 2y ago
> build a extremely minimal kernel (i.e. ditch Linux entirely) and link your application against necessary bits of code It would be nice, but this is really hard to do when modern software has so many layers of crud. Good luck getting say, a PyTorch app, to work doing this without some serious time investment.
- bluGill 2y ago
  Most of that effort should be sharable. if you know you will only have one python process you can get rid of a lot of cruft. If you know you will be running in a VM then you only need the driver for the network interface the VM provides not every network interface every designed (often including ones that your hardware doesn't even physically support). So while there is serious time investment it isn't nearly as much as it would be to write a competitor to linux.
  - 2y ago

fhuici 2y ago

[disclaimer: I'm one of the authors of the paper] I 100% agree, containers are an amazing dev env/reprodicble env tool! In fact, we think they're the perfect marriage to the unikernels (specialized VMs) we used in the paper; on kraft.cloud , a cloud platform we built, we use Dockerfiles to specify apps/filesystems, and transparently convert them to unikernels for deployment. The end result is the convenience of containers with the power of unikernels (eg, millisecond cold starts, scale to zero and autoscale, reduced TCB, etc).

andrewpolidori 2y ago

While reproducible build envs are a nice feature of using containers, they aren't the primary benefit. The primary benefit is resource usage and orchestration. Rather than duplicating entire aspects of an OS stack (which might ne considered wasteful) they allow for workloads to share aspects of the system they run on while maintaining a kind of logical isolation. This allows for more densely packed workloads and more effective use of resources. This is a reason why the tech was developed and pushed by google and adopted by hyperscalers.

benreesman 2y ago

Containers have their place, if you’re racking and running your own gear life gets a lot better than screwing around with IPMI. But reproducible infrastructure as code is just orthogonal to that: everything from Salt to Nix is credible in that role. Containerizing on top of a Xen hypervisor never made sense to me.

oceanplexian 2y ago
All the younger engineers I talk to think you would need to be Albert Einstein to bootstrap a bare metal server. As someone who made a living doing this at scale, where we would build a new datacenter every 2-4 weeks using 100% open source or off the shelf tools, I completely disagree. I think PXE booting some servers and running a binary on them is 90% easier than most container orchestration engines, Kubernetes control plane, and all the other problems engineers seem to have invented for themselves. I also think it’s almost always much more performant. Engineers don’t have an intuition to realize that their XXLarge-SuperDuper instance is actually a 5 year old Xeon they’re sharing with 4 other customers. Cloud Prociders obfuscate this as much as possible, and charge a King’s ransom if you want modern, dedicated hardware.
- benreesman 2y ago
  I’ll urge that we don’t compare Albert Einstein to a k9s expert. No one will ever be considered for a Nobel over Kubernetes.

kkfx 2y ago

NixOS and Guix System offer a far lighter and more reproducible approach, who also not push many running images build by unknown on the internet direct in production, full of outdated deps, wasting in the meantime storage and cpu resources...

oneplane 2y ago
Yet it doesn't even come close to a fraction of the adoption scale of containers, no matter how good it is. Ecosystems matter more than individual quality.
- kkfx 2y ago
  That's because some interested parties have advertised containers, because they are good to sell as pre-built stuff, nice to sell VPS and alike etc, while pure IaC is useful for anyone and invite NOT to be dependent on third party platforms. It's not a technical matter, it's a human, economical matter and actually... Most people are poor, following the largest scale means following poverty not a good thing.
- bigstrat2003 2y ago
  I think that's a gross overstatement. Ecosystems matter, yes. But they don't matter more than quality.
  - oneplane 2y ago
    I think Betamax might have something to say about that.

bluGill 2y ago

There is no such thing as a reproducible build environment anymore. You can get a temporary reproducible build environment, but any sane security policy will have certificates that expire and that in turn means that in a couple years your build environment won't be reproducible anymore.

andix 2y ago
If you run multiple instances of a container image, you get a reproducible environment. If you run a docker build multiple times, and copy a few files into the container, you get a reproducible container image. It is not a hash perfect duplicate, but functionally equivalent. If builds of your favourite programming language are reproducible or not, is not really related to VM vs. Container.
cesarb 2y ago
> but any sane security policy will have certificates that expire and that in turn means that in a couple years your build environment won't be reproducible anymore. "Reproducible" is usually defined as "identical output except for the cryptographic signature at the end" (and that should be the only use for a certificate in your build environment, a high-quality build environment should be self-contained and have no network access). That is, once you remove the signature, the built artifacts should be bit-by-bit identical.
- bluGill 2y ago
  I said environment not build.

belter 2y ago

You are absolutely correct, and the creators of Docker did mention that was the core reason. Unfortunately your comment comes 10 years too late for many.

nimbius 2y ago

we arguably already had this with things like python venv. the articles main point still remains, containers are a slow and bloated answer to this problem. I concede youll need containers for Kubernetes, and Kubernetes on the surface is a very good idea, but this level of infrastructure automation exists already in things like foreman and openstack. designs like shift-on-stack trade simplicity of traditional hardware for ever byzantine levels of brittle versioned complexity...so ultimately instead of fixing the problem we invoke the god of immutability, destroy and rebuild, and hope the problem fixes itself somehow...its really quite comical. baremetal rust/python/go with good architecture and CI will absolutely crush container workloads in a fraction of disk, CPU, RAM, and personal frustration.

jchw 2y ago
Python venv is language specific, doesn't handle the interpreter version and doesn't handle C libraries. I really don't understand why people do this: I get having a distaste for containers but some people, seeing the massive success of OCI images, mainly seem content on trying to figure out how to discredit its popularity, rather than trying to understand why it's popular. The former may be good for contrarian Internet forums, but the latter is more practically useful and interesting. I say this with some level of understanding as I also have a distaste for containers and Docker is not my preferred way to do "hermetic" or "reproducible" (I am a huge Nix proponent.) I want to get past the "actually it was clearly useless from the start" because it wasn't...
jbverschoor 2y ago
Not really, you'd still need a proper chroot / etc. Check out https://github.com/jrz/container-shell https://github.com/jrz/container-shell

cduzz 2y ago

I tell people "An OCI container is a way to turn any random runtime into a statically linked binary." It is very useful for managing dependency hell, or at least moving it into "API dependencies" not "Library dependencies", it is handy for pickling a CI/CD release engineering infrastructure. It's not a security boundary. (I'm 100% agreeing with parent, in case I sound contentious)

scarby2 2y ago
> It's not a security boundary. It is a security boundary, just not necessarily the best one.
- andix 2y ago
  I would even claim it's a pretty good security boundary, good enough for most applications.
- bayindirh 2y ago
  It's an incidental security boundary because CGroups happen to isolate the process fairly well.
  - vilunov 2y ago
    Yeah, but that's not an incidental property of *namespaces* (of which cgroups is only one isolation axis), that was the requirement when namespaces were designed.
    - bayindirh 2y ago
      Yeah, I know. Namespaces are pretty cool outside containers too. My comment was more of a soft jab against using containers as the ultimate "thing" for anything and everything. I prefer to use them as "statically linked binaries" for short lived processes (like document building, etc.). But, whenever someone abuses containers (like adding an HTTPs fronting container in front of anything which can handle HTTPS on its own) I'm displeased. Relevant XKCD: https://xkcd.com/1988/ https://xkcd.com/1988/

BobbyTables2 2y ago

I agree with you, but the world seems to think otherwise!

flanked-evergl 2y ago

Here are some projects that run docker containers on top of micro/lightweight VMs: - https://github.com/firecracker-microvm/firecracker-containerd https://github.com/firecracker-microvm/firecracker-container... - https://github.com/kuasar-io/kuasar https://github.com/kuasar-io/kuasar - https://github.com/kata-containers/kata-containers https://github.com/kata-containers/kata-containers - https://github.com/QuarkContainer/Quark https://github.com/QuarkContainer/Quark - https://github.com/google/gvisor https://github.com/google/gvisor - https://github.com/containers/libkrun https://github.com/containers/libkrun

lttlrck 2y ago

The main advantage in my use case is in fact isolation (network and volumes) and a well defined API enabling management of those containers in production (not k8s, a tiny subnet of that perhaps). The isolation could be achieved using namespaces directly. But the API, tooling and registry add a lot of value that would otherwise require a lot of development. Also last time I looked hypervisors aren't possible on all cloud vendors, unless you have a bare metal server. This matters in my case. Maybe it has changed in the past 3 years. When docker fits it's great. Same can be said of k8s, where there are a whole bunch of additional benefits. Swings and roundabouts.

deleted 2y ago

[deleted]