5 ms·

Crun: Fast and lightweight OCI runtime and C library for running containers

109 points by nateb2022 3y ago

eska 3y ago
* crun could go much lower than that, and require < 1M. The used 4MB is a hard limit set directly in Podman before calling the OCI runtime.* Enough room to hide a lisp implementation!
snapplebobapple 3y ago
For those of us not into this space, does this replace docker or kubernetes or just some piece of the puzzle that could be bolted into both to replace some component?
- rektide 3y ago
  Kubernetes needs an OCI runtime to run containers with. Crun is one implementation it can use. Docker also appears to be able to use crun for it's engine as well. https://github.com/containers/crun/issues/37 https://github.com/containers/crun/issues/37
- linkdd 3y ago
  Take a car, the engine is the OCI runtime, crun, containerd, etc.. The whole car is Docker, or Podman. Kubernetes is the city where the cars run. --- EDIT: For more details, OCI (Open Container Initiative) defines a few things: - an image format (how to create them) - a runtime specification (how to run them) - ... A container runtime is an implementation of one of those things. Docker is a set of tools hidden under a unified CLI which gives you the ability to: - create images - upload/download images to/from a registry - run images with volumes, networking configuration, environment variables, ... But Docker is only one host. Kubernetes is an interface to abstract a cluster of hosts. Everything is described as a "resource" which goes through the "control loop": 1. the user uses the k8s REST API to create/read/update/delete the resource 2. the k8s api server will contact admission controllers (via webhook) to authorize (and/or mutate) the action 3. the action is persisted to a distributed database (usually, etcd) 4. then, controllers are notified of the change and will run the side effects This is the simplified version. But what is stored in the distributed database is called the "desired" state, and controllers have the duty of observing the real state (the "observed" state) and make it converge towards the "desired" state. So a "Pod" controller's job will be to observe Docker instances, to check what containers are running, and start/stop the containers based on what "Pod" resources exists in k8s's database. A "Deployment" controller's job will be to observe the "Pod" resources in the k8s database and create/update/delete them based on what "Deployment" resources exists in k8s's database. etc... In theory, Kubernetes does not need docker. You could have a "proxmox" controller which would start/stop virtual machines instead. Kubernetes provides a lot of tooling for storage management, secret management, networking, workload management, etc... so that you can manage it all with a unified REST API. The very nature of the "control loop" makes it very extensible, allowing you to build layers of abstraction on top of layers of abstraction on top of layers of abstraction ... A real "onion cloud" if I dare say it.
  - benatkin 3y ago
    > The whole car is Docker, or Podman. Kubernetes is the city where the cars run. Kubernetes doesn't use Docker or Podman.
    - linkdd 3y ago
      And there are more than 2 models of cars. The metaphor is still valid.
    - rumdz 3y ago
      What does Kubernetes use instead?
      - pipe_connector 3y ago
        kubernetes can use anything that conforms to the CRI interface, which in practice is either CRI-O (RedHat) or Containerd (Docker, Inc.). Podman and Docker are also consumers of both of those engines
- p_l 3y ago
  It replaces `runc` which is used by most non-docker container runtimes to actually start the container. Thus the punny name. When using kubernetes, the hierarchy is as follows: 1. kubernetes master tells kubelet what to do (sort of, not important here) 2. kubelet uses CRI-compatible runtime to start containers 3. containerd or CRI-O handle management of containers and start them using runc or crun 4. runc/crun are the applications that setup the final environment of application to run in container, using resources (mounts, devices, etc) provided to them by upper layers. They also handle things like sending stdout/stderr to logs, or setting up a pseudoterminal to talk to a program in container, etc.
  - cpuguy83 3y ago
    I mean, Docker is using runc by default as well.
- cmckn 3y ago
  Kubelet speaks CRI to containerd, which speaks OCI to runc (or crun). Docker can be wedged in there between Kubernetes and containerd (which was originally part of Docker). The OCI implementation is the lowest-level component in the “container stack”.
andrewstuart 3y ago
systemd has nspawn, often overlooked ion the container discussion. https://wiki.archlinux.org/title/systemd-nspawn https://wiki.archlinux.org/title/systemd-nspawn "systemd-nspawn is like the chroot command, but it is a chroot on steroids. systemd-nspawn may be used to run a command or OS in a light-weight namespace container. It is more powerful than chroot since it fully virtualizes the file system hierarchy, as well as the process tree, the various IPC subsystems and the host and domain name. "
- wmf 3y ago
  nspawn (and LXC/LXD) is overlooked because people mostly want application containers not system containers.
  - cpufry 3y ago
    yea thats what my general understanding of that was too, people want the most shrink wrappiest and minimal virtualisation boundary
  - alexgartrell 3y ago
    I actually disagree that this is the reason for the aversion. Instead I think it comes down to a couple of things: 1. Containers are commodity. For commodities, price wins first, then marketing. Docker is equally free (as in beer) and is far better marketed 2. There are a very low number of people working on systemd-nspawn and a very high number of people working on docker (and the ecosystem). 3. Dockerfiles and images are ubiquitous. They're easy to support from other run times, but if you're already in the ecosystem, what's the incentive to change?
    - thayne 3y ago
      There is also the fact that some people have a knee jerk reaction to hate anything associated with systemd. Although, I think that is less significant than the two reasons you listed.
      - 3y ago
pm90 3y ago
I can’t go into too much detail but: IME the benefits are somewhat doubtful and it can run into weird issues that are hard to debug (because C) at scale. I wouldn’t use it unless I had a maintainer or a C expert on my team. The other OCI runtimes are written in go and are (generally) easier to debug.
- yjftsjthsd-h 3y ago
  Why is Go easier to debug? It kinda sounds like you're more familiar with one language than another and are basing your assessment of the tool on that.
  - Gigachad 3y ago
    Less undefined behavior I would assume.
  - winrid 3y ago
    Easier to track allocations and tie them to structures.
    - dilyevsky 3y ago
      Technically if you use jemalloc, which most everyone should do anyway, it comes with built-in instrumentation but you need to enable it compile time and generally not many are aware of this.
      - winrid 3y ago
        I work on many C/C++ projects and none of them use jemalloc. I also use several databases written in C, or C++, and none of them use jemalloc either. I do however wish Mongo offered the option: https://jira.mongodb.org/browse/SERVER-39325 https://jira.mongodb.org/browse/SERVER-39325
        dilyevsky 3y ago
        I think tcmalloc will output protos too now which google pprof tool understands. If you’re using standard glibc malloc you’re probably leaving a lot of performance on the table
Gigachad 3y ago
I'd be very cautious about running such a core piece of infrastructure in an insecure language. Especially one without the level of scrutiny other projects like linux have.
singpolyma3 3y ago
I'm no fan of Go, but "I used C instead" is not a selling point
- pjmlp 3y ago
  Indeed for all its design issues, it is still way better than using plain old C, unless there is no way around it.
- gscrivano 3y ago
  being the main author of crun, I can clarify that statement: I am not a fan of Go _for this particular use case_. Using C instead of Go avoided a bunch of the workarounds that exists in runc to workaround the Go runtime, e.g. https://github.com/opencontainers/runc/blob/main/libcontainer/nsenter/nsexec.c https://github.com/opencontainers/runc/blob/main/libcontaine...
seckuriti 3y ago
I try to avoid setuid binaries written in memory-unsafe languages. This feels like the wrong direction.
- hencoappel 3y ago
  Looks like there is youki [1] for that. [1] https://github.com/containers/youki https://github.com/containers/youki
- gscrivano 3y ago
  crun is not a setuid program
akkartik 3y ago
I went looking for an answer to the obvious question, and there is indeed a Rust version. https://github.com/containers/youki#motivation https://github.com/containers/youki#motivation has a nice comparison with both runc and crun.
hkt 3y ago
> A Lua binding is available. Bliss. Fantastic. I'm going to write something with this. I remain delighted every single time somebody rewrites a thing in C.

109 points by nateb2022 3y ago

eska 3y ago

* crun could go much lower than that, and require < 1M. The used 4MB is a hard limit set directly in Podman before calling the OCI runtime.* Enough room to hide a lisp implementation!

snapplebobapple 3y ago

For those of us not into this space, does this replace docker or kubernetes or just some piece of the puzzle that could be bolted into both to replace some component?

rektide 3y ago
Kubernetes needs an OCI runtime to run containers with. Crun is one implementation it can use. Docker also appears to be able to use crun for it's engine as well. https://github.com/containers/crun/issues/37 https://github.com/containers/crun/issues/37
linkdd 3y ago
Take a car, the engine is the OCI runtime, crun, containerd, etc.. The whole car is Docker, or Podman. Kubernetes is the city where the cars run. --- EDIT: For more details, OCI (Open Container Initiative) defines a few things: - an image format (how to create them) - a runtime specification (how to run them) - ... A container runtime is an implementation of one of those things. Docker is a set of tools hidden under a unified CLI which gives you the ability to: - create images - upload/download images to/from a registry - run images with volumes, networking configuration, environment variables, ... But Docker is only one host. Kubernetes is an interface to abstract a cluster of hosts. Everything is described as a "resource" which goes through the "control loop": 1. the user uses the k8s REST API to create/read/update/delete the resource 2. the k8s api server will contact admission controllers (via webhook) to authorize (and/or mutate) the action 3. the action is persisted to a distributed database (usually, etcd) 4. then, controllers are notified of the change and will run the side effects This is the simplified version. But what is stored in the distributed database is called the "desired" state, and controllers have the duty of observing the real state (the "observed" state) and make it converge towards the "desired" state. So a "Pod" controller's job will be to observe Docker instances, to check what containers are running, and start/stop the containers based on what "Pod" resources exists in k8s's database. A "Deployment" controller's job will be to observe the "Pod" resources in the k8s database and create/update/delete them based on what "Deployment" resources exists in k8s's database. etc... In theory, Kubernetes does not need docker. You could have a "proxmox" controller which would start/stop virtual machines instead. Kubernetes provides a lot of tooling for storage management, secret management, networking, workload management, etc... so that you can manage it all with a unified REST API. The very nature of the "control loop" makes it very extensible, allowing you to build layers of abstraction on top of layers of abstraction on top of layers of abstraction ... A real "onion cloud" if I dare say it.
- benatkin 3y ago
  > The whole car is Docker, or Podman. Kubernetes is the city where the cars run. Kubernetes doesn't use Docker or Podman.
  - linkdd 3y ago
    And there are more than 2 models of cars. The metaphor is still valid.
  - rumdz 3y ago
    What does Kubernetes use instead?
    - pipe_connector 3y ago
      kubernetes can use anything that conforms to the CRI interface, which in practice is either CRI-O (RedHat) or Containerd (Docker, Inc.). Podman and Docker are also consumers of both of those engines
p_l 3y ago
It replaces `runc` which is used by most non-docker container runtimes to actually start the container. Thus the punny name. When using kubernetes, the hierarchy is as follows: 1. kubernetes master tells kubelet what to do (sort of, not important here) 2. kubelet uses CRI-compatible runtime to start containers 3. containerd or CRI-O handle management of containers and start them using runc or crun 4. runc/crun are the applications that setup the final environment of application to run in container, using resources (mounts, devices, etc) provided to them by upper layers. They also handle things like sending stdout/stderr to logs, or setting up a pseudoterminal to talk to a program in container, etc.
- cpuguy83 3y ago
  I mean, Docker is using runc by default as well.
cmckn 3y ago
Kubelet speaks CRI to containerd, which speaks OCI to runc (or crun). Docker can be wedged in there between Kubernetes and containerd (which was originally part of Docker). The OCI implementation is the lowest-level component in the “container stack”.

andrewstuart 3y ago

systemd has nspawn, often overlooked ion the container discussion. https://wiki.archlinux.org/title/systemd-nspawn https://wiki.archlinux.org/title/systemd-nspawn "systemd-nspawn is like the chroot command, but it is a chroot on steroids. systemd-nspawn may be used to run a command or OS in a light-weight namespace container. It is more powerful than chroot since it fully virtualizes the file system hierarchy, as well as the process tree, the various IPC subsystems and the host and domain name. "

wmf 3y ago
nspawn (and LXC/LXD) is overlooked because people mostly want application containers not system containers.
- cpufry 3y ago
  yea thats what my general understanding of that was too, people want the most shrink wrappiest and minimal virtualisation boundary
- alexgartrell 3y ago
  I actually disagree that this is the reason for the aversion. Instead I think it comes down to a couple of things: 1. Containers are commodity. For commodities, price wins first, then marketing. Docker is equally free (as in beer) and is far better marketed 2. There are a very low number of people working on systemd-nspawn and a very high number of people working on docker (and the ecosystem). 3. Dockerfiles and images are ubiquitous. They're easy to support from other run times, but if you're already in the ecosystem, what's the incentive to change?
  - thayne 3y ago
    There is also the fact that some people have a knee jerk reaction to hate anything associated with systemd. Although, I think that is less significant than the two reasons you listed.
    - 3y ago

pm90 3y ago

I can’t go into too much detail but: IME the benefits are somewhat doubtful and it can run into weird issues that are hard to debug (because C) at scale. I wouldn’t use it unless I had a maintainer or a C expert on my team. The other OCI runtimes are written in go and are (generally) easier to debug.

yjftsjthsd-h 3y ago
Why is Go easier to debug? It kinda sounds like you're more familiar with one language than another and are basing your assessment of the tool on that.
- Gigachad 3y ago
  Less undefined behavior I would assume.
- winrid 3y ago
  Easier to track allocations and tie them to structures.
  - dilyevsky 3y ago
    Technically if you use jemalloc, which most everyone should do anyway, it comes with built-in instrumentation but you need to enable it compile time and generally not many are aware of this.
    - winrid 3y ago
      I work on many C/C++ projects and none of them use jemalloc. I also use several databases written in C, or C++, and none of them use jemalloc either. I do however wish Mongo offered the option: https://jira.mongodb.org/browse/SERVER-39325 https://jira.mongodb.org/browse/SERVER-39325
      - dilyevsky 3y ago
        I think tcmalloc will output protos too now which google pprof tool understands. If you’re using standard glibc malloc you’re probably leaving a lot of performance on the table

Gigachad 3y ago

I'd be very cautious about running such a core piece of infrastructure in an insecure language. Especially one without the level of scrutiny other projects like linux have.

singpolyma3 3y ago

I'm no fan of Go, but "I used C instead" is not a selling point

pjmlp 3y ago
Indeed for all its design issues, it is still way better than using plain old C, unless there is no way around it.
gscrivano 3y ago
being the main author of crun, I can clarify that statement: I am not a fan of Go _for this particular use case_. Using C instead of Go avoided a bunch of the workarounds that exists in runc to workaround the Go runtime, e.g. https://github.com/opencontainers/runc/blob/main/libcontainer/nsenter/nsexec.c https://github.com/opencontainers/runc/blob/main/libcontaine...

seckuriti 3y ago

I try to avoid setuid binaries written in memory-unsafe languages. This feels like the wrong direction.

hencoappel 3y ago
Looks like there is youki [1] for that. [1] https://github.com/containers/youki https://github.com/containers/youki
gscrivano 3y ago
crun is not a setuid program

akkartik 3y ago

I went looking for an answer to the obvious question, and there is indeed a Rust version. https://github.com/containers/youki#motivation https://github.com/containers/youki#motivation has a nice comparison with both runc and crun.

hkt 3y ago

> A Lua binding is available. Bliss. Fantastic. I'm going to write something with this. I remain delighted every single time somebody rewrites a thing in C.