3 ms·

Half-OT: What are alternative runtimes and why would you use them?

by k__ 6y ago

mfer 6y ago
For a runtime in your Kubernetes cluster there are containerd and cri-o. These are good for Docker / Open Container Initiative images. There are others... some for non-Docker image support. There are people running other things than just Docker these days. They are more niche case.
freedomben 6y ago
In the OpenShift world we use CRI-O and it has been awesome for us. I've never actually had it be a problem. Occasionally have to SSH into a node and inspect with crictl to see what's going on but it's almost always PEBKAC that points at CRI-O when it's not CRI-O's fault. I'd definitely recommend looking at it.
- sandGorgon 6y ago
  How has been your experience with crictl ? Does it do everything that docker cli does ? Build, pull, etc ?
  - jsmith45 6y ago
    Only docker, buildkit, podman, buildah, and some non-Dockerfile based tools are able to build. CRI-O or containerd do not have build functionality. It does have pretty much everything else you expect like pull, ps, run exec, attach, rm, rmi, etc. If you need builds, I'd suggest either run dockerd alongside a containerd kubelet, or use buildkit.
    - sandGorgon 6y ago
      That's super valuable. Thanks! I did not know about buildkit (and buildctl), but now see it. Very valuable. Thanks!!!
jsmith45 6y ago
The main alternative runtime that I know of (at the level of containerd) is CRI-O. These runtimes are at the level of fetching images, preparing manifests etc. I'm not really sure what benefits they provide. CRI-O is intended to be kubernetes specific, and thus lacks any features that containerd would have that k8s does not need. This in theory ought to mean smaller, lighter, more easily auditable code. There is another lower level of runtime, the OCI runtime, of which the main implementation is runc. Alternatives have interesting attributes, like `runv` running containers in VMs with their own kernel to get even grater isolation, `runhcs` which is the OCI runtime for running windows containers, etc. Most if not all of the higher level runtimes allow switching out the OCI runtime, but in general sticking with the default of `runc` is fine.
- ghaff 6y ago
  Yeah, the terminology around "runtime" is confusing and is used inconsistently. As you say, the actual runtime is something like runc which CRI-O (and I believe containerd) normally uses. CRI-O, as the name suggests, is an implementation of the Kubernetes Container Runtime Interface--which should work with any OCI-compliant runtime.
seabrookmx 6y ago
One such alternative runtime with tangible differences to containerd/cri-o is Kata[1], which actually runs OCI images as microvm's. This has some benefits if the applications you're running are untrusted/need additional sandboxing, such as if you're running a PaaS on bare-metal k8s. [1]: https://github.com/kata-containers/runtime https://github.com/kata-containers/runtime
- k__ 6y ago
  I did some research, and it also seems that there is a way for containerd to manage Firecracker microvms.
eska 6y ago
A simple example would be mostly running Linux applications on Linux machines (= containers), but having a Windows application that needs to run in a Windows VM on a Linux machine (VM, or a container in a VM).

Half-OT: What are alternative runtimes and why would you use them?

by k__ 6y ago

Half-OT: What are alternative runtimes and why would you use them?

mfer 6y ago

For a runtime in your Kubernetes cluster there are containerd and cri-o. These are good for Docker / Open Container Initiative images. There are others... some for non-Docker image support. There are people running other things than just Docker these days. They are more niche case.

freedomben 6y ago

In the OpenShift world we use CRI-O and it has been awesome for us. I've never actually had it be a problem. Occasionally have to SSH into a node and inspect with crictl to see what's going on but it's almost always PEBKAC that points at CRI-O when it's not CRI-O's fault. I'd definitely recommend looking at it.

sandGorgon 6y ago
How has been your experience with crictl ? Does it do everything that docker cli does ? Build, pull, etc ?
- jsmith45 6y ago
  Only docker, buildkit, podman, buildah, and some non-Dockerfile based tools are able to build. CRI-O or containerd do not have build functionality. It does have pretty much everything else you expect like pull, ps, run exec, attach, rm, rmi, etc. If you need builds, I'd suggest either run dockerd alongside a containerd kubelet, or use buildkit.
  - sandGorgon 6y ago
    That's super valuable. Thanks! I did not know about buildkit (and buildctl), but now see it. Very valuable. Thanks!!!

jsmith45 6y ago

The main alternative runtime that I know of (at the level of containerd) is CRI-O. These runtimes are at the level of fetching images, preparing manifests etc. I'm not really sure what benefits they provide. CRI-O is intended to be kubernetes specific, and thus lacks any features that containerd would have that k8s does not need. This in theory ought to mean smaller, lighter, more easily auditable code. There is another lower level of runtime, the OCI runtime, of which the main implementation is runc. Alternatives have interesting attributes, like `runv` running containers in VMs with their own kernel to get even grater isolation, `runhcs` which is the OCI runtime for running windows containers, etc. Most if not all of the higher level runtimes allow switching out the OCI runtime, but in general sticking with the default of `runc` is fine.

ghaff 6y ago
Yeah, the terminology around "runtime" is confusing and is used inconsistently. As you say, the actual runtime is something like runc which CRI-O (and I believe containerd) normally uses. CRI-O, as the name suggests, is an implementation of the Kubernetes Container Runtime Interface--which should work with any OCI-compliant runtime.

seabrookmx 6y ago

One such alternative runtime with tangible differences to containerd/cri-o is Kata[1], which actually runs OCI images as microvm's. This has some benefits if the applications you're running are untrusted/need additional sandboxing, such as if you're running a PaaS on bare-metal k8s. [1]: https://github.com/kata-containers/runtime https://github.com/kata-containers/runtime

k__ 6y ago
I did some research, and it also seems that there is a way for containerd to manage Firecracker microvms.

eska 6y ago

A simple example would be mostly running Linux applications on Linux machines (= containers), but having a Windows application that needs to run in a Windows VM on a Linux machine (VM, or a container in a VM).