3 ms·

Show HN: krunvm – Create and run lightweight VMs from OCI images

82 points by slpnix 6y ago

SomaticPirate 6y ago
I’m honestly just curious how they managed to get a KVM based solution working on MacOS ARM64. Buildah and few other dependencies appeared to be Linux only... but the brew install worked..
- slpnix 6y ago
  The key is that libkrun (https://github.com/containers/libkrun https://github.com/containers/libkrun), the library that krunvm uses for running the VMs, as recently integrated support for Hypervisor.framework on ARM64, in addition to KVM. As for buildah, the Homebrew repo contains a build that includes this PR (https://github.com/containers/storage/pull/811 https://github.com/containers/storage/pull/811).
akkishore 6y ago
This is awesome. How does this compare with Firecracker - https://github.com/firecracker-microvm/firecracker https://github.com/firecracker-microvm/firecracker Why only TCP is available? Network stack needs to be implemented?
- DoomHotel 6y ago
  It's using a different method of loading the kernel: https://github.com/containers/libkrunfw https://github.com/containers/libkrunfw Firecracker is using rust-vmm: https://github.com/rust-vmm https://github.com/rust-vmm
- slpnix 6y ago
  krunvm uses libkrun (https://github.com/containers/libkrun https://github.com/containers/libkrun) for executing the VM, and while the later is also based in rust-vmm and shares some code with Firecracker and Cloud-Hypervisor, it's specialized in the process isolation use case. This means it implements a different set of devices (most notably, virtio-fs instead of virtio-blk, and virtio-vsock+TSI (Transparent Socket Impersonation) instead of virtio-net), and it takes the form of a dynamic library instead of a final binary. In fact, the networking limitations are caused by this use of virtio-vsock+TSI. TSI (WIP implementation here: https://github.com/containers/libkrunfw/blob/main/patches/0003-Implement-TSI-WIP.patch https://github.com/containers/libkrunfw/blob/main/patches/00...) is an experimental mechanism that provides inbound and outbound networking capabilities to the guest, with zero-configuration and minimal footprint, by transparently replacing user-space AF_INET sockets with AF_TSI, that have both an AF_INET and AF_VSOCK personality. TSI has the additional advantage that, for the host side, all connections appear to come and go to the process acting as a VMM (in this case, krunvm, as it links directly with libkrun), which makes it very container-friendly in a way that even side-cars (such as Istio) work out-of-the-box.
- ivzhh 6y ago
  Also, firecracker did not merge virtio-fs [1]. So I guess it is a scope and priority decision: > To not trade off our core use case simplifying assumptions, we'll need something like a way to build different Firecracker "specialized variants" or alternatively a plugin system. This is something worth discussing for 2021. Again this is not a guarantee, but virtio-fs could fit in such a scheme, especially if it's a rust-vmm crate we can easily import over. [1]: https://github.com/firecracker-microvm/firecracker/pull/1351#issuecomment-667085798 https://github.com/firecracker-microvm/firecracker/pull/1351...
hibbelig 6y ago
How would this compare for current users of Docker for Mac? If I use Docker for Mac to run two containers, I get one VM that runs two containers, right? And if I use this, then I get two VMs, one per container? How does the resource usage compare? I have to tell Docker for Mac how much RAM to allocate, and that's then shared between the containers. Can krunvm use as much RAM as the container needs, or does it also need some pre-allocation?
- slpnix 6y ago
  I can't comment on the comparison with Docker for Mac bacause, honestly, I have never used it. With krunvm, each session initiated with "krunvm start" is an independent lightweight VM. The maximum amount of RAM the VM can use is configured with the "mem" flag (or it uses what's configured by default), but the VMM will always try to use the minimum possible amount of RAM by returning the pages the guest is no longer using to the host (virtio-balloon's free page reporting feature). That said, you can also start a single VM (using "krunvm start"), run podman's service ("podman system service...") inside it, switch to another terminal in macOS and execute multiple containers inside the VM using "podman remote" [1]. Now I'm thinking I should probably write a tutorial about this option ;-) [1] https://www.redhat.com/sysadmin/podman-clients-macos-windows https://www.redhat.com/sysadmin/podman-clients-macos-windows

82 points by slpnix 6y ago

I’m honestly just curious how they managed to get a KVM based solution working on MacOS ARM64. Buildah and few other dependencies appeared to be Linux only... but the brew install worked..

slpnix 6y ago
The key is that libkrun (https://github.com/containers/libkrun https://github.com/containers/libkrun), the library that krunvm uses for running the VMs, as recently integrated support for Hypervisor.framework on ARM64, in addition to KVM. As for buildah, the Homebrew repo contains a build that includes this PR (https://github.com/containers/storage/pull/811 https://github.com/containers/storage/pull/811).

akkishore 6y ago

This is awesome. How does this compare with Firecracker - https://github.com/firecracker-microvm/firecracker https://github.com/firecracker-microvm/firecracker Why only TCP is available? Network stack needs to be implemented?

DoomHotel 6y ago
It's using a different method of loading the kernel: https://github.com/containers/libkrunfw https://github.com/containers/libkrunfw Firecracker is using rust-vmm: https://github.com/rust-vmm https://github.com/rust-vmm
slpnix 6y ago
krunvm uses libkrun (https://github.com/containers/libkrun https://github.com/containers/libkrun) for executing the VM, and while the later is also based in rust-vmm and shares some code with Firecracker and Cloud-Hypervisor, it's specialized in the process isolation use case. This means it implements a different set of devices (most notably, virtio-fs instead of virtio-blk, and virtio-vsock+TSI (Transparent Socket Impersonation) instead of virtio-net), and it takes the form of a dynamic library instead of a final binary. In fact, the networking limitations are caused by this use of virtio-vsock+TSI. TSI (WIP implementation here: https://github.com/containers/libkrunfw/blob/main/patches/0003-Implement-TSI-WIP.patch https://github.com/containers/libkrunfw/blob/main/patches/00...) is an experimental mechanism that provides inbound and outbound networking capabilities to the guest, with zero-configuration and minimal footprint, by transparently replacing user-space AF_INET sockets with AF_TSI, that have both an AF_INET and AF_VSOCK personality. TSI has the additional advantage that, for the host side, all connections appear to come and go to the process acting as a VMM (in this case, krunvm, as it links directly with libkrun), which makes it very container-friendly in a way that even side-cars (such as Istio) work out-of-the-box.
ivzhh 6y ago
Also, firecracker did not merge virtio-fs [1]. So I guess it is a scope and priority decision: > To not trade off our core use case simplifying assumptions, we'll need something like a way to build different Firecracker "specialized variants" or alternatively a plugin system. This is something worth discussing for 2021. Again this is not a guarantee, but virtio-fs could fit in such a scheme, especially if it's a rust-vmm crate we can easily import over. [1]: https://github.com/firecracker-microvm/firecracker/pull/1351#issuecomment-667085798 https://github.com/firecracker-microvm/firecracker/pull/1351...

hibbelig 6y ago

How would this compare for current users of Docker for Mac? If I use Docker for Mac to run two containers, I get one VM that runs two containers, right? And if I use this, then I get two VMs, one per container? How does the resource usage compare? I have to tell Docker for Mac how much RAM to allocate, and that's then shared between the containers. Can krunvm use as much RAM as the container needs, or does it also need some pre-allocation?

slpnix 6y ago
I can't comment on the comparison with Docker for Mac bacause, honestly, I have never used it. With krunvm, each session initiated with "krunvm start" is an independent lightweight VM. The maximum amount of RAM the VM can use is configured with the "mem" flag (or it uses what's configured by default), but the VMM will always try to use the minimum possible amount of RAM by returning the pages the guest is no longer using to the host (virtio-balloon's free page reporting feature). That said, you can also start a single VM (using "krunvm start"), run podman's service ("podman system service...") inside it, switch to another terminal in macOS and execute multiple containers inside the VM using "podman remote" [1]. Now I'm thinking I should probably write a tutorial about this option ;-) [1] https://www.redhat.com/sysadmin/podman-clients-macos-windows https://www.redhat.com/sysadmin/podman-clients-macos-windows