15 ms·

Execute Docker Containers as QEMU MicroVMs

178 points by DarkPlayer 5y ago

eatonphil 5y ago
There are a few existing projects out there like this (running Docker images as virtual machines, specifically) if folks are interested. Slim [0] is the one I can remember off the top of my head. I think there are a couple more. Still, neat to have the walkthrough here in this post. https://github.com/ottomatica/slim https://github.com/ottomatica/slim
- hardwaresofton 5y ago
  A couple more: https://github.com/containers/krunvm https://github.com/containers/krunvm https://github.com/weaveworks/ignite https://github.com/weaveworks/ignite
rwmj 5y ago
https://katacontainers.io/ https://katacontainers.io/ ?
- bonzini 5y ago
  Yes, indeed. However it's nice to see directly the mechanisms that let Kata do its magic.
deleted 5y ago
[deleted]
riobard 5y ago
A few years ago I invested in a small startup called `hyper.sh`. It open sourced a container runtime called `runV` which provided exactly this: security of virtual machines plus convenience of containers. The project later merged with Intel Clear Container to become what's now called Kata Containers (https://katacontainers.io/ https://katacontainers.io/) and is now widely used by several Internet giants like Alibaba and Baidu. The startup was acquired by Ant Finance a couple of years ago. (I recorded a podcast with one of hyper.sh engineer if you can listen to Mandarin https://pan.icu/25 https://pan.icu/25)
- temp_praneshp 5y ago
  Probably off topic: Back in 2014-15 at my first job, when I was working on openstack, they used to show up at the summits. They were super smart and very generous with their time when I had questions. I wondered sometime in 2020 what happened to them, I'm happy they had a decent exit.
- polskibus 5y ago
  How does it differ from Firecracker?
  - riobard 5y ago
    I'm not familiar with later development, but AFAIK Firecracker came much later and now you can actually use Firecracker as Kata Container's hypervisor in addition to QEMU.
- deleted 5y ago
  [deleted]
- cptnapalm 5y ago
  I was looking at Kata containers a few days ago. I'm pretty new to trying to use VMs/containers for services; purely hobby level. Couldn't figure out how to use them, but that's not necessarily a knock on them as I also can't get OpenBSD wireguard to work either.
- lifty 5y ago
  I worked with their tech, testing it, and I loved the product. It was definitely ahead of its time. Similar in some ways to what Fly is doing these days, without the edge.
- XorNot 5y ago
  I used runV with drone.io (on top of Media) to run distributed on-demand VM builders for GitHub enterprise (we were building physical machine images to deploy so needed VM isolation). It actually worked great, and I've struggled to get as quite a flexible CI system at other jobs since then (the big advantage was it looked like Docker, so with compose you could either spin a metal-like nested VM or just pull in some DB containers in your build instance).
forty 5y ago
Isn't firecracker an AWS tech?
- cpach 5y ago
  That’s correct. https://github.com/firecracker-microvm/firecracker https://github.com/firecracker-microvm/firecracker
  - remram 5y ago
    The article wrongly states that Fly.io created firecracker.
- jjacobson93 5y ago
  Yeah, the author is incorrect. Fly.io uses Firecracker but they didn’t create it.
  - tptacek 5y ago
    Whoah, missed that.
- bhawks 5y ago
  If you're splitting hairs firecracker (aws) is an offshoot of crosvm from chrome/Google which actually was a greenfield vmm :) anyway memory safe virtualization for the win.
encryptluks2 5y ago
Why not run containers in VMs in containers in VMs? :) Seriously, VMs are hardly as secure as many people want to believe unless you're utilizing enclaves and even that has vulnerabilities. I think a better approach is Seccomp and whatever other filtering makes sense.
- riobard 5y ago
  That's the approach taken by Google's gVisor (at the cost of I/O and network performance).
  - encryptluks2 5y ago
    gVisor does more than filtering, they basically reimplemented the syscalls in an application kernel. At least with seccomp the performance overhead is minimal.
    - remram 5y ago
      How does gVisor fair against KVM and other hardware-accelerated VM solutions (firecracker)?
  - fsociety 5y ago
    gVisor, for better or for worse, does a whole lot of other things than just seccomp filtering, and it shows in performance tests.
  - tptacek 5y ago
    No, that's really not at all what gVisor is. gVisor is best thought of as user-mode Linux --- a complete reimplementation of most of the OS kernel. It's not a system call filter; it's something much closer to a VM than to seccomp. gVisor is a very cool codebase. As an illustration of the approach: it includes its own TCP/IP stack; we use it in our command-line dev tool to allow people to SSH to their VMs over WireGuard without having to install WireGuard or obtain privileges to manage WireGuard.
- handrous 5y ago
ashishbijlani 5y ago
> Can we somehow combine the advantages of the docker ecosystem with VMs? Shameless plug: this is exactly what our goal is with https://kwarantine.xyz https://kwarantine.xyz We are creating a new hypervisor (from scratch) that can run strongly isolated Docker/LXC containers.
- mikepurvis 5y ago
  Is this what gvisor is? https://github.com/google/gvisor https://github.com/google/gvisor
  - ashishbijlani 5y ago
    No, gVisor is from Google. They emulate system calls in user-space and use VMs, which increases runtime performance overhead. We use hardware virtualization to directly run containers -- no I/O emulation, no expensive VM exits, scale as needed. Initial comparison with FC/GVisor/Xen here: https://github.com/ashishbijlani/kwarantine https://github.com/ashishbijlani/kwarantine
    - tptacek 5y ago
      It sounds like you just said "yes, but what we're building is faster". The userland Linux emulation is a security benefit, not a liability.
    - monocasa 5y ago
      I'm not sure gvisor requires vm exits. Their first backend used ptrace very similarly to how user mode Linux worked. Minor quip though since ptrace might even be slower than vm exits; your core point stands.
      - rkeene2 5y ago
        User Mode Linux is still around and works well. I use it when I need a "fakeroot" without any special privileges on the host. https://rkeene.org/viewer/tmp/fakeroot.sh.htm https://rkeene.org/viewer/tmp/fakeroot.sh.htm
thekevjames 5y ago
I had fun exploring Docker->VM conversion a while back [1], though the larger goal in my case was to be able to make the build path to custom GCP VM Images a bit simpler. Exciting to see other cases where folks are finding this sort of flow useful! 1: https://thekev.in/blog/2019-08-05-dockerfile-bootable-vm/index.html https://thekev.in/blog/2019-08-05-dockerfile-bootable-vm/ind...
gravypod 5y ago
Something I'd be very interested in: building a PXE image from something declarative like Dockerfiles.
- laurencerowe 5y ago
  Google Container Optimized OS is basically this I think. It's what's used when you start a GCE instance with a docker image. https://cloud.google.com/container-optimized-os/ https://cloud.google.com/container-optimized-os/
- justincormack 5y ago
  Try LinuxKit https://github.com/linuxkit/linuxkit https://github.com/linuxkit/linuxkit
- jonjonsonjr 5y ago
  I don't think I'd ever call a Dockerfile declarative.
tptacek 5y ago
As I understand the landscape here, the big enabling win of microvms is faster boot time; there's a cool qemu-lite slide deck that goes into detail about how they cut down boot time: https://www.linux-kvm.org/images/d/d2/03x05B-Chao_Peng-Light_Weight_Virtualization_with_QEMU_KVM.pdf https://www.linux-kvm.org/images/d/d2/03x05B-Chao_Peng-Light... The big win was slashing away the BIOS stuff. We use AWS's Firecracker to turn our customers Docker containers into Firecracker microvms (Firecracker is Amazon's Rust VMM, the engine for Fargate and Lambda). Anecdotally: in my dev environment, the difference between Firecracker boot times and native Docker container startup is imperceptible; the logging we do swamps the VM boot stuff. It's very fast.
OldGoodNewBad 5y ago
I think a lot of folks are going out of their way to misunderstand what happened. Yes there are other similar projects and containers. No, none come from a long established COMMUNITY RUN PROJECT. This is something akin to the difference between VirtualBox and OpenBSD’s vmd. Ones a product with a “free” tier, the other is a community project.
dzonga 5y ago
I understand, it's cool to do content marketing. but folks proof-read your articles. Firecracker was created by AWS and rightly states so on the page.
stefanha 5y ago
For an even more lightweight approach to running containers in VMs see: https://github.com/containers/krunvm https://github.com/containers/krunvm It's powered by https://github.com/containers/libkrun https://github.com/containers/libkrun.

178 points by DarkPlayer 5y ago

There are a few existing projects out there like this (running Docker images as virtual machines, specifically) if folks are interested. Slim [0] is the one I can remember off the top of my head. I think there are a couple more. Still, neat to have the walkthrough here in this post. https://github.com/ottomatica/slim https://github.com/ottomatica/slim

hardwaresofton 5y ago
A couple more: https://github.com/containers/krunvm https://github.com/containers/krunvm https://github.com/weaveworks/ignite https://github.com/weaveworks/ignite

rwmj 5y ago

https://katacontainers.io/ https://katacontainers.io/ ?

bonzini 5y ago
Yes, indeed. However it's nice to see directly the mechanisms that let Kata do its magic.

deleted 5y ago

[deleted]

riobard 5y ago

A few years ago I invested in a small startup called `hyper.sh`. It open sourced a container runtime called `runV` which provided exactly this: security of virtual machines plus convenience of containers. The project later merged with Intel Clear Container to become what's now called Kata Containers (https://katacontainers.io/ https://katacontainers.io/) and is now widely used by several Internet giants like Alibaba and Baidu. The startup was acquired by Ant Finance a couple of years ago. (I recorded a podcast with one of hyper.sh engineer if you can listen to Mandarin https://pan.icu/25 https://pan.icu/25)

temp_praneshp 5y ago
Probably off topic: Back in 2014-15 at my first job, when I was working on openstack, they used to show up at the summits. They were super smart and very generous with their time when I had questions. I wondered sometime in 2020 what happened to them, I'm happy they had a decent exit.
polskibus 5y ago
How does it differ from Firecracker?
- riobard 5y ago
  I'm not familiar with later development, but AFAIK Firecracker came much later and now you can actually use Firecracker as Kata Container's hypervisor in addition to QEMU.
deleted 5y ago
[deleted]
cptnapalm 5y ago
I was looking at Kata containers a few days ago. I'm pretty new to trying to use VMs/containers for services; purely hobby level. Couldn't figure out how to use them, but that's not necessarily a knock on them as I also can't get OpenBSD wireguard to work either.
lifty 5y ago
I worked with their tech, testing it, and I loved the product. It was definitely ahead of its time. Similar in some ways to what Fly is doing these days, without the edge.
XorNot 5y ago
I used runV with drone.io (on top of Media) to run distributed on-demand VM builders for GitHub enterprise (we were building physical machine images to deploy so needed VM isolation). It actually worked great, and I've struggled to get as quite a flexible CI system at other jobs since then (the big advantage was it looked like Docker, so with compose you could either spin a metal-like nested VM or just pull in some DB containers in your build instance).

forty 5y ago

Isn't firecracker an AWS tech?

cpach 5y ago
That’s correct. https://github.com/firecracker-microvm/firecracker https://github.com/firecracker-microvm/firecracker
- remram 5y ago
  The article wrongly states that Fly.io created firecracker.
jjacobson93 5y ago
Yeah, the author is incorrect. Fly.io uses Firecracker but they didn’t create it.
- tptacek 5y ago
  Whoah, missed that.
bhawks 5y ago
If you're splitting hairs firecracker (aws) is an offshoot of crosvm from chrome/Google which actually was a greenfield vmm :) anyway memory safe virtualization for the win.

encryptluks2 5y ago

Why not run containers in VMs in containers in VMs? :) Seriously, VMs are hardly as secure as many people want to believe unless you're utilizing enclaves and even that has vulnerabilities. I think a better approach is Seccomp and whatever other filtering makes sense.

riobard 5y ago
That's the approach taken by Google's gVisor (at the cost of I/O and network performance).
- encryptluks2 5y ago
  gVisor does more than filtering, they basically reimplemented the syscalls in an application kernel. At least with seccomp the performance overhead is minimal.
  - remram 5y ago
    How does gVisor fair against KVM and other hardware-accelerated VM solutions (firecracker)?
- fsociety 5y ago
  gVisor, for better or for worse, does a whole lot of other things than just seccomp filtering, and it shows in performance tests.
- tptacek 5y ago
  No, that's really not at all what gVisor is. gVisor is best thought of as user-mode Linux --- a complete reimplementation of most of the OS kernel. It's not a system call filter; it's something much closer to a VM than to seccomp. gVisor is a very cool codebase. As an illustration of the approach: it includes its own TCP/IP stack; we use it in our command-line dev tool to allow people to SSH to their VMs over WireGuard without having to install WireGuard or obtain privileges to manage WireGuard.
handrous 5y ago

ashishbijlani 5y ago

> Can we somehow combine the advantages of the docker ecosystem with VMs? Shameless plug: this is exactly what our goal is with https://kwarantine.xyz https://kwarantine.xyz We are creating a new hypervisor (from scratch) that can run strongly isolated Docker/LXC containers.

mikepurvis 5y ago
Is this what gvisor is? https://github.com/google/gvisor https://github.com/google/gvisor
- ashishbijlani 5y ago
  No, gVisor is from Google. They emulate system calls in user-space and use VMs, which increases runtime performance overhead. We use hardware virtualization to directly run containers -- no I/O emulation, no expensive VM exits, scale as needed. Initial comparison with FC/GVisor/Xen here: https://github.com/ashishbijlani/kwarantine https://github.com/ashishbijlani/kwarantine
  - tptacek 5y ago
    It sounds like you just said "yes, but what we're building is faster". The userland Linux emulation is a security benefit, not a liability.
  - monocasa 5y ago
    I'm not sure gvisor requires vm exits. Their first backend used ptrace very similarly to how user mode Linux worked. Minor quip though since ptrace might even be slower than vm exits; your core point stands.
    - rkeene2 5y ago
      User Mode Linux is still around and works well. I use it when I need a "fakeroot" without any special privileges on the host. https://rkeene.org/viewer/tmp/fakeroot.sh.htm https://rkeene.org/viewer/tmp/fakeroot.sh.htm

thekevjames 5y ago

I had fun exploring Docker->VM conversion a while back [1], though the larger goal in my case was to be able to make the build path to custom GCP VM Images a bit simpler. Exciting to see other cases where folks are finding this sort of flow useful! 1: https://thekev.in/blog/2019-08-05-dockerfile-bootable-vm/index.html https://thekev.in/blog/2019-08-05-dockerfile-bootable-vm/ind...

gravypod 5y ago

Something I'd be very interested in: building a PXE image from something declarative like Dockerfiles.

laurencerowe 5y ago
Google Container Optimized OS is basically this I think. It's what's used when you start a GCE instance with a docker image. https://cloud.google.com/container-optimized-os/ https://cloud.google.com/container-optimized-os/
justincormack 5y ago
Try LinuxKit https://github.com/linuxkit/linuxkit https://github.com/linuxkit/linuxkit
jonjonsonjr 5y ago
I don't think I'd ever call a Dockerfile declarative.

tptacek 5y ago

As I understand the landscape here, the big enabling win of microvms is faster boot time; there's a cool qemu-lite slide deck that goes into detail about how they cut down boot time: https://www.linux-kvm.org/images/d/d2/03x05B-Chao_Peng-Light_Weight_Virtualization_with_QEMU_KVM.pdf https://www.linux-kvm.org/images/d/d2/03x05B-Chao_Peng-Light... The big win was slashing away the BIOS stuff. We use AWS's Firecracker to turn our customers Docker containers into Firecracker microvms (Firecracker is Amazon's Rust VMM, the engine for Fargate and Lambda). Anecdotally: in my dev environment, the difference between Firecracker boot times and native Docker container startup is imperceptible; the logging we do swamps the VM boot stuff. It's very fast.

OldGoodNewBad 5y ago

I think a lot of folks are going out of their way to misunderstand what happened. Yes there are other similar projects and containers. No, none come from a long established COMMUNITY RUN PROJECT. This is something akin to the difference between VirtualBox and OpenBSD’s vmd. Ones a product with a “free” tier, the other is a community project.

dzonga 5y ago

I understand, it's cool to do content marketing. but folks proof-read your articles. Firecracker was created by AWS and rightly states so on the page.

stefanha 5y ago

For an even more lightweight approach to running containers in VMs see: https://github.com/containers/krunvm https://github.com/containers/krunvm It's powered by https://github.com/containers/libkrun https://github.com/containers/libkrun.