8 ms·

Firecracker: Start a VM in less than a second

381 points by ArmandGrillet 6y ago

majkinetor 6y ago
> Firecracker can start a VM in less than a second! So this makes it comparable to containers when speed in question. Anybody using this instead typical VM's in production (and not being Amazon) ?
- tptacek 6y ago
  Yep. At Fly.io, we run customer containers on our own hardware around the world --- the normal workflow just pushes Docker containers to our registry --- by converting them into root filesystems and running them as Firecracker instances.
  - alexchamberlain 6y ago
    What exactly is involved in converting them to root filesystems?
    - iampims 6y ago
      I’d love to know more about this as well. That’s the part usually nobody talks about.
    - e12e 6y ago
      Have a look at: https://stackoverflow.com/questions/23436613/how-can-i-convert-a-docker-image-into-a-vagrant-virtualbox-box#42845276 https://stackoverflow.com/questions/23436613/how-can-i-conve... Or "slim": https://news.ycombinator.com/item?id=20182141 https://news.ycombinator.com/item?id=20182141 Typically, a docker filesystem doesn't include a proper unit (docker will inject "tiny init") - so you would probably have to add a kernel, and init, somehow. I think I'd prefer to just create a vm, rather than re-use the pre-built docker.
      - e12e 6y ago
        Ed: a proper "init" not "unit"
    - CodesInChaos 6y ago
      The linked article contains a script building the file system. It creates a block device using `qemu-img`, adds an empty filesystem using `mkfs.ext4` and then simply mounts it and copies in the files. The previous posts cover this topic in more detail: Day 43: Building VM images - https://jvns.ca/blog/2021/01/21/day-43--building-vm-images/ https://jvns.ca/blog/2021/01/21/day-43--building-vm-images/ Day 44: Building my VMs with Docker - https://jvns.ca/blog/2021/01/22/day-44--got-some-vms-to-start-in-firecracker/ https://jvns.ca/blog/2021/01/22/day-44--got-some-vms-to-star...
    - nicwolff 6y ago
      It may not be what fly.io does but there's a walkthrough of the basic idea here https://iximiuz.com/en/posts/from-docker-container-to-bootable-linux-disk-image/ https://iximiuz.com/en/posts/from-docker-container-to-bootab...
    - tptacek 6y ago
      It's not a big secret or anything but it's changed recently and Jerome would do a better job of describing it than I would; apart from the filesystem stuff, we have an init we wrote in Rust that does a bunch of the lifting.
    - jeromegn 6y ago
      We use containerd with the devmapper snapshotter! Works nicely. We create a hard link to the resulting device for the root drive inside firecracker.
  - freedomben 6y ago
    May I ask why you use firecracker, especially when you already have Docker images in your registry? Do you need root and/or a kernel in your containers for your application?
    - tptacek 6y ago
      Because of multitenancy. It isn't safe to run jobs from different customers alongside each other in namespaced OS containers; instead, we give every customer instance its own VM and its own OS. This is the same model that Lambda and Fargate use (of course, that's what Amazon implemented Firecracker for). I linked to a blog post we wrote about the rationales here, upthread.
      - freedomben 6y ago
        Cool, thanks much. I'll find the link.
freedomben 6y ago
Another incredible blog post from Julia Evans. Julia if you're reading, I'm a big fan. One request: could you add the date/time to the post, preferably near the title? I do see a `<time>` tag in the HTML, but it doesn't render any human readable text. The datetime is also part of the URL which is "good enough," but it always takes me a minute to remember that and Ctrl+F won't find it. I'm a little ADD about knowing when things were published, so far from an average person. If you like it the way it is, then don't worry about me. I just wanted to throw it out there.
- chrisweekly 6y ago
  +1 you're not alone; seconding the request -- and esp the kudos/praise for Julia, one of my "secret heroes"
- eyelidlessness 6y ago
  On reading this, I went to see if the time tag renders in reader mode, assuming that’s part of why it’s there, but at least on my mobile Safari it surprisingly does not. I’ll second the request. And also second that I’m a big fan of Julia Evans.
- forix 6y ago
  +1 on the ADD about timestamps :) and a +1 for Julia. Following her on Twitter and she's awesome!
- nix23 6y ago
  >Another incredible blog post from Julia Evans. Same same, everything grounded and then dug deep.
- nccyr 6y ago
  Totally agree but in the meantime, I just noticed that she has the dates on the main page: https://jvns.ca/ https://jvns.ca/
koala_man 6y ago
You can use QEmu snapshots to start a VM in under a second (but more than 125ms). I used this in 2009 for an IRC bot that safely evaluated arbitrary shell commands for demonstration purposes, and a fork thereof is still chugging along to this day. https://github.com/geirha/shbot https://github.com/geirha/shbot
- bonzini 6y ago
  QEMU can start a VM quite quickly: https://asciinema.org/a/jlTamarDTOVpO9wYAjXcnNZZt https://asciinema.org/a/jlTamarDTOVpO9wYAjXcnNZZt For a generic build like you'll find in a Linux distro, QEMU startup is slow mostly because it links to dozens of shared libraries (103 on Fedora 33).
tptacek 6y ago
Regarding performance, I wrote a bit about this a few months ago in a post comparing different workload isolation techniques: https://fly.io/blog/sandboxing-and-workload-isolation/ https://fly.io/blog/sandboxing-and-workload-isolation/ Two useful links from my Pinboard research for that post: A person at Red Hat optimizing QEMU boot time: http://oirase.annexia.org/tmp/paper.pdf http://oirase.annexia.org/tmp/paper.pdf An Intel deck talking about qemu-lite: http://events17.linuxfoundation.org/sites/events/files/slides/Light%20weight%20virtualization%20with%20QEMU%26KVM_0.pdf http://events17.linuxfoundation.org/sites/events/files/slide... (The other thing to follow up on if you're interested in the background on this stuff is kvmtool). In both cases, a big part of the answer seems to be eliminating BIOS overhead; getting rid of oproms appears to have been the single biggest win for Intel. But the Red Hat article also finds lots of overhead in QEMU itself, and both pieces talk about kernel config issues (for instance, scrubbing the kernel you boot of subsystems that have expensive initcalls). By comparison: Firecracker is purpose-built in Rust for this one task, provides no BIOS, and offers only network, block, keyboard, and serial device support --- with tiny drivers (the serial support is less than 300 lines of code).
- bogomipz 6y ago
  Regarding your first link I had questions about the following passages from it: >"People like to say “Docker isn’t a security boundary”, but that’s not so true anymore, though it once was." Could you explain why that wasn't true and why it is now? >"Systems security people spent almost a decade dunking on Docker because of all the gaps in this simplified container model. But nobody really runs containers like this anymore." Could you elaborate? People don't run container like what? It isn't clear from the preceding paragraph.
  - tptacek 6y ago
    Modern Docker running in default configurations is reasonably well locked down. I wouldn't use it for multitenant scenarios, though.
    - varun_chopra 6y ago
ignoramous 6y ago
The blog post doesn't mention it but Firecracker was originally based on crosvm [0] built at Google by the ChromeOS team for a WSL-like Linux sandbox on top of ChromeOS running debian-buster containers viz. penguin (afaik) in a gentoo-based VM viz. termina [1]. crosvm inturn is part of a much bigger crostini project [2], which I find to be super fascinating, as it supports UI workloads (over Wayland and X). If you're using ChromeOS to run any Linux app, you're using crostini which launches those apps in crosvm managed sandboxes (Container inside a VM) in seconds. In not so distant future, it looks like Android will sandbox platform workloads (running outside the Android framework?) managed by crosvm, to considerably improve security [3]. I looked for but couldn't find ChromeOS GCP instances. I mean, ChromeOS might be a great platform to run multi-tenant server workloads at this point. [0] https://archive.is/T1ZNJ https://archive.is/T1ZNJ [1] One can run custom Linux containers (other than the debian-based penguin) but not VMs (other than termina) at this point: https://chromeos.dev/en/linux/linux-on-chromeos-faq https://chromeos.dev/en/linux/linux-on-chromeos-faq [2] https://chromium.googlesource.com/chromiumos/docs/+/master/containers_and_vms.md https://chromium.googlesource.com/chromiumos/docs/+/master/c... [3] https://lwn.net/Articles/836693/ https://lwn.net/Articles/836693/
- antonvs 6y ago
  > ... launches those apps in crosvm sandboxes in seconds. Technically true, but the initial VM launch is not all that quick. E.g., on a Pixelbook i7 launching a terminal session without the VM started, it takes about 20 seconds for the VM itself to initiate, and another 30 seconds for things like volume mounting and starting the container running Debian, for a total of 50 seconds. Subsequent launches once the VM is started are much quicker, just a few seconds. > I looked for but couldn't find ChromeOS GCP instances You're looking for Container-Optimize OS: https://cloud.google.com/container-optimized-os/ https://cloud.google.com/container-optimized-os/ , which is Chromium OS based. It's the default OS for e.g. GKE nodes, so Google probably runs pretty large numbers of them.
- oauea 6y ago
  And for some reason Crostini still isn't available on the majority of Chromebooks.
jabl 6y ago
I read somewhere (?) that firecracker has poorer io performance than KVM/qemu. Is that still the case, or has it been fixed?
- riyakhanna1983 6y ago
  You can read about its IO performance against QEMU in their NSDI paper (table 8,9): https://www.usenix.org/system/files/nsdi20-paper-agache.pdf https://www.usenix.org/system/files/nsdi20-paper-agache.pdf Even though the startup times are fast, the IO performance is poor compared to native. This is primarily because of IO emulation. We've been working on a new hypervisor that can directly run isolated containers (no VMs). Email me if you are interested in learning more.
  - nhooyr 6y ago
    Hi can you post more info on this new hypervisor publicly? Who's involved?
    - riyakhanna1983 6y ago
      We will post more publicly soon.
- StreamBright 6y ago
  I am not so sure about that. Amazon uses Firecracker because of its performance and how lightweight it is. Qemu VMs used to be heavier and I think Qemu devs started a project to have a lightweight version like Firecracker recently.
daxfohl 6y ago
I just got this working in windows preview build too. If you enable KVM and rebuild the WSL2 kernel, then you can follow the Linux firecracker demo on github step by step and it works. I was able to launch 400 concurrent firecracker VMs on my laptop in 60 sec.
- someonewithpc 6y ago
  60s total or each? I haven't tried WSL2 but I had to help people with _really_ bad docker performance running in WSL (php page load took around 60s)
  - daxfohl 6y ago
    Total. And it was like the first 200 were in 15 to 20 sec.
syntaxing 6y ago
Are VM and containers different in this context? I’m confused where Firecracker would be better than Docker for this usage.
- StreamBright 6y ago
  Firecracker would be better from the security isolation point of view. It can use everything that Docker has to offer in the security domain and some more. https://github.com/firecracker-microvm/firecracker/blob/master/docs/jailer.md https://github.com/firecracker-microvm/firecracker/blob/mast...
FourSigma 6y ago
What is the difference between Firecracker vs LXC/LXD?
- e12e 6y ago
  Linux containers are containers, not VMs. They are more like docker (although, lxd/lxc typically are used more like jails/VMs - a "full" user land, rather than just an application binary, like with a docker container wrapping a service implemented in go). Technically, docker/lxc uses kernel namespaces to isolate a process tree - firecracker starts up a virtual machine.
- dharmab 6y ago
  When a VM context switch happens, the CPU uses extensions like Intel VMX to isolate the virtual machine code from the host code. Usually the hypervisor also forces a cache flush to mitigate CPU vulnerabilities as well.
- StreamBright 6y ago
  VMs vs containers. One uses KVM under the hood the other uses cgroups. Btw. you can run Firecracker VMs with container isolation on the top. https://github.com/firecracker-microvm/firecracker/blob/master/docs/jailer.md https://github.com/firecracker-microvm/firecracker/blob/mast...
ahachete 6y ago
I wrote a similar post some weeks ago, going to similar deepths to script launching a VM with Firecracker. My main goal then was to provide the necessary automation to help using cloud images for the VMs, so you can easily leverage a wide array of existing images. Most of the credit is due to cloud-init, which helps automate instance configuration after boot. "Automation to run VMs based on vanilla Cloud Images on Firecracker": https://ongres.com/blog/automation-to-run-vms-based-on-vanilla-cloud-images-on-firecracker/ https://ongres.com/blog/automation-to-run-vms-based-on-vanil...
m00dy 6y ago
Has anyone know that Firecracker can access GPU ?
- ptomato 6y ago
  not currently, though there's some people working on that. https://github.com/firecracker-microvm/firecracker/issues/1179 https://github.com/firecracker-microvm/firecracker/issues/11...
williesleg 6y ago
Great! Now we can run all the shit code from the H1b's that need constant reboots! Yay innovation!
jzer0cool 6y ago
Surprise I have not yet encountered firecracker until now with your post. Thanks for sharing.
movedx 6y ago
This is where I believe we should have invested our efforts as an industry: making VMs faster, not designing and adopting containers. If we can get VMs to launch even remotely as fast as containers, that possibly means the end of containers due to VMs offering far superior isolation.
- tptacek 6y ago
  I probably agree about container runtimes, but containers as a specification for a preconfigured unit of compute have lots of value outside the runtimes. We don't use namespaced containers at all at Fly.io, and run everything inside of jailed Firecrackers. But we get enormous value out of container tooling, and the most common and simplest way of deploying something to Fly is simply to have our tooling push a Docker image over to Fly's repo. I think containers have been largely a force for good. They're not as much a force for good as some might claim, but that's a different argument than that they were a waste of time.
dilyevsky 6y ago
Anyone evaluated this for io (disk) performance? Any different from kvm?
jobe_br 6y ago
Re: if there’s something similar for macOS, it’s built-in. The new Virtualization framework in macOS 11 (Big Sur) is very very quick. User space with support for Virtio spec.
deleted 6y ago
[deleted]
amarti 6y ago
Great post! :D Actually, Firecracker is so cool that we've used it to develop our own native alternative to Kubernetes based on running application containers as microVMs... https://opennebula.io/opennebula-kubernetes-comparing-two-container-orchestration-models/ https://opennebula.io/opennebula-kubernetes-comparing-two-co...

381 points by ArmandGrillet 6y ago

> Firecracker can start a VM in less than a second! So this makes it comparable to containers when speed in question. Anybody using this instead typical VM's in production (and not being Amazon) ?

tptacek 6y ago
Yep. At Fly.io, we run customer containers on our own hardware around the world --- the normal workflow just pushes Docker containers to our registry --- by converting them into root filesystems and running them as Firecracker instances.
- alexchamberlain 6y ago
  What exactly is involved in converting them to root filesystems?
  - iampims 6y ago
    I’d love to know more about this as well. That’s the part usually nobody talks about.
  - e12e 6y ago
    Have a look at: https://stackoverflow.com/questions/23436613/how-can-i-convert-a-docker-image-into-a-vagrant-virtualbox-box#42845276 https://stackoverflow.com/questions/23436613/how-can-i-conve... Or "slim": https://news.ycombinator.com/item?id=20182141 https://news.ycombinator.com/item?id=20182141 Typically, a docker filesystem doesn't include a proper unit (docker will inject "tiny init") - so you would probably have to add a kernel, and init, somehow. I think I'd prefer to just create a vm, rather than re-use the pre-built docker.
    - e12e 6y ago
      Ed: a proper "init" not "unit"
  - CodesInChaos 6y ago
    The linked article contains a script building the file system. It creates a block device using `qemu-img`, adds an empty filesystem using `mkfs.ext4` and then simply mounts it and copies in the files. The previous posts cover this topic in more detail: Day 43: Building VM images - https://jvns.ca/blog/2021/01/21/day-43--building-vm-images/ https://jvns.ca/blog/2021/01/21/day-43--building-vm-images/ Day 44: Building my VMs with Docker - https://jvns.ca/blog/2021/01/22/day-44--got-some-vms-to-start-in-firecracker/ https://jvns.ca/blog/2021/01/22/day-44--got-some-vms-to-star...
  - nicwolff 6y ago
    It may not be what fly.io does but there's a walkthrough of the basic idea here https://iximiuz.com/en/posts/from-docker-container-to-bootable-linux-disk-image/ https://iximiuz.com/en/posts/from-docker-container-to-bootab...
  - tptacek 6y ago
    It's not a big secret or anything but it's changed recently and Jerome would do a better job of describing it than I would; apart from the filesystem stuff, we have an init we wrote in Rust that does a bunch of the lifting.
  - jeromegn 6y ago
    We use containerd with the devmapper snapshotter! Works nicely. We create a hard link to the resulting device for the root drive inside firecracker.
- freedomben 6y ago
  May I ask why you use firecracker, especially when you already have Docker images in your registry? Do you need root and/or a kernel in your containers for your application?
  - tptacek 6y ago
    Because of multitenancy. It isn't safe to run jobs from different customers alongside each other in namespaced OS containers; instead, we give every customer instance its own VM and its own OS. This is the same model that Lambda and Fargate use (of course, that's what Amazon implemented Firecracker for). I linked to a blog post we wrote about the rationales here, upthread.
    - freedomben 6y ago
      Cool, thanks much. I'll find the link.

freedomben 6y ago

Another incredible blog post from Julia Evans. Julia if you're reading, I'm a big fan. One request: could you add the date/time to the post, preferably near the title? I do see a `<time>` tag in the HTML, but it doesn't render any human readable text. The datetime is also part of the URL which is "good enough," but it always takes me a minute to remember that and Ctrl+F won't find it. I'm a little ADD about knowing when things were published, so far from an average person. If you like it the way it is, then don't worry about me. I just wanted to throw it out there.

chrisweekly 6y ago
+1 you're not alone; seconding the request -- and esp the kudos/praise for Julia, one of my "secret heroes"
eyelidlessness 6y ago
On reading this, I went to see if the time tag renders in reader mode, assuming that’s part of why it’s there, but at least on my mobile Safari it surprisingly does not. I’ll second the request. And also second that I’m a big fan of Julia Evans.
forix 6y ago
+1 on the ADD about timestamps :) and a +1 for Julia. Following her on Twitter and she's awesome!
nix23 6y ago
>Another incredible blog post from Julia Evans. Same same, everything grounded and then dug deep.
nccyr 6y ago
Totally agree but in the meantime, I just noticed that she has the dates on the main page: https://jvns.ca/ https://jvns.ca/

koala_man 6y ago

You can use QEmu snapshots to start a VM in under a second (but more than 125ms). I used this in 2009 for an IRC bot that safely evaluated arbitrary shell commands for demonstration purposes, and a fork thereof is still chugging along to this day. https://github.com/geirha/shbot https://github.com/geirha/shbot

bonzini 6y ago
QEMU can start a VM quite quickly: https://asciinema.org/a/jlTamarDTOVpO9wYAjXcnNZZt https://asciinema.org/a/jlTamarDTOVpO9wYAjXcnNZZt For a generic build like you'll find in a Linux distro, QEMU startup is slow mostly because it links to dozens of shared libraries (103 on Fedora 33).

tptacek 6y ago

Regarding performance, I wrote a bit about this a few months ago in a post comparing different workload isolation techniques: https://fly.io/blog/sandboxing-and-workload-isolation/ https://fly.io/blog/sandboxing-and-workload-isolation/ Two useful links from my Pinboard research for that post: A person at Red Hat optimizing QEMU boot time: http://oirase.annexia.org/tmp/paper.pdf http://oirase.annexia.org/tmp/paper.pdf An Intel deck talking about qemu-lite: http://events17.linuxfoundation.org/sites/events/files/slides/Light%20weight%20virtualization%20with%20QEMU%26KVM_0.pdf http://events17.linuxfoundation.org/sites/events/files/slide... (The other thing to follow up on if you're interested in the background on this stuff is kvmtool). In both cases, a big part of the answer seems to be eliminating BIOS overhead; getting rid of oproms appears to have been the single biggest win for Intel. But the Red Hat article also finds lots of overhead in QEMU itself, and both pieces talk about kernel config issues (for instance, scrubbing the kernel you boot of subsystems that have expensive initcalls). By comparison: Firecracker is purpose-built in Rust for this one task, provides no BIOS, and offers only network, block, keyboard, and serial device support --- with tiny drivers (the serial support is less than 300 lines of code).

bogomipz 6y ago
Regarding your first link I had questions about the following passages from it: >"People like to say “Docker isn’t a security boundary”, but that’s not so true anymore, though it once was." Could you explain why that wasn't true and why it is now? >"Systems security people spent almost a decade dunking on Docker because of all the gaps in this simplified container model. But nobody really runs containers like this anymore." Could you elaborate? People don't run container like what? It isn't clear from the preceding paragraph.
- tptacek 6y ago
  Modern Docker running in default configurations is reasonably well locked down. I wouldn't use it for multitenant scenarios, though.
  - varun_chopra 6y ago

ignoramous 6y ago

The blog post doesn't mention it but Firecracker was originally based on crosvm [0] built at Google by the ChromeOS team for a WSL-like Linux sandbox on top of ChromeOS running debian-buster containers viz. penguin (afaik) in a gentoo-based VM viz. termina [1]. crosvm inturn is part of a much bigger crostini project [2], which I find to be super fascinating, as it supports UI workloads (over Wayland and X). If you're using ChromeOS to run any Linux app, you're using crostini which launches those apps in crosvm managed sandboxes (Container inside a VM) in seconds. In not so distant future, it looks like Android will sandbox platform workloads (running outside the Android framework?) managed by crosvm, to considerably improve security [3]. I looked for but couldn't find ChromeOS GCP instances. I mean, ChromeOS might be a great platform to run multi-tenant server workloads at this point. [0] https://archive.is/T1ZNJ https://archive.is/T1ZNJ [1] One can run custom Linux containers (other than the debian-based penguin) but not VMs (other than termina) at this point: https://chromeos.dev/en/linux/linux-on-chromeos-faq https://chromeos.dev/en/linux/linux-on-chromeos-faq [2] https://chromium.googlesource.com/chromiumos/docs/+/master/containers_and_vms.md https://chromium.googlesource.com/chromiumos/docs/+/master/c... [3] https://lwn.net/Articles/836693/ https://lwn.net/Articles/836693/

antonvs 6y ago
> ... launches those apps in crosvm sandboxes in seconds. Technically true, but the initial VM launch is not all that quick. E.g., on a Pixelbook i7 launching a terminal session without the VM started, it takes about 20 seconds for the VM itself to initiate, and another 30 seconds for things like volume mounting and starting the container running Debian, for a total of 50 seconds. Subsequent launches once the VM is started are much quicker, just a few seconds. > I looked for but couldn't find ChromeOS GCP instances You're looking for Container-Optimize OS: https://cloud.google.com/container-optimized-os/ https://cloud.google.com/container-optimized-os/ , which is Chromium OS based. It's the default OS for e.g. GKE nodes, so Google probably runs pretty large numbers of them.
oauea 6y ago
And for some reason Crostini still isn't available on the majority of Chromebooks.

jabl 6y ago

I read somewhere (?) that firecracker has poorer io performance than KVM/qemu. Is that still the case, or has it been fixed?

riyakhanna1983 6y ago
You can read about its IO performance against QEMU in their NSDI paper (table 8,9): https://www.usenix.org/system/files/nsdi20-paper-agache.pdf https://www.usenix.org/system/files/nsdi20-paper-agache.pdf Even though the startup times are fast, the IO performance is poor compared to native. This is primarily because of IO emulation. We've been working on a new hypervisor that can directly run isolated containers (no VMs). Email me if you are interested in learning more.
- nhooyr 6y ago
  Hi can you post more info on this new hypervisor publicly? Who's involved?
  - riyakhanna1983 6y ago
    We will post more publicly soon.
StreamBright 6y ago
I am not so sure about that. Amazon uses Firecracker because of its performance and how lightweight it is. Qemu VMs used to be heavier and I think Qemu devs started a project to have a lightweight version like Firecracker recently.

daxfohl 6y ago

I just got this working in windows preview build too. If you enable KVM and rebuild the WSL2 kernel, then you can follow the Linux firecracker demo on github step by step and it works. I was able to launch 400 concurrent firecracker VMs on my laptop in 60 sec.

someonewithpc 6y ago
60s total or each? I haven't tried WSL2 but I had to help people with _really_ bad docker performance running in WSL (php page load took around 60s)
- daxfohl 6y ago
  Total. And it was like the first 200 were in 15 to 20 sec.

syntaxing 6y ago

Are VM and containers different in this context? I’m confused where Firecracker would be better than Docker for this usage.

StreamBright 6y ago
Firecracker would be better from the security isolation point of view. It can use everything that Docker has to offer in the security domain and some more. https://github.com/firecracker-microvm/firecracker/blob/master/docs/jailer.md https://github.com/firecracker-microvm/firecracker/blob/mast...

FourSigma 6y ago

What is the difference between Firecracker vs LXC/LXD?

e12e 6y ago
Linux containers are containers, not VMs. They are more like docker (although, lxd/lxc typically are used more like jails/VMs - a "full" user land, rather than just an application binary, like with a docker container wrapping a service implemented in go). Technically, docker/lxc uses kernel namespaces to isolate a process tree - firecracker starts up a virtual machine.
dharmab 6y ago
When a VM context switch happens, the CPU uses extensions like Intel VMX to isolate the virtual machine code from the host code. Usually the hypervisor also forces a cache flush to mitigate CPU vulnerabilities as well.
StreamBright 6y ago
VMs vs containers. One uses KVM under the hood the other uses cgroups. Btw. you can run Firecracker VMs with container isolation on the top. https://github.com/firecracker-microvm/firecracker/blob/master/docs/jailer.md https://github.com/firecracker-microvm/firecracker/blob/mast...

ahachete 6y ago

I wrote a similar post some weeks ago, going to similar deepths to script launching a VM with Firecracker. My main goal then was to provide the necessary automation to help using cloud images for the VMs, so you can easily leverage a wide array of existing images. Most of the credit is due to cloud-init, which helps automate instance configuration after boot. "Automation to run VMs based on vanilla Cloud Images on Firecracker": https://ongres.com/blog/automation-to-run-vms-based-on-vanilla-cloud-images-on-firecracker/ https://ongres.com/blog/automation-to-run-vms-based-on-vanil...

m00dy 6y ago

Has anyone know that Firecracker can access GPU ?

ptomato 6y ago
not currently, though there's some people working on that. https://github.com/firecracker-microvm/firecracker/issues/1179 https://github.com/firecracker-microvm/firecracker/issues/11...

williesleg 6y ago

Great! Now we can run all the shit code from the H1b's that need constant reboots! Yay innovation!

jzer0cool 6y ago

Surprise I have not yet encountered firecracker until now with your post. Thanks for sharing.

movedx 6y ago

This is where I believe we should have invested our efforts as an industry: making VMs faster, not designing and adopting containers. If we can get VMs to launch even remotely as fast as containers, that possibly means the end of containers due to VMs offering far superior isolation.

tptacek 6y ago
I probably agree about container runtimes, but containers as a specification for a preconfigured unit of compute have lots of value outside the runtimes. We don't use namespaced containers at all at Fly.io, and run everything inside of jailed Firecrackers. But we get enormous value out of container tooling, and the most common and simplest way of deploying something to Fly is simply to have our tooling push a Docker image over to Fly's repo. I think containers have been largely a force for good. They're not as much a force for good as some might claim, but that's a different argument than that they were a waste of time.

dilyevsky 6y ago

Anyone evaluated this for io (disk) performance? Any different from kvm?

jobe_br 6y ago

Re: if there’s something similar for macOS, it’s built-in. The new Virtualization framework in macOS 11 (Big Sur) is very very quick. User space with support for Virtio spec.

deleted 6y ago

[deleted]

amarti 6y ago

Great post! :D Actually, Firecracker is so cool that we've used it to develop our own native alternative to Kubernetes based on running application containers as microVMs... https://opennebula.io/opennebula-kubernetes-comparing-two-container-orchestration-models/ https://opennebula.io/opennebula-kubernetes-comparing-two-co...