11 ms·

FreeBSD on Firecracker

346 points by jmmv 3y ago

laurencerowe 3y ago
It's a shame neither AWS nor macOS on ARM support nested virtualization. It would would make it far easier to develop and deploy Firecracker based tech.
- saurik 3y ago
  FWIW, AWS a1.metal instances are pretty small and thereby reasonably cost effective for working with virtualization tech.
  - Rapzid 3y ago
    Their metal offerings are puny in general though(as in, not a ton of options).
- znpy 3y ago
  Afaik you can do virtualisation on the .metal variants. Actually you can do virtualisation on any instance type afaik, but only with .metal instances you can use hardware acceleration.
getcrunk 3y ago
So firecracker vs v8 isolates if only doing js or wasm?
tedunangst 3y ago
It's funny how many one second pauses turn out to be less than necessary. How many sysadmins took meaningful action because the system paused when they had an invalid machine uuid?
- cperciva 3y ago
  Probably a significant proportion of the sysadmins who experienced that one-second pause. The "print a message telling the user that we're rebooting, then wait a second to let them read the console before we go ahead and reboot", on the other hand...
  - tedunangst 3y ago
    Maybe I'm just different. I've watched a great many openbsd boot sequences, which tend to have a great many pauses, and I've never paid any special attention to the lines that come before pauses vs lines that come after pauses.
    - cperciva 3y ago
      I suspect that FreeBSD has fewer pauses than OpenBSD... especially after the work I've done over the past few years to speed it up. If anyone in the OpenBSD world is interested in speeding up your boot process I'd be happy to share tips and code. It's a bit daunting to start with but with some good tools it becomes a lot of fun.
      - pseudostem 3y ago
        Thank you! That would be me. I am clueless on amd64 on how to speed up the boot process and maybe change a few fonts during boot. I understand the reasons for no "how-tos". But sometimes they make sense for people like me. I wouldn't mind delving a bit deeper given some direction.
        cperciva 3y ago
        The first thing you'll want to do is port my TSLOG code to the OpenBSD kernel and start instrumenting things there. Send me an email, there's too much detail to get into for an HN comment.
  - slim 3y ago
    if the machine boots in 20ms, I think that message is actually useful, because something would reboot your machine and you'd think you got logged out because you blinked
garganzol 3y ago
I never really realized that Firecracker VM is a full-blown machine and not just some sort of a Linux container tech. At first, it may sound like an ineffective approach, but if you take a closer look on a real-world usage example such as fly.io, you will be surprised: micro-VMs are very small and capable.
- NovemberWhiskey 3y ago
  There's no way an "enterprise grade" cloud vendor like AWS would allow co-tenancy of containers (for ECS, Lambda etc) from different customers within a single VM - it's the reason Firecracker exists.
  - deleted 3y ago
    [deleted]
  - monocasa 3y ago
    Does google? I know they use gvisor in production, which is ultimately enforced by a normal kernel (with a ton of sandboxing on top of it).
    - eddythompson80 3y ago
      Google is moving away from gvisor as well. The "process sandbox" wars are over. Everybody lost, hypervisors won. That's it. It feels incredibly wasteful after all. Hypervisors don't share mm, scheduler, etc. It's a lot of wasted resources. Google came in with gvisor at the last minute to try to say "no, sandboxes aren't dead. Look at our approach with gvisor". They lost too and are now moving away from it.
      - lima 3y ago
        > Google is moving away from gvisor as well. I've been wondering about this - are they really?
        3y ago
mnsc 3y ago
Here's the recent BSDCan talk from Colin that was posted a couple of days ago. https://youtu.be/MT3cdeuRTzs?si=l6baNriUjcvy0ZOE https://youtu.be/MT3cdeuRTzs?si=l6baNriUjcvy0ZOE
- cperciva 3y ago
  FWIW, this is basically the same material -- after my BSDCan talk the FreeBSD Journal said "hey, that was a great talk, can you turn it into an article for us", and after the FreeBSD Journal article was published ;login: asked if they could republish it.
doublerabbit 3y ago
I'm not wanting to sound snoody. What use-cases do firecracker instances and the likes chime? I use FreeBSD for everything from my colocated servers, to my own PC. By no means am I developer; seasoned Unix Admin at best. Bare-metal forever but welcome to the future. Especially anything that contributes to the OS. However I hear buzz words like Lambda and Firecracker and really have no idea where the usage is. I get docker, containers, barely understand k8s but why do you need to spin up a VM only to tear it down compared to where you could just spin up a VM and use it when you really need to. Always there, always when. Is it purely a cloud experience, cost saving exercise?
- artificial 3y ago
  FaaS, function as a service. Depending on how software is packaged and the expectations the richness a VM, like Firecraker, provides may be useful. Many of these tradeoffs are for velocity, I can run X easily on Y.
- Datagenerator 3y ago
  IoT devices can execute short lived actions by calling remote Functions. The provider wants complete isolation and wipes these micro VMs after every few seconds and let's the user pay for use. The response from these can be anything, voice, data or API responses.
- turtlebits 3y ago
  Just about every single company can benefit from scaling as traffic is never consistent 24/7. Most don't bother as the effort outweighs the savings, but the potential is there. Things like lambda and firecracker make it much easier.
- lproven 3y ago
  ... "snooty"? https://www.merriam-webster.com/dictionary/snooty https://www.merriam-webster.com/dictionary/snooty
  - zaps 3y ago
    snotty
andrewstuart 3y ago
qemu has microvm, inspired by firecracker https://qemu.readthedocs.io/en/latest/system/i386/microvm.html https://qemu.readthedocs.io/en/latest/system/i386/microvm.ht...
- bonzini 3y ago
  I wonder how many of these workarounds are needed with QEMU! Some of course will be needed because they are fixes for bugs in FreeBSD.
ruslan 3y ago
> on a virtual machine with 1 CPU and 128 MB of RAM, the FreeBSD kernel can boot in under 20 ms; Oh, my... how could I achieve the same on real hardware without VMs ? ;)
- dikei 3y ago
  Kernel boot is plenty fast with real hardware, usually under 1 seconds. It's everything else that's slow. For example, this is my machine Startup finished in 14.552s (firmware) + 2.885s (loader) + 741ms (kernel) + 23.116s (initrd) + 11.191s (userspace) = 52.488s
  - yencabulator 3y ago
    The loader+initrd+userspace time there sounds unreasonable. I've had libvirt bog standard qemu-kvm (not a microvm) creating a new Ubuntu VM from a disk image & booting to a login prompt in under 10 seconds for more than a decade. This is without fiddling with virtio, doing hardware scans for PCI, VGA, SATA and such, and booting via Grub (your "loader"). Those should be pretty comparable!
    - dikei 3y ago
      Ah, yes, you can ignore the loader + initrd. I'm multi-boot so the loader has 2 seconds timeout, and initrd is waiting for me to input my LUKS password.
nocarrier 3y ago
Once Colin's patches land on FreeBSD and Firecracker, the total boot time for the kernel is under 20ms. Absolutely incredible times that we live in.
- arp242 3y ago
  How does this compare to Linux on Firecracker? I can find some numbers with a basic internet search, but I'm not sure if these numbers are comparable for various reasons (they're a few years old, it's unclear if they use the same method to measure boot times, or even have the same definition of "boot time").
Havoc 3y ago
Toyed around with firecracker a bit. Does what it promises on boot times, but still a pretty gnarly experience. e.g. After doing a victory dance for getting it to boot I was rather deflated to find out that getting networking takes another lengthy tutorial
- bborud 3y ago
  I think there is definitively room for someone to add a lot of value to this by creating some automation tools. It would be really nice to be able to download a single binary, fire it up, have both a web interface and an API available, be able to configure it quickly, have it download whatever it needs for you etc.
ryanrussell 3y ago
Firecracker is amazing, but has a lot of edge cases that need documentation. A huge thank you to Colin Percival for sharing this. Particularly love the "Once the low-hanging fruit was out of the way" line... which to Colin means custom bus_dma patch(es). Now anyone can now enjoy for free: "with 1 CPU and 128 MB of RAM, the FreeBSD kernel can boot in under 20 ms" If you're used to devops with k8s clusters or lots of docker, this is absolutely amazing.

346 points by jmmv 3y ago

It's a shame neither AWS nor macOS on ARM support nested virtualization. It would would make it far easier to develop and deploy Firecracker based tech.

saurik 3y ago
FWIW, AWS a1.metal instances are pretty small and thereby reasonably cost effective for working with virtualization tech.
- Rapzid 3y ago
  Their metal offerings are puny in general though(as in, not a ton of options).
znpy 3y ago
Afaik you can do virtualisation on the .metal variants. Actually you can do virtualisation on any instance type afaik, but only with .metal instances you can use hardware acceleration.

getcrunk 3y ago

So firecracker vs v8 isolates if only doing js or wasm?

tedunangst 3y ago

It's funny how many one second pauses turn out to be less than necessary. How many sysadmins took meaningful action because the system paused when they had an invalid machine uuid?

cperciva 3y ago
Probably a significant proportion of the sysadmins who experienced that one-second pause. The "print a message telling the user that we're rebooting, then wait a second to let them read the console before we go ahead and reboot", on the other hand...
- tedunangst 3y ago
  Maybe I'm just different. I've watched a great many openbsd boot sequences, which tend to have a great many pauses, and I've never paid any special attention to the lines that come before pauses vs lines that come after pauses.
  - cperciva 3y ago
    I suspect that FreeBSD has fewer pauses than OpenBSD... especially after the work I've done over the past few years to speed it up. If anyone in the OpenBSD world is interested in speeding up your boot process I'd be happy to share tips and code. It's a bit daunting to start with but with some good tools it becomes a lot of fun.
    - pseudostem 3y ago
      Thank you! That would be me. I am clueless on amd64 on how to speed up the boot process and maybe change a few fonts during boot. I understand the reasons for no "how-tos". But sometimes they make sense for people like me. I wouldn't mind delving a bit deeper given some direction.
      - cperciva 3y ago
        The first thing you'll want to do is port my TSLOG code to the OpenBSD kernel and start instrumenting things there. Send me an email, there's too much detail to get into for an HN comment.
- slim 3y ago
  if the machine boots in 20ms, I think that message is actually useful, because something would reboot your machine and you'd think you got logged out because you blinked

garganzol 3y ago

I never really realized that Firecracker VM is a full-blown machine and not just some sort of a Linux container tech. At first, it may sound like an ineffective approach, but if you take a closer look on a real-world usage example such as fly.io, you will be surprised: micro-VMs are very small and capable.

NovemberWhiskey 3y ago
There's no way an "enterprise grade" cloud vendor like AWS would allow co-tenancy of containers (for ECS, Lambda etc) from different customers within a single VM - it's the reason Firecracker exists.
- deleted 3y ago
  [deleted]
- monocasa 3y ago
  Does google? I know they use gvisor in production, which is ultimately enforced by a normal kernel (with a ton of sandboxing on top of it).
  - eddythompson80 3y ago
    Google is moving away from gvisor as well. The "process sandbox" wars are over. Everybody lost, hypervisors won. That's it. It feels incredibly wasteful after all. Hypervisors don't share mm, scheduler, etc. It's a lot of wasted resources. Google came in with gvisor at the last minute to try to say "no, sandboxes aren't dead. Look at our approach with gvisor". They lost too and are now moving away from it.
    - lima 3y ago
      > Google is moving away from gvisor as well. I've been wondering about this - are they really?
      - 3y ago

mnsc 3y ago

Here's the recent BSDCan talk from Colin that was posted a couple of days ago. https://youtu.be/MT3cdeuRTzs?si=l6baNriUjcvy0ZOE https://youtu.be/MT3cdeuRTzs?si=l6baNriUjcvy0ZOE

cperciva 3y ago
FWIW, this is basically the same material -- after my BSDCan talk the FreeBSD Journal said "hey, that was a great talk, can you turn it into an article for us", and after the FreeBSD Journal article was published ;login: asked if they could republish it.

doublerabbit 3y ago

I'm not wanting to sound snoody. What use-cases do firecracker instances and the likes chime? I use FreeBSD for everything from my colocated servers, to my own PC. By no means am I developer; seasoned Unix Admin at best. Bare-metal forever but welcome to the future. Especially anything that contributes to the OS. However I hear buzz words like Lambda and Firecracker and really have no idea where the usage is. I get docker, containers, barely understand k8s but why do you need to spin up a VM only to tear it down compared to where you could just spin up a VM and use it when you really need to. Always there, always when. Is it purely a cloud experience, cost saving exercise?

artificial 3y ago
FaaS, function as a service. Depending on how software is packaged and the expectations the richness a VM, like Firecraker, provides may be useful. Many of these tradeoffs are for velocity, I can run X easily on Y.
Datagenerator 3y ago
IoT devices can execute short lived actions by calling remote Functions. The provider wants complete isolation and wipes these micro VMs after every few seconds and let's the user pay for use. The response from these can be anything, voice, data or API responses.
turtlebits 3y ago
Just about every single company can benefit from scaling as traffic is never consistent 24/7. Most don't bother as the effort outweighs the savings, but the potential is there. Things like lambda and firecracker make it much easier.
lproven 3y ago
... "snooty"? https://www.merriam-webster.com/dictionary/snooty https://www.merriam-webster.com/dictionary/snooty
- zaps 3y ago
  snotty

andrewstuart 3y ago

qemu has microvm, inspired by firecracker https://qemu.readthedocs.io/en/latest/system/i386/microvm.html https://qemu.readthedocs.io/en/latest/system/i386/microvm.ht...

bonzini 3y ago
I wonder how many of these workarounds are needed with QEMU! Some of course will be needed because they are fixes for bugs in FreeBSD.

ruslan 3y ago

> on a virtual machine with 1 CPU and 128 MB of RAM, the FreeBSD kernel can boot in under 20 ms; Oh, my... how could I achieve the same on real hardware without VMs ? ;)

dikei 3y ago
Kernel boot is plenty fast with real hardware, usually under 1 seconds. It's everything else that's slow. For example, this is my machine Startup finished in 14.552s (firmware) + 2.885s (loader) + 741ms (kernel) + 23.116s (initrd) + 11.191s (userspace) = 52.488s
- yencabulator 3y ago
  The loader+initrd+userspace time there sounds unreasonable. I've had libvirt bog standard qemu-kvm (not a microvm) creating a new Ubuntu VM from a disk image & booting to a login prompt in under 10 seconds for more than a decade. This is without fiddling with virtio, doing hardware scans for PCI, VGA, SATA and such, and booting via Grub (your "loader"). Those should be pretty comparable!
  - dikei 3y ago
    Ah, yes, you can ignore the loader + initrd. I'm multi-boot so the loader has 2 seconds timeout, and initrd is waiting for me to input my LUKS password.

nocarrier 3y ago

Once Colin's patches land on FreeBSD and Firecracker, the total boot time for the kernel is under 20ms. Absolutely incredible times that we live in.

arp242 3y ago
How does this compare to Linux on Firecracker? I can find some numbers with a basic internet search, but I'm not sure if these numbers are comparable for various reasons (they're a few years old, it's unclear if they use the same method to measure boot times, or even have the same definition of "boot time").

Havoc 3y ago

Toyed around with firecracker a bit. Does what it promises on boot times, but still a pretty gnarly experience. e.g. After doing a victory dance for getting it to boot I was rather deflated to find out that getting networking takes another lengthy tutorial

bborud 3y ago
I think there is definitively room for someone to add a lot of value to this by creating some automation tools. It would be really nice to be able to download a single binary, fire it up, have both a web interface and an API available, be able to configure it quickly, have it download whatever it needs for you etc.

ryanrussell 3y ago

Firecracker is amazing, but has a lot of edge cases that need documentation. A huge thank you to Colin Percival for sharing this. Particularly love the "Once the low-hanging fruit was out of the way" line... which to Colin means custom bus_dma patch(es). Now anyone can now enjoy for free: "with 1 CPU and 128 MB of RAM, the FreeBSD kernel can boot in under 20 ms" If you're used to devops with k8s clusters or lots of docker, this is absolutely amazing.