11 ms·

The FreeBSD/Firecracker Platform

227 points by cperciva 4y ago

anotherhue 4y ago
Incredible. Maybe we can use this to shoe horn in docker support without the long lived VM
- nix23 4y ago
  https://bastillebsd.org/ https://bastillebsd.org/
agapon 4y ago
Nice development. Thank you, Colin!
rwmj 4y ago
Is it better / worse (in terms of say performance or security) than Bhyve?
- eyberg 4y ago
  Firecracker trades run-time performance in favor of faster boot-time latency. It is something that is being addressed but afaik still exists. This of course doesn't mean anything if you want firecracker to boot a multi-gig JVM installation (the larger the filesystem generally, expect a longer boot time). Security wise there are some minor differences. For instance bhyve supports virtio-rng but firecracker doesn't want to: https://github.com/firecracker-microvm/firecracker/issues/1671 https://github.com/firecracker-microvm/firecracker/issues/16... . I think if your app requires fast boot time and your app supports that than it's fine (so services that spin up and down on demand) but apps that daemonize for extended periods of time or take forever to initialize probably not a great fit.
tinco 4y ago
Nice! If I'd be looking into running FreeBSD it would be because I heard about its supposed security and I/O performance characteristics. Do those qualities hold up when run on firecracker this way? What applications would you choose FreeBSD for?
- LunaSea 4y ago
  Is I/O performance still better on FreeBSD after the last round of Linux improvements like io_uring and eBPF?
- dangerface 4y ago
  In my experience FreeBSD performance and security are great but not significantly better than linux. ZFS support on freebsd seems better than linux certainly its been supported a lot longer, bhyve works really well for virtualising freebsd and linux never tried it with windows. The biggest strength is the ports / package manager works better than anything I have tried on linux and it sticks to unix principles closer than linux. The biggest weakness for freebsd is docker and mono support is very poor. There is a desktop distro but in general any gui stuff linux beats freebsd hands down I only use freebsd for server stuff. I use it for production web servers, backup / storage servers, routers, virtualisation servers. I find its rock solid and more consistent than linux.
  - exogenousdata 4y ago
    Agreed that the lack of docket support is frustrating given how pervasive docker has become. However I’ve been using jails[0] in my home lab for 12 years and it’s a delight to use. [0] - https://docs.freebsd.org/en/books/handbook/jails/ https://docs.freebsd.org/en/books/handbook/jails/
    - soupbowl 4y ago
      I love jails, bastille as a jail manager/builder is really great. https://github.com/BastilleBSD/bastille https://github.com/BastilleBSD/bastille
qalmakka 4y ago
FreeBSD really deserves all the love it can get. It is an amazing system to work with and develop on, and I'll never stop evangelizing everyone about it.
- nurettin 4y ago
  In your opinion, what is the best thing that FreeBSD offers?
  - dangerface 4y ago
    Consistency, I have been using linux for 20+ years but if its not redhat or debian based I couldn't tell you how to start or stop services etc as its always different from the unix standard and every other linux. FreeBSD is unix it doesn't try to be that and something else at the same time.
    - nix23 4y ago
      >but if its not redhat or debian based I couldn't tell you how to start or stop services The same way as in freebsd "service blabla start" it's a compatibility hook.
      - dangerface 4y ago
        on fedora its systemctl instead of service because they are all systemd up, other distros might use something other than systemd and a compatibility hook is entirely dependant on the distro deciding it cares about backwards compatibility which isn't the case for fedora or debian at least in comparison to freebsd you can still build work the old school way.
        cesarb 4y ago
        > on fedora its systemctl instead of service because they are all systemd up Fedora (and its derivatives like RHEL/CentOS/etc) still has the "service" command, it forwards to the corresponding systemctl command. The same for Debian (the comments in Debian's "service" command even mention it's a modified version of the RHEL/Fedora one).
dkjaudyeqooe 4y ago
A welcome development. I wonder how quickly it boots.
- kevans91 4y ago
  Colin's done a lot of work in improving boot time on EC2 and in general, so I'm a little surprised he didn't actually mention the boot time in this article. In 2022Q1 he had boot time down to 8s (in EC2, IIRC); based on how cut down the FIRECRACKER config he added is and that there's no loader involvement, I don't think it'd be too surprising if he hits ~5-6s or less.
  - cperciva 4y ago
    It took me four days to get the blog post written and I only finished it last night because the toddler went to sleep early. I didn't want to drag it out any longer! But to answer the question: On small VMs, we can reach /sbin/init in under 50 ms and /etc/rc finishes running in under 500 ms. (Larger VMs take longer -- past a few GB of RAM, the time for initializing paging tables and launching APs starts to dominate.)
foomer 4y ago
That's a neat technical challenge solved, but I'm surprised by how much effort is being put in for these operating systems that basically no-one uses.
- dangerface 4y ago
  Lots of people and places use freebsd like netflix. I only see the popularity of freebsd increasing as linux moves more and more away from unix.
  - yamtaddle 4y ago
    I'm slowly switching over for server-side, and next time I try a free OS desktop it'll probably be freebsd. Only thing I miss is docker, which I mostly just use as a package manager and service manager so I don't have to screw with distro-specific trivia for those things, and that's less of a problem on FreeBSD anyway. I've long tolerated Linux but never loved it, and it's moving away from anything I'm interested in with each year, it seems (the competing mostly-bad image-based app distribution systems and the way Wayland's going are... yikes). Glad FreeBSD's there as an alternative.
    - tcmart14 4y ago
      On FreeBSD, I would recommend looking at jails and something like CBSD as a way to manage jails. https://github.com/cbsd/cbsd https://github.com/cbsd/cbsd
      - yamtaddle 4y ago
        Right, I'm familiar with them, but a huge part of the benefit of docker is that it's basically a cross-distro (and, to some degree, cross-platform) very up-to-date server-oriented package manager. There are some efforts to create something like that for Jails but AFAIK none of them have gotten very far. Docker's benefit is that ~everyone uses it so there's usually an official docker image for anything one might want to run, kept in step with releases, plus a substantial history of images for rollbacks and testing and such. It turns "how do I install and run this?" from a bunch of platform- and distro-specific instructions into `docker-compose up` for everything, with the bonus that the compose file also tells you exactly where data & config for the service live, so it's also concise, guaranteed-accurate, and extremely useful documentation. Old LTS Ubuntu? Can run the same version of a service, with the same command, as on bleeding-edge Arch. On Arch, exact same command. Debian, Red Hat? Same command. MacOS, if you've got docker set up, same command. Windows, ditto. The latter two, non-natively, sure, but it works. Learn docker, and you can use it everywhere, unlike becoming proficient with, say, dpkg/apt or portage or pacman. It can even kinda save you from having to worry about how a given platform's init system and process management works, at least for simple use cases. Again, if you're doing most or everything on FreeBSD, development and hosting both, those benefits mostly vanish. But the reality is many workplaces or even personal dev+hosting environments are mixed, and docker suits those better than anything else I'm aware of—but if FreeBSD is in the mix, it's the odd-man-out. It's good not (only) because of its kinda-Jails-like isolation, but because of the excellent coverage of the image repository.
plainOldText 4y ago
Could OpenBSD run Firecracker? I think it would be a nice development. Security focused OS meets security focused microVM.
- 0x457 4y ago
  No, Firecracker built on top of KVM. IIRC KVM only available on Linux and Illumos. There used to be FreeBSD port, but it's dead.
jpeeler 4y ago
From the article: "but Firecracker's implementation had two bugs: It placed the MPTable in the wrong place (above the advertised top of system memory rather than in the last kB) and it set a field containing the number of table entries to zero rather than the appropriate count. In both cases, Linux accepts the broken behaviour; so I added a "bug for bug compatibility" option to the FreeBSD MPTable code." I'm pretty sure that discoveries like these is why it's good to have operating system diversity. Conversely, sometimes I do wonder if having everyone on the same OS would allow us to have way more cool things and still have less bugs. But it doesn't matter too much because it'll never happen...
- capableweb 4y ago
  > But it doesn't matter too much because it'll never happen... I'm not saying you're wrong or right, but it's worth remembering that it's hard to guess how it'll be in the future. Maybe Apple acquires Microsoft at one point (or vice-versa) and eventually ends up buying Linux (somehow, don't ask me how), and we'll end up with one OS for absolutely everything. Weirder stuff have happened. There used to be multiple contenders for what would end up as "the internet", with many countries having their own versions. I'm sure at that point some people had similar sentiments running through their heads, something like "wow, wouldn't it be cool if all these various networks was just one instead? But that'll never happen..." Similarly how we today look at messaging services not being interoperable with each other, but seems this is slowly (veeery slowly) changing.
tiffanyh 4y ago
I don't know if I should be amazed at how 1-person can contribute to an OS ... or if I should be saddened at the current state of OS's given the 10,000s of developers working OS's. (Another person who comes to mind is Matt Dillon of DragonflyBSD and his huge contributions)
- cperciva 4y ago
  I had a lot of help. I was driving this, but it was absolutely not a one-person effort.
ComputerGuru 4y ago
@cperciva > The FreeBSD kernel now handles such duplicate environment variables by appending suffixes, so that we end up with virtio_mmio.device, virtio_mmio.device_1, virtio_mmio.device_2, et cetera, Was there any discussion of a cleaner alternative to this, perhaps extending the existing APIs with the ability to retrieve multiple values for one key (without breaking backwards compatibility, obviously) or concatenating the values with some sort of delimiter (colon, literal new line, ascii RS, or even null)?
- cperciva 4y ago
  Problem is that in general we want foo=bar to replace any previous value of foo.
  - ComputerGuru 4y ago
    I understand, that's why I was asking about an additional api (get_xx_multi, etc) that would return an array or composite string or whatever, while get_xx would continue to return just the single/latest value. Not being able to tell whether or not foo was specified twice or foo and foo_1 were specified separately just gives me flashbacks to a bygone era of osdev and gives me security creepy-crawlies. (also how do you handle `device=xxx device=yyy device_1=zzz`?)
    - cperciva 4y ago
      If there's other users for this I'll write a getenv_multi which takes a callback function. Didn't seem like much point doing that until we want to use this somewhere else though. We end up with device=xxx and device_1=yyy but if you mean that someone specifies device_1=zzz after that, it would show up as device_1_1=zzz. I'm inclined to file that one under "if a user is deliberately obtuse, they get what they deserve".
      - ComputerGuru 4y ago
        > We end up with device=xxx and device_1=yyy but if you mean ... No, I meant the first example. I'm trying to think of realistic non-obtuse cases like an entirely different spec calls for name_1, name_2, etc. but for some reason a numbered key is repeated. It's mostly an academic concern though, no worries! Thanks for the awesome work you do for the FreeBSD community, as always.

227 points by cperciva 4y ago

anotherhue 4y ago

Incredible. Maybe we can use this to shoe horn in docker support without the long lived VM

nix23 4y ago
https://bastillebsd.org/ https://bastillebsd.org/

agapon 4y ago

Nice development. Thank you, Colin!

rwmj 4y ago

Is it better / worse (in terms of say performance or security) than Bhyve?

eyberg 4y ago
Firecracker trades run-time performance in favor of faster boot-time latency. It is something that is being addressed but afaik still exists. This of course doesn't mean anything if you want firecracker to boot a multi-gig JVM installation (the larger the filesystem generally, expect a longer boot time). Security wise there are some minor differences. For instance bhyve supports virtio-rng but firecracker doesn't want to: https://github.com/firecracker-microvm/firecracker/issues/1671 https://github.com/firecracker-microvm/firecracker/issues/16... . I think if your app requires fast boot time and your app supports that than it's fine (so services that spin up and down on demand) but apps that daemonize for extended periods of time or take forever to initialize probably not a great fit.

tinco 4y ago

Nice! If I'd be looking into running FreeBSD it would be because I heard about its supposed security and I/O performance characteristics. Do those qualities hold up when run on firecracker this way? What applications would you choose FreeBSD for?

LunaSea 4y ago
Is I/O performance still better on FreeBSD after the last round of Linux improvements like io_uring and eBPF?
dangerface 4y ago
In my experience FreeBSD performance and security are great but not significantly better than linux. ZFS support on freebsd seems better than linux certainly its been supported a lot longer, bhyve works really well for virtualising freebsd and linux never tried it with windows. The biggest strength is the ports / package manager works better than anything I have tried on linux and it sticks to unix principles closer than linux. The biggest weakness for freebsd is docker and mono support is very poor. There is a desktop distro but in general any gui stuff linux beats freebsd hands down I only use freebsd for server stuff. I use it for production web servers, backup / storage servers, routers, virtualisation servers. I find its rock solid and more consistent than linux.
- exogenousdata 4y ago
  Agreed that the lack of docket support is frustrating given how pervasive docker has become. However I’ve been using jails[0] in my home lab for 12 years and it’s a delight to use. [0] - https://docs.freebsd.org/en/books/handbook/jails/ https://docs.freebsd.org/en/books/handbook/jails/
  - soupbowl 4y ago
    I love jails, bastille as a jail manager/builder is really great. https://github.com/BastilleBSD/bastille https://github.com/BastilleBSD/bastille

qalmakka 4y ago

FreeBSD really deserves all the love it can get. It is an amazing system to work with and develop on, and I'll never stop evangelizing everyone about it.

nurettin 4y ago
In your opinion, what is the best thing that FreeBSD offers?
- dangerface 4y ago
  Consistency, I have been using linux for 20+ years but if its not redhat or debian based I couldn't tell you how to start or stop services etc as its always different from the unix standard and every other linux. FreeBSD is unix it doesn't try to be that and something else at the same time.
  - nix23 4y ago
    >but if its not redhat or debian based I couldn't tell you how to start or stop services The same way as in freebsd "service blabla start" it's a compatibility hook.
    - dangerface 4y ago
      on fedora its systemctl instead of service because they are all systemd up, other distros might use something other than systemd and a compatibility hook is entirely dependant on the distro deciding it cares about backwards compatibility which isn't the case for fedora or debian at least in comparison to freebsd you can still build work the old school way.
      - cesarb 4y ago
        > on fedora its systemctl instead of service because they are all systemd up Fedora (and its derivatives like RHEL/CentOS/etc) still has the "service" command, it forwards to the corresponding systemctl command. The same for Debian (the comments in Debian's "service" command even mention it's a modified version of the RHEL/Fedora one).

dkjaudyeqooe 4y ago

A welcome development. I wonder how quickly it boots.

kevans91 4y ago
Colin's done a lot of work in improving boot time on EC2 and in general, so I'm a little surprised he didn't actually mention the boot time in this article. In 2022Q1 he had boot time down to 8s (in EC2, IIRC); based on how cut down the FIRECRACKER config he added is and that there's no loader involvement, I don't think it'd be too surprising if he hits ~5-6s or less.
- cperciva 4y ago
  It took me four days to get the blog post written and I only finished it last night because the toddler went to sleep early. I didn't want to drag it out any longer! But to answer the question: On small VMs, we can reach /sbin/init in under 50 ms and /etc/rc finishes running in under 500 ms. (Larger VMs take longer -- past a few GB of RAM, the time for initializing paging tables and launching APs starts to dominate.)

foomer 4y ago

That's a neat technical challenge solved, but I'm surprised by how much effort is being put in for these operating systems that basically no-one uses.

dangerface 4y ago
Lots of people and places use freebsd like netflix. I only see the popularity of freebsd increasing as linux moves more and more away from unix.
- yamtaddle 4y ago
  I'm slowly switching over for server-side, and next time I try a free OS desktop it'll probably be freebsd. Only thing I miss is docker, which I mostly just use as a package manager and service manager so I don't have to screw with distro-specific trivia for those things, and that's less of a problem on FreeBSD anyway. I've long tolerated Linux but never loved it, and it's moving away from anything I'm interested in with each year, it seems (the competing mostly-bad image-based app distribution systems and the way Wayland's going are... yikes). Glad FreeBSD's there as an alternative.
  - tcmart14 4y ago
    On FreeBSD, I would recommend looking at jails and something like CBSD as a way to manage jails. https://github.com/cbsd/cbsd https://github.com/cbsd/cbsd
    - yamtaddle 4y ago
      Right, I'm familiar with them, but a huge part of the benefit of docker is that it's basically a cross-distro (and, to some degree, cross-platform) very up-to-date server-oriented package manager. There are some efforts to create something like that for Jails but AFAIK none of them have gotten very far. Docker's benefit is that ~everyone uses it so there's usually an official docker image for anything one might want to run, kept in step with releases, plus a substantial history of images for rollbacks and testing and such. It turns "how do I install and run this?" from a bunch of platform- and distro-specific instructions into `docker-compose up` for everything, with the bonus that the compose file also tells you exactly where data & config for the service live, so it's also concise, guaranteed-accurate, and extremely useful documentation. Old LTS Ubuntu? Can run the same version of a service, with the same command, as on bleeding-edge Arch. On Arch, exact same command. Debian, Red Hat? Same command. MacOS, if you've got docker set up, same command. Windows, ditto. The latter two, non-natively, sure, but it works. Learn docker, and you can use it everywhere, unlike becoming proficient with, say, dpkg/apt or portage or pacman. It can even kinda save you from having to worry about how a given platform's init system and process management works, at least for simple use cases. Again, if you're doing most or everything on FreeBSD, development and hosting both, those benefits mostly vanish. But the reality is many workplaces or even personal dev+hosting environments are mixed, and docker suits those better than anything else I'm aware of—but if FreeBSD is in the mix, it's the odd-man-out. It's good not (only) because of its kinda-Jails-like isolation, but because of the excellent coverage of the image repository.

plainOldText 4y ago

Could OpenBSD run Firecracker? I think it would be a nice development. Security focused OS meets security focused microVM.

0x457 4y ago
No, Firecracker built on top of KVM. IIRC KVM only available on Linux and Illumos. There used to be FreeBSD port, but it's dead.

jpeeler 4y ago

From the article: "but Firecracker's implementation had two bugs: It placed the MPTable in the wrong place (above the advertised top of system memory rather than in the last kB) and it set a field containing the number of table entries to zero rather than the appropriate count. In both cases, Linux accepts the broken behaviour; so I added a "bug for bug compatibility" option to the FreeBSD MPTable code." I'm pretty sure that discoveries like these is why it's good to have operating system diversity. Conversely, sometimes I do wonder if having everyone on the same OS would allow us to have way more cool things and still have less bugs. But it doesn't matter too much because it'll never happen...

capableweb 4y ago
> But it doesn't matter too much because it'll never happen... I'm not saying you're wrong or right, but it's worth remembering that it's hard to guess how it'll be in the future. Maybe Apple acquires Microsoft at one point (or vice-versa) and eventually ends up buying Linux (somehow, don't ask me how), and we'll end up with one OS for absolutely everything. Weirder stuff have happened. There used to be multiple contenders for what would end up as "the internet", with many countries having their own versions. I'm sure at that point some people had similar sentiments running through their heads, something like "wow, wouldn't it be cool if all these various networks was just one instead? But that'll never happen..." Similarly how we today look at messaging services not being interoperable with each other, but seems this is slowly (veeery slowly) changing.

tiffanyh 4y ago

I don't know if I should be amazed at how 1-person can contribute to an OS ... or if I should be saddened at the current state of OS's given the 10,000s of developers working OS's. (Another person who comes to mind is Matt Dillon of DragonflyBSD and his huge contributions)

cperciva 4y ago
I had a lot of help. I was driving this, but it was absolutely not a one-person effort.

ComputerGuru 4y ago

@cperciva > The FreeBSD kernel now handles such duplicate environment variables by appending suffixes, so that we end up with virtio_mmio.device, virtio_mmio.device_1, virtio_mmio.device_2, et cetera, Was there any discussion of a cleaner alternative to this, perhaps extending the existing APIs with the ability to retrieve multiple values for one key (without breaking backwards compatibility, obviously) or concatenating the values with some sort of delimiter (colon, literal new line, ascii RS, or even null)?

cperciva 4y ago
Problem is that in general we want foo=bar to replace any previous value of foo.
- ComputerGuru 4y ago
  I understand, that's why I was asking about an additional api (get_xx_multi, etc) that would return an array or composite string or whatever, while get_xx would continue to return just the single/latest value. Not being able to tell whether or not foo was specified twice or foo and foo_1 were specified separately just gives me flashbacks to a bygone era of osdev and gives me security creepy-crawlies. (also how do you handle `device=xxx device=yyy device_1=zzz`?)
  - cperciva 4y ago
    If there's other users for this I'll write a getenv_multi which takes a callback function. Didn't seem like much point doing that until we want to use this somewhere else though. We end up with device=xxx and device_1=yyy but if you mean that someone specifies device_1=zzz after that, it would show up as device_1_1=zzz. I'm inclined to file that one under "if a user is deliberately obtuse, they get what they deserve".
    - ComputerGuru 4y ago
      > We end up with device=xxx and device_1=yyy but if you mean ... No, I meant the first example. I'm trying to think of realistic non-obtuse cases like an entirely different spec calls for name_1, name_2, etc. but for some reason a numbered key is repeated. It's mostly an academic concern though, no worries! Thanks for the awesome work you do for the FreeBSD community, as always.