14 ms·

We replaced Firecracker with QEMU

409 points by hugodutka 3y ago

wiradikusuma 3y ago
Fly uses Firecracker, and they host long-running processes. I wonder what's their opinion about it.
- gorbak25 3y ago
  I think their usecase makes a lot of sense as their workloads consume a predefined amount of ram. As a customer you rent a VM with a specified amount of memory so fly.io does not care about reclaiming it from a running VM.
  - no_wizard 3y ago
    Depends on if they're using smart memory allocation to keep costs lower, IE, if they can pattern that certain workloads only need N amount of memory at Y time, they can effectively borrow memory from one VM for usage in another that has an opposite statistical likelihood of needing that memory. This is why paying for dedicated memory is often more expensive than its counter part, because that dedicated memory is not considered as part of pooling.
    - adql 3y ago
      I think it would be easier to just enable KSM (KVM can use it to deduplicate memory and save some RAM on duplicate blocks like people loading same libraries into memory)
    - nerpderp82 3y ago
      Shit like this is how you burn customers and then they find out you aren't changing the sheets, they change hotels. Just because you can doesn't mean you should.
      - ushakov 3y ago
        It’s a common technique though. I believe it’s called oversubscription, where you rent the same hardware to more tenants hoping they won’t use it all at once. Fly.io themselves admitted they’re oversubscribed and AWS is doing the same for years now Source: https://fly.io/blog/the-serverless-server/ https://fly.io/blog/the-serverless-server/
        tptacek 3y ago
        We're not using virtio memory ballooning right now (we've never had the time to get it working).
      - geodel 3y ago
        >.. they change hotels And then they learn all hotels are doing exactly same thing. One hotel doing is risk, all hotels doing is industry standard. Airlines, hotels, restaurants, doctors and so on oversubscribe all the time. Whoever complains are free to move and add to their further disappointments.
        imtringued 3y ago
        Memory (DDR4) is like 1.50€ per GB nowadays. There is no need to skimp on it. The most expensive component is still the CPU if you actually want to do work and not just idle your webserver 99% of the time.
        no_wizard 3y ago
        this ignores just how competitive these services have become and how much competition there is in this marketplace. Anything that allows a company to balance user experience with cost of hardware will be evaluated. The fact is this is super common because even though RAM is super cheap, at scale, these things add up. If you as a provider don't have to spend 120K on RAM and can use that on more CPUs or something else, it can mean having a competitive edge over the provider that does have to spend that 120K.
    - flaminHotSpeedo 3y ago
      Generalized oversubscription like that is very challenging if not impossible to do securely, since you want to keep workloads isolated to single tenant numa nodes. E.g. using the firecracker jailer: https://github.com/firecracker-microvm/firecracker/blob/main/docs/jailer.md https://github.com/firecracker-microvm/firecracker/blob/main...
- tptacek 3y ago
  We like Firecracker. People should use whatever makes sense for them.
gwd 3y ago
Listen people, Firecracker is NOT A HYPERVISOR. A hypervisor runs right on the hardware. KVM is a hypervisor. Firecracker is a process that controls KVM. If you want to call firecracker (and QEMU, when used in conjunction with KVM) a VMM ("virtual machine monitor") I won't complain. But please please please, we need a word for what KVM and Xen are, and "hypervisor" is the best fit. Stop using that word for a user-level process like Firecracker.
- jstarks 3y ago
  Keep fighting the good fight, friend. Although I’ll note that the line between a VMM and hypervisor are not always clear. E.g., KVM includes some things that other hypervisors delegate to the VMM (such as instruction completion). And macOS’s hypervisor.framework is almost a pass through to the CPU’s raw capabilities.
- MuffinFlavored 3y ago
  > virtual machine monitor Is it good to think of libvirt as a virtual machine mointor, or is that more "virtual machine management"?
  - zbrozek 3y ago
    I'd love to get a clear explanation of what libvirt actually does. As far as I can tell it's a qemu argument assembler and launcher. For my own use-case, I just launch qemu from systemd unit files: https://wiki.archlinux.org/title/QEMU#With_systemd_service https://wiki.archlinux.org/title/QEMU#With_systemd_service
    - MuffinFlavored 3y ago
      > As far as I can tell it's a qemu argument assembler Raises a few questions to me: Can you use KVM/do KVM stuff without QEMU? Can you do libvirt stuff without QEMU? Hoping the answers to both aren't useless/"technically, but why would you want to?"
      - 3y ago
londons_explore 3y ago
I really want VM's to integrate 'smarter' with the host. For example, if I'm running 5 VM's, there is a good chance that many of the pages are identical. Not only do I want those pages to be deduplicated, but I want them to be zero-copy (ie. not deduplicated after-the-fact by some daemon). To do that, the guest block cache needs to be integrated with the host block-cache, so that whenever some guest application tries to map data from disk, the host notices that another virtual machine has already caused this data to be loaded, so we can just map the same page of already loaded data into the VM that is asking.
- scarface_74 3y ago
  This seems like a security issue waiting to happen when you’re running code from different users.
- jarym 3y ago
  Well that's all nice, but that would also need to be compute-efficient for it to be worthwhile and near-real-time dedupe of memory pages would be a REALLY tough challenge.
  - londons_explore 3y ago
    Pretty straightforward for disk blocks. Many VM disks are already de-duped, either through snapshopping or through copy on write host filesystems. The host block cache will end up deduplicating it automatically because all the 'copies' lead back to the same block on disk.
- gorbak25 3y ago
  I've tried to use virtio-pmem + DAX for the page cache to not be duplicated between the guest and the host. In practice the RAM overhead of virtio-pmem is unacceptable and it doesn't support discard operations at all. So yes a better solution would be needed.
- re-thc 3y ago
  KVM has KSM (kernel samepage merging) since a long time ago that de-duplicates pages.
blinkingled 3y ago
> The main issue we've had with QEMU is that it has too many options you need to configure. For instance, enabling your VM to return unused RAM to the host requires at least three challenging tasks This just works on Hyper-V Linux guests btw. For all the crap MS gets they do some things very right.
- freeone3000 3y ago
  It kind of just works. It’s actually broken under Debian 13 for some reason; memory usage infinitely balloons if the feature is enabled.
  - blinkingled 3y ago
    13 is debian-testing so I guess Microsoft still has time to make it work - last I checked it wasn't yet on Azure supported list.
    - freeone3000 3y ago
      For context: Most developers run testing since debian-stable averages two years behind
amarshall 3y ago
No mention of Cloud Hypervisor [1]…perhaps they don’t know about it? It’s based in part on Firecracker and supports free page reporting, virtio-blk-pci, PCI passthrough, and (I believe) discard in virtio-blk. [1]: https://www.cloudhypervisor.org/ https://www.cloudhypervisor.org/
- hugodutka 3y ago
  We do, and we'd love to use it in the future. We've found that it's not ready for prime time yet and it's missing some features. The biggest problem was that it does not support discard operations yet. Here's a short writeup we did about VMMs that we considered: https://github.com/hocus-dev/hocus/blob/main/rfd/0002-workspace-runtime.md https://github.com/hocus-dev/hocus/blob/main/rfd/0002-worksp...
  - amarshall 3y ago
    Thanks for the link to the elaboration! FYI footnotes 3 and 4 seem to be swapped.
    - selcuka 3y ago
      > footnotes 3 and 4 seem to be swapped Maybe they are async footnotes and there is a race condition. /s
k__ 3y ago
"Firecracker...'s excellent for running short-lived workloads...A little-known fact about Firecracker is its lack of support... for long-lived workloads." Okay.
lessaligned 3y ago
"the fork was very very bad for eating soup - this is a story about how we migrated to a spoon" ...firecracker does fine what it was designed to - short running fast start workloads. (oh, and the article starts by slightly misusing a bunch of technical terms, firecracker's not technically a hypervisor per se)
- sheepscreek 3y ago
  ..so is it more to support directly deploying functions to the cloud? Like, what AWS Lambda and CloudFront Functions might be built on?
  - yjftsjthsd-h 3y ago
    I'm pretty sure firecracker was literally created to underlie AWS Lambda. EDIT: Okay, https://www.geekwire.com/2018/firecracker-amazon-web-services-reinvents-serverless-computing-infrastructure-open-source-reputation/ https://www.geekwire.com/2018/firecracker-amazon-web-service... says my "pretty sure" memory is in fact correct.
    - zokier 3y ago
      That being said, firecracker also runs long-running tasks on AWS in the form of Fargate
    - bkettle 3y ago
      As does the paper [1] with details in section 4.1. [1]: https://www.usenix.org/system/files/nsdi20-paper-agache.pdf https://www.usenix.org/system/files/nsdi20-paper-agache.pdf
  - dathinab 3y ago
    yes, it was created originally for AWS Lambda mainly it's optimized to run code only shortly (init time max 10s, max usage is 15min, and default max request time 130s AFIK) also it's focused on thin server less functions, like e.g. deserialize some request, run some thin simple business logic and then delegate to other lambdas based on it. This kind of functions often have similar memory usage per-call and if a call is an outlier it can just discard the VM instance soon after (i.e. at most after starting up a new instance, i.e. at most 10s later)
deleted 3y ago
[deleted]
rgbrenner 3y ago
"Firecracker's RAM footprint starts low, but once a workload inside allocates RAM, Firecracker will never return it to the host system." Firecracker has a balloon device you can inflate (ie: acquire as much memory inside the VM as possible) and then deflate... returning the memory to the host. You can do this while the VM is running. https://github.com/firecracker-microvm/firecracker/blob/main/docs/ballooning.md https://github.com/firecracker-microvm/firecracker/blob/main...
- 0xbadcafebee 3y ago
  That and the fact that this was after "several weeks of testing" tells me this team doesn't have much virtualization experience. Firecracker is designed to quickly virtualize 1 headless stateless app (like a container), not run hundreds of different programs in a developer environment.
- paulv 3y ago
  The first footnote says If you squint hard enough, you'll find that Firecracker does support dynamic memory management with a technique called ballooning. However, in practice, it's not usable. To reclaim memory, you need to make sure that the guest OS isn't using it, which, for a general-purpose workload, is nearly impossible
  - deleted 3y ago
    [deleted]
  - adql 3y ago
    Yeah it's pretty hard problem as you'd need to defragment physical memory (while fixing all the virtual-to-physical mappings) to make contiguous block to free
  - dspillett 3y ago
    > which, for a general-purpose workload, is nearly impossible That depends on the workload and the maximum memory allocated to the guest OS. A lot of workloads rely on the OS cache/buffers to manage IO so unless RAM is quite restricted you can call in to release that pretty easily prior to having the balloon driver do its thing. In fact I'd not be surprised to be told the balloon process does this automatically itself. If the workload does its own IO management and memory allocation (something like SQL Server which will eat what RAM it can and does its own IO cacheing) or the VM's memory allocation is too small for OS caching to be a significant use after the rest of the workload (you might pair memory down to the bare minimum like this for a “fairly static content” server that doesn't see much variation in memory needs and can be allowed to swap a little if things grow temporarily), then I'd believe is it more difficult. That is hardly the use case for firecracker though so if that is the sort of workload being run perhaps reassessing the tool used for the job was the right call. Having said that my use of VMs is generally such that I can give them a good static amount of RAM for their needs and don't need to worry about dynamic allocation, so I'm far from a subject expert here. And, isn't firecraker more geared towards short-lived VMs, quick to spin up, do a job, spin down immediately (or after only a short idle timeout if the VM might answer another request if one comes in immediately or is already queued), so you are better off cycling VMs, which is probably happening anyway, than messing around with memory balloons? Again, I'm not talking from a position of personal experience here so corrections/details welcome!
ComputerGuru 3y ago
The article did an ok job of explaining the firecracker limitations they ran into but it was extremely skimpy when it came to qemu and just rushed to the conclusion “we did a lot of work so try our product.”
- naikrovek 3y ago
  yeah I was reading so I could find out what they did. I understand that they need to sell their product but jeez. don't leave us hanging like that
  - hugodutka 3y ago
    I didn't want to go into all the technical details, but we have another write-up that goes into details about RAM management: https://github.com/hocus-dev/hocus/blob/main/rfd/0003-workspace-ram-management.md https://github.com/hocus-dev/hocus/blob/main/rfd/0003-worksp... Other than making sure we release unused memory to the host, we didn't customize QEMU that much. Although we do have a cool layered storage solution - basically a faster alternative to QCOW2 that's also VMM independent. It's called overlaybd, and was created and implemented in Alibaba. That will probably be another blog post. https://github.com/containerd/overlaybd https://github.com/containerd/overlaybd
    - ComputerGuru 3y ago
      > I didn't want to go into all the technical details HN is here for the technical details ;)
      - gunapologist99 3y ago
        that should be the HN motto!
        xen2xen1 3y ago
        Thirded.. Ed.
heavyset_go 3y ago
Someone posted this and then immediately deleted their comment: https://qemu.readthedocs.io/en/latest/system/i386/microvm.html https://qemu.readthedocs.io/en/latest/system/i386/microvm.ht... I didn't know it existed until they posted, but QEMU has a Firecracker-inspired target: > microvm is a machine type inspired by Firecracker and constructed after its machine model. > It’s a minimalist machine type without PCI nor ACPI support, designed for short-lived guests. microvm also establishes a baseline for benchmarking and optimizing both QEMU and guest operating systems, since it is optimized for both boot time and footprint.
CompuIves 3y ago
At CodeSandbox we use Firecracker for hosting development environments, and I agree with the points. Though I don't think that means you should not use Firecracker for running long-lived workloads. We reclaim memory with a memory balloon device, for the disk trimming we discard (& compress) the disk, and for i/o speed we use io_uring (which we only use for scratch disks, the project disks are network disks). It's a tradeoff. It's more work and does require custom implementations. For us that made sense, because in return we get a lightweight VMM that we can more easily extend with functionality like memory snapshotting and live VM cloning [1][2]. [1]: https://codesandbox.io/blog/how-we-clone-a-running-vm-in-2-seconds https://codesandbox.io/blog/how-we-clone-a-running-vm-in-2-s... [2]: https://codesandbox.io/blog/cloning-microvms-using-userfaultfd https://codesandbox.io/blog/cloning-microvms-using-userfault...
- alexeldeib 3y ago
  These blogs are wonderful. I'd read them before figuring out firecracker snapshot/restore, but wanted to say it here. > i/o speed we use io_uring custom io_uring based driver for the VM block devices? or what do you mean here?
  - CompuIves 3y ago
    Thank you! > custom io_uring based driver for the VM block devices? or what do you mean here? We're using the async io backend that's shipped with Firecracker for our scratch disks.
- VWWHFSfQ 3y ago
  I don't know if this is relevant, but I've been intrigued by DragonflyBSD's "vkernel" [0] feature which (supposedly) allows for cloning the entire runtime state of the machine (established TCP connections, etc.) into a completely new userland memory space. I think they use it mostly for kernel debugging right now, but it's interesting to think about the possibilities of being able to just clone an entire running operating system to a new computer without interrupting even a single instruction. [0] https://www.dragonflybsd.org/docs/handbook/vkernel/ https://www.dragonflybsd.org/docs/handbook/vkernel/
Havoc 3y ago
I toyed with it a bit and was delighted to get it running. Only to discover getting even basic networking going is another mission in itself. Light is cool but for many tasks that level of Spartan is overkill If I’m investing time in light it might as well be wasm tech
datadeft 3y ago
Tl;dr: We tried to misuse technology and we failed. If Firecracker was developed for a single binary executed fir a short period of time why do you try to use it for multiple executables running for a long time? Does it make any sense to even try?
- mike_hock 3y ago
  Where in the "sales" pitch on the fancy-CSS website as well as the README does it say only to use it for single-shot workloads? I think the complaints are perfectly valid.
  - datadeft 3y ago
    "Firecracker is an alternative to QEMU that is purpose-built for running serverless functions and containers safely and efficiently, and nothing more." [1] Interesting. I guess we are reading a different website. 1. https://firecracker-microvm.github.io/ https://firecracker-microvm.github.io/
- zokier 3y ago
  AWS uses firecracker to execute long-running Fargate tasks; its hardly misuse
mgaunard 3y ago
I know a good way to make a process make the most of the hardware and play cooperatively with other processes: don't use virtualization. I will never understand the whole virtual machine and cloud craze. Your operating system is better than any hypervisor at sharing resources efficiently.
- akira2501 3y ago
  > cloud craze. Automatic scaling is great. Cloud parallelization (a.k.a fork) is absolutely wild once you get it rolling. Code deployments are incredibly simple. Never having to worry about physical machines or variable traffic loads is worth the small overhead they charge me for the wrapper. The generic system wide permissions model is an absolute joy once you get over the learning curve.
- kdmytro 3y ago
  I want to segregate, not to share.
- dhruvrrp 3y ago
  In this context (the blog post) and the reason firecracker was created, was to isolate workloads. And if youre running untrusted code, then using a virtualized environment is the easiest (id even say best) way to go about it.
- deleted 3y ago
  [deleted]
- Freedom2 3y ago
  After reading the README of virtualization tools (and looking at the author) I discovered the benefits of using them. I recommend also giving that a try.
  - mgaunard 3y ago
    I do have to use it since someone early on in the company I work at decided to do everything with AWS and Kubernetes. The fact of the matter is that it's just inefficient, slow and expensive. Bare metal is simple, fast, and keeps you in control.
favflam 3y ago
I know that Firecracker does not let you bind mount volumes, but QEMU does. So, we changed to QEMU from Firecracker. If you run the workloads in Kubernetes, you just have to change a single value in a yaml file to change the runtime. I would be scared to let unknown persons use QEMU that bind mounts volumes as that is a huge security risk. Firecracker, I think, was designed from the start to run un-sanitized workloads, hence, no bind mounting.
- brunoborges 3y ago
  > you just have to change a single value in a yaml file Most dangerous 12-words sentence.
jhatemyjob 3y ago
I came to the same conclusion as OP. QEMU is the most stable, hackable, well-supported VM hypervisor on the market. Setting it up is a pain, but once you get it set up with all your custom scripts, you never have to do it again. Ever. Even in your next project.
deleted 3y ago
[deleted]
alpb 3y ago
Presumably this doesn't use the "microvm" machine type in QEMU? (also on front page right now https://news.ycombinator.com/item?id=36673945 https://news.ycombinator.com/item?id=36673945)

409 points by hugodutka 3y ago

wiradikusuma 3y ago

Fly uses Firecracker, and they host long-running processes. I wonder what's their opinion about it.

gorbak25 3y ago
I think their usecase makes a lot of sense as their workloads consume a predefined amount of ram. As a customer you rent a VM with a specified amount of memory so fly.io does not care about reclaiming it from a running VM.
- no_wizard 3y ago
  Depends on if they're using smart memory allocation to keep costs lower, IE, if they can pattern that certain workloads only need N amount of memory at Y time, they can effectively borrow memory from one VM for usage in another that has an opposite statistical likelihood of needing that memory. This is why paying for dedicated memory is often more expensive than its counter part, because that dedicated memory is not considered as part of pooling.
  - adql 3y ago
    I think it would be easier to just enable KSM (KVM can use it to deduplicate memory and save some RAM on duplicate blocks like people loading same libraries into memory)
  - nerpderp82 3y ago
    Shit like this is how you burn customers and then they find out you aren't changing the sheets, they change hotels. Just because you can doesn't mean you should.
    - ushakov 3y ago
      It’s a common technique though. I believe it’s called oversubscription, where you rent the same hardware to more tenants hoping they won’t use it all at once. Fly.io themselves admitted they’re oversubscribed and AWS is doing the same for years now Source: https://fly.io/blog/the-serverless-server/ https://fly.io/blog/the-serverless-server/
      - tptacek 3y ago
        We're not using virtio memory ballooning right now (we've never had the time to get it working).
    - geodel 3y ago
      >.. they change hotels And then they learn all hotels are doing exactly same thing. One hotel doing is risk, all hotels doing is industry standard. Airlines, hotels, restaurants, doctors and so on oversubscribe all the time. Whoever complains are free to move and add to their further disappointments.
      - imtringued 3y ago
        Memory (DDR4) is like 1.50€ per GB nowadays. There is no need to skimp on it. The most expensive component is still the CPU if you actually want to do work and not just idle your webserver 99% of the time.
        no_wizard 3y ago
        this ignores just how competitive these services have become and how much competition there is in this marketplace. Anything that allows a company to balance user experience with cost of hardware will be evaluated. The fact is this is super common because even though RAM is super cheap, at scale, these things add up. If you as a provider don't have to spend 120K on RAM and can use that on more CPUs or something else, it can mean having a competitive edge over the provider that does have to spend that 120K.
  - flaminHotSpeedo 3y ago
    Generalized oversubscription like that is very challenging if not impossible to do securely, since you want to keep workloads isolated to single tenant numa nodes. E.g. using the firecracker jailer: https://github.com/firecracker-microvm/firecracker/blob/main/docs/jailer.md https://github.com/firecracker-microvm/firecracker/blob/main...
tptacek 3y ago
We like Firecracker. People should use whatever makes sense for them.

gwd 3y ago

Listen people, Firecracker is NOT A HYPERVISOR. A hypervisor runs right on the hardware. KVM is a hypervisor. Firecracker is a process that controls KVM. If you want to call firecracker (and QEMU, when used in conjunction with KVM) a VMM ("virtual machine monitor") I won't complain. But please please please, we need a word for what KVM and Xen are, and "hypervisor" is the best fit. Stop using that word for a user-level process like Firecracker.

jstarks 3y ago
Keep fighting the good fight, friend. Although I’ll note that the line between a VMM and hypervisor are not always clear. E.g., KVM includes some things that other hypervisors delegate to the VMM (such as instruction completion). And macOS’s hypervisor.framework is almost a pass through to the CPU’s raw capabilities.
MuffinFlavored 3y ago
> virtual machine monitor Is it good to think of libvirt as a virtual machine mointor, or is that more "virtual machine management"?
- zbrozek 3y ago
  I'd love to get a clear explanation of what libvirt actually does. As far as I can tell it's a qemu argument assembler and launcher. For my own use-case, I just launch qemu from systemd unit files: https://wiki.archlinux.org/title/QEMU#With_systemd_service https://wiki.archlinux.org/title/QEMU#With_systemd_service
  - MuffinFlavored 3y ago
    > As far as I can tell it's a qemu argument assembler Raises a few questions to me: Can you use KVM/do KVM stuff without QEMU? Can you do libvirt stuff without QEMU? Hoping the answers to both aren't useless/"technically, but why would you want to?"
    - 3y ago

londons_explore 3y ago

I really want VM's to integrate 'smarter' with the host. For example, if I'm running 5 VM's, there is a good chance that many of the pages are identical. Not only do I want those pages to be deduplicated, but I want them to be zero-copy (ie. not deduplicated after-the-fact by some daemon). To do that, the guest block cache needs to be integrated with the host block-cache, so that whenever some guest application tries to map data from disk, the host notices that another virtual machine has already caused this data to be loaded, so we can just map the same page of already loaded data into the VM that is asking.

scarface_74 3y ago
This seems like a security issue waiting to happen when you’re running code from different users.
jarym 3y ago
Well that's all nice, but that would also need to be compute-efficient for it to be worthwhile and near-real-time dedupe of memory pages would be a REALLY tough challenge.
- londons_explore 3y ago
  Pretty straightforward for disk blocks. Many VM disks are already de-duped, either through snapshopping or through copy on write host filesystems. The host block cache will end up deduplicating it automatically because all the 'copies' lead back to the same block on disk.
gorbak25 3y ago
I've tried to use virtio-pmem + DAX for the page cache to not be duplicated between the guest and the host. In practice the RAM overhead of virtio-pmem is unacceptable and it doesn't support discard operations at all. So yes a better solution would be needed.
re-thc 3y ago
KVM has KSM (kernel samepage merging) since a long time ago that de-duplicates pages.

blinkingled 3y ago

> The main issue we've had with QEMU is that it has too many options you need to configure. For instance, enabling your VM to return unused RAM to the host requires at least three challenging tasks This just works on Hyper-V Linux guests btw. For all the crap MS gets they do some things very right.

freeone3000 3y ago
It kind of just works. It’s actually broken under Debian 13 for some reason; memory usage infinitely balloons if the feature is enabled.
- blinkingled 3y ago
  13 is debian-testing so I guess Microsoft still has time to make it work - last I checked it wasn't yet on Azure supported list.
  - freeone3000 3y ago
    For context: Most developers run testing since debian-stable averages two years behind

amarshall 3y ago

No mention of Cloud Hypervisor [1]…perhaps they don’t know about it? It’s based in part on Firecracker and supports free page reporting, virtio-blk-pci, PCI passthrough, and (I believe) discard in virtio-blk. [1]: https://www.cloudhypervisor.org/ https://www.cloudhypervisor.org/

hugodutka 3y ago
We do, and we'd love to use it in the future. We've found that it's not ready for prime time yet and it's missing some features. The biggest problem was that it does not support discard operations yet. Here's a short writeup we did about VMMs that we considered: https://github.com/hocus-dev/hocus/blob/main/rfd/0002-workspace-runtime.md https://github.com/hocus-dev/hocus/blob/main/rfd/0002-worksp...
- amarshall 3y ago
  Thanks for the link to the elaboration! FYI footnotes 3 and 4 seem to be swapped.
  - selcuka 3y ago
    > footnotes 3 and 4 seem to be swapped Maybe they are async footnotes and there is a race condition. /s

k__ 3y ago

"Firecracker...'s excellent for running short-lived workloads...A little-known fact about Firecracker is its lack of support... for long-lived workloads." Okay.

lessaligned 3y ago

"the fork was very very bad for eating soup - this is a story about how we migrated to a spoon" ...firecracker does fine what it was designed to - short running fast start workloads. (oh, and the article starts by slightly misusing a bunch of technical terms, firecracker's not technically a hypervisor per se)

sheepscreek 3y ago
..so is it more to support directly deploying functions to the cloud? Like, what AWS Lambda and CloudFront Functions might be built on?
- yjftsjthsd-h 3y ago
  I'm pretty sure firecracker was literally created to underlie AWS Lambda. EDIT: Okay, https://www.geekwire.com/2018/firecracker-amazon-web-services-reinvents-serverless-computing-infrastructure-open-source-reputation/ https://www.geekwire.com/2018/firecracker-amazon-web-service... says my "pretty sure" memory is in fact correct.
  - zokier 3y ago
    That being said, firecracker also runs long-running tasks on AWS in the form of Fargate
  - bkettle 3y ago
    As does the paper [1] with details in section 4.1. [1]: https://www.usenix.org/system/files/nsdi20-paper-agache.pdf https://www.usenix.org/system/files/nsdi20-paper-agache.pdf
- dathinab 3y ago
  yes, it was created originally for AWS Lambda mainly it's optimized to run code only shortly (init time max 10s, max usage is 15min, and default max request time 130s AFIK) also it's focused on thin server less functions, like e.g. deserialize some request, run some thin simple business logic and then delegate to other lambdas based on it. This kind of functions often have similar memory usage per-call and if a call is an outlier it can just discard the VM instance soon after (i.e. at most after starting up a new instance, i.e. at most 10s later)

deleted 3y ago

[deleted]

rgbrenner 3y ago

"Firecracker's RAM footprint starts low, but once a workload inside allocates RAM, Firecracker will never return it to the host system." Firecracker has a balloon device you can inflate (ie: acquire as much memory inside the VM as possible) and then deflate... returning the memory to the host. You can do this while the VM is running. https://github.com/firecracker-microvm/firecracker/blob/main/docs/ballooning.md https://github.com/firecracker-microvm/firecracker/blob/main...

0xbadcafebee 3y ago
That and the fact that this was after "several weeks of testing" tells me this team doesn't have much virtualization experience. Firecracker is designed to quickly virtualize 1 headless stateless app (like a container), not run hundreds of different programs in a developer environment.
paulv 3y ago
The first footnote says If you squint hard enough, you'll find that Firecracker does support dynamic memory management with a technique called ballooning. However, in practice, it's not usable. To reclaim memory, you need to make sure that the guest OS isn't using it, which, for a general-purpose workload, is nearly impossible
- deleted 3y ago
  [deleted]
- adql 3y ago
  Yeah it's pretty hard problem as you'd need to defragment physical memory (while fixing all the virtual-to-physical mappings) to make contiguous block to free
- dspillett 3y ago
  > which, for a general-purpose workload, is nearly impossible That depends on the workload and the maximum memory allocated to the guest OS. A lot of workloads rely on the OS cache/buffers to manage IO so unless RAM is quite restricted you can call in to release that pretty easily prior to having the balloon driver do its thing. In fact I'd not be surprised to be told the balloon process does this automatically itself. If the workload does its own IO management and memory allocation (something like SQL Server which will eat what RAM it can and does its own IO cacheing) or the VM's memory allocation is too small for OS caching to be a significant use after the rest of the workload (you might pair memory down to the bare minimum like this for a “fairly static content” server that doesn't see much variation in memory needs and can be allowed to swap a little if things grow temporarily), then I'd believe is it more difficult. That is hardly the use case for firecracker though so if that is the sort of workload being run perhaps reassessing the tool used for the job was the right call. Having said that my use of VMs is generally such that I can give them a good static amount of RAM for their needs and don't need to worry about dynamic allocation, so I'm far from a subject expert here. And, isn't firecraker more geared towards short-lived VMs, quick to spin up, do a job, spin down immediately (or after only a short idle timeout if the VM might answer another request if one comes in immediately or is already queued), so you are better off cycling VMs, which is probably happening anyway, than messing around with memory balloons? Again, I'm not talking from a position of personal experience here so corrections/details welcome!

ComputerGuru 3y ago

The article did an ok job of explaining the firecracker limitations they ran into but it was extremely skimpy when it came to qemu and just rushed to the conclusion “we did a lot of work so try our product.”

naikrovek 3y ago
yeah I was reading so I could find out what they did. I understand that they need to sell their product but jeez. don't leave us hanging like that
- hugodutka 3y ago
  I didn't want to go into all the technical details, but we have another write-up that goes into details about RAM management: https://github.com/hocus-dev/hocus/blob/main/rfd/0003-workspace-ram-management.md https://github.com/hocus-dev/hocus/blob/main/rfd/0003-worksp... Other than making sure we release unused memory to the host, we didn't customize QEMU that much. Although we do have a cool layered storage solution - basically a faster alternative to QCOW2 that's also VMM independent. It's called overlaybd, and was created and implemented in Alibaba. That will probably be another blog post. https://github.com/containerd/overlaybd https://github.com/containerd/overlaybd
  - ComputerGuru 3y ago
    > I didn't want to go into all the technical details HN is here for the technical details ;)
    - gunapologist99 3y ago
      that should be the HN motto!
      - xen2xen1 3y ago
        Thirded.. Ed.

heavyset_go 3y ago

Someone posted this and then immediately deleted their comment: https://qemu.readthedocs.io/en/latest/system/i386/microvm.html https://qemu.readthedocs.io/en/latest/system/i386/microvm.ht... I didn't know it existed until they posted, but QEMU has a Firecracker-inspired target: > microvm is a machine type inspired by Firecracker and constructed after its machine model. > It’s a minimalist machine type without PCI nor ACPI support, designed for short-lived guests. microvm also establishes a baseline for benchmarking and optimizing both QEMU and guest operating systems, since it is optimized for both boot time and footprint.

CompuIves 3y ago

At CodeSandbox we use Firecracker for hosting development environments, and I agree with the points. Though I don't think that means you should not use Firecracker for running long-lived workloads. We reclaim memory with a memory balloon device, for the disk trimming we discard (& compress) the disk, and for i/o speed we use io_uring (which we only use for scratch disks, the project disks are network disks). It's a tradeoff. It's more work and does require custom implementations. For us that made sense, because in return we get a lightweight VMM that we can more easily extend with functionality like memory snapshotting and live VM cloning [1][2]. [1]: https://codesandbox.io/blog/how-we-clone-a-running-vm-in-2-seconds https://codesandbox.io/blog/how-we-clone-a-running-vm-in-2-s... [2]: https://codesandbox.io/blog/cloning-microvms-using-userfaultfd https://codesandbox.io/blog/cloning-microvms-using-userfault...

alexeldeib 3y ago
These blogs are wonderful. I'd read them before figuring out firecracker snapshot/restore, but wanted to say it here. > i/o speed we use io_uring custom io_uring based driver for the VM block devices? or what do you mean here?
- CompuIves 3y ago
  Thank you! > custom io_uring based driver for the VM block devices? or what do you mean here? We're using the async io backend that's shipped with Firecracker for our scratch disks.
VWWHFSfQ 3y ago
I don't know if this is relevant, but I've been intrigued by DragonflyBSD's "vkernel" [0] feature which (supposedly) allows for cloning the entire runtime state of the machine (established TCP connections, etc.) into a completely new userland memory space. I think they use it mostly for kernel debugging right now, but it's interesting to think about the possibilities of being able to just clone an entire running operating system to a new computer without interrupting even a single instruction. [0] https://www.dragonflybsd.org/docs/handbook/vkernel/ https://www.dragonflybsd.org/docs/handbook/vkernel/

Havoc 3y ago

I toyed with it a bit and was delighted to get it running. Only to discover getting even basic networking going is another mission in itself. Light is cool but for many tasks that level of Spartan is overkill If I’m investing time in light it might as well be wasm tech

datadeft 3y ago

Tl;dr: We tried to misuse technology and we failed. If Firecracker was developed for a single binary executed fir a short period of time why do you try to use it for multiple executables running for a long time? Does it make any sense to even try?

mike_hock 3y ago
Where in the "sales" pitch on the fancy-CSS website as well as the README does it say only to use it for single-shot workloads? I think the complaints are perfectly valid.
- datadeft 3y ago
  "Firecracker is an alternative to QEMU that is purpose-built for running serverless functions and containers safely and efficiently, and nothing more." [1] Interesting. I guess we are reading a different website. 1. https://firecracker-microvm.github.io/ https://firecracker-microvm.github.io/
zokier 3y ago
AWS uses firecracker to execute long-running Fargate tasks; its hardly misuse

mgaunard 3y ago

I know a good way to make a process make the most of the hardware and play cooperatively with other processes: don't use virtualization. I will never understand the whole virtual machine and cloud craze. Your operating system is better than any hypervisor at sharing resources efficiently.

akira2501 3y ago
> cloud craze. Automatic scaling is great. Cloud parallelization (a.k.a fork) is absolutely wild once you get it rolling. Code deployments are incredibly simple. Never having to worry about physical machines or variable traffic loads is worth the small overhead they charge me for the wrapper. The generic system wide permissions model is an absolute joy once you get over the learning curve.
kdmytro 3y ago
I want to segregate, not to share.
dhruvrrp 3y ago
In this context (the blog post) and the reason firecracker was created, was to isolate workloads. And if youre running untrusted code, then using a virtualized environment is the easiest (id even say best) way to go about it.
deleted 3y ago
[deleted]
Freedom2 3y ago
After reading the README of virtualization tools (and looking at the author) I discovered the benefits of using them. I recommend also giving that a try.
- mgaunard 3y ago
  I do have to use it since someone early on in the company I work at decided to do everything with AWS and Kubernetes. The fact of the matter is that it's just inefficient, slow and expensive. Bare metal is simple, fast, and keeps you in control.

favflam 3y ago

I know that Firecracker does not let you bind mount volumes, but QEMU does. So, we changed to QEMU from Firecracker. If you run the workloads in Kubernetes, you just have to change a single value in a yaml file to change the runtime. I would be scared to let unknown persons use QEMU that bind mounts volumes as that is a huge security risk. Firecracker, I think, was designed from the start to run un-sanitized workloads, hence, no bind mounting.

brunoborges 3y ago
> you just have to change a single value in a yaml file Most dangerous 12-words sentence.

jhatemyjob 3y ago

I came to the same conclusion as OP. QEMU is the most stable, hackable, well-supported VM hypervisor on the market. Setting it up is a pain, but once you get it set up with all your custom scripts, you never have to do it again. Ever. Even in your next project.

deleted 3y ago

[deleted]

alpb 3y ago

Presumably this doesn't use the "microvm" machine type in QEMU? (also on front page right now https://news.ycombinator.com/item?id=36673945 https://news.ycombinator.com/item?id=36673945)