4 ms·

great to hear more details about snapshot/restore in the wild, plenty about firecracker but seemingly much less about this exciting feature in real usecases. l

great to hear more details about snapshot/restore in the wild, plenty about firecracker but seemingly much less about this exciting feature in real usecases. looking forward to the unwritten details / future posts too, particularly: - How to handle network and IP duplicates on cloned VMs and - Turning a Dockerfile into a rootfs for the MicroVM (quickly)

nhoughto 4y ago
also interested in the upper limit of a micro vm, like how big can it get? 64gb memory? not really micro any more and maybe a traditional VM would be a better fit.
- CompuIves 4y ago
  The biggest VM we've been running for dev environments have 12GB RAM, 8vCPUs and 30GB disk. I've also done some tests with 16GB RAM and that worked well too. Have yet to find an upper limit. Another (unrelated) test we've done is on overprovisioning memory. We were able to run 200 VMs (all running Vite dev server where a file was changed every second) with 2GB RAM per VM, on a node with 128GB RAM. Because we were mapping the memory files on disk directly to the VM, the VM would automatically "swap" the memory back to the memory file when it had memory pressure. The bottleneck here was CPU.
- scarface74 4y ago
  AWS’s serverless Docker solution - Fargate - based on Firecracker supports up to 30GB of RAM and 4 vCPUs. Unrelated TIL: AWS Fargate has supported Windows since last October. I work at AWS and “specialize” in serverless and I didn’t know that.
  - easton 4y ago
    I have to imagine that Fargate on Windows doesn't use Firecracker though, right? Firecracker needs kernel level changes to work properly, and the open source version doesn't let you run anything but Linux.
    - scarface74 4y ago
      I have no idea how it works under the hood. Knowing what I know about Firecracker from watching the publicly available videos, I was shocked and thought it would never happen. On the other hand, CodeBuild has supported Windows containers for years and at least CodeBuild for Linux is based on Fargate, so the service team figured something out. (I had to figure out how to word that. I can’t say “they figured it out” since I work for the same company. But I couldn’t say “we” since I’m so far removed from any service team in the consulting department that it would be disingenuous)
- evandrofisico 4y ago
  The "micro" in microvms is less about size and more about resources. A typical virtual machine under Xen or KVM (para)virtualizes a lot of hardware and emulates a lot of devices, so that the operating system sees it as a normal machine. The microVM emulates the minimal possible set of devices needed to run, such as disks and network devices, and in the specific case of firecracker, through the use of the virtio model. So it can theoretically use huge amounts of memory of a large vCPU count and still be a microvm.
CompuIves 4y ago
Thank you! Right now our main use case is cloning development environments so we can provide a fresh running dev env for every branch and PR. However there are many other interesting applications, like speeding up CI jobs with VMs that start from a snapshot. I'll make sure we write about the other topics as well. For the network, we run the VM in its own network namespace on the host, and we give every VM the same IP. We then use an iptable rule to rewrite every incoming and outgoing packet to the IP that the host has assigned for the VM.
- nhoughto 4y ago
  Yeah the CI case is really interesting, it’s generally reproducible and declarative so a good fit that way, and time waiting for things to start is a big deal in CI. Another use case I was thinking of was stateful compilers like scala where warming up the compiler is expensive, often a CI task too.
rad_gruchalski 4y ago
Regarding turning Dockerfiles into a MicroVM: https://gruchalski.com/posts/2021-03-23-introducing-firebuild/ https://gruchalski.com/posts/2021-03-23-introducing-firebuil..., on GitHub: https://github.com/combust-labs/firebuild https://github.com/combust-labs/firebuild. This could get you started. Plenty of moving parts in that problem. Many root OS’s, many inits, … Difficult to pull this off by one person without any particular reason so I kinda suspended the project but who knows, seems like people want it so might be a good idea to reboot it. Disclaimer: I’m the author.
- silasb 4y ago
  I think fly.io does something like this as well?
shriphani 4y ago
The bit about turning a dockerfile into a rootfs. A docker image is just a tarball of tarballs. We do something like this: - you can dump the image using `docker save <name>`. - you can then get a list of the tarballs in this image by extracting this tarball and reading the file `manifest.json`; `Config` -> `Layers` will give you a list of tarballs (see undocker for how to do this: https://github.com/larsks/undocker https://github.com/larsks/undocker) - Untar these in a directory and use linux tools to convert this dir to a rootfs.

nhoughto 4y ago

also interested in the upper limit of a micro vm, like how big can it get? 64gb memory? not really micro any more and maybe a traditional VM would be a better fit.

CompuIves 4y ago
The biggest VM we've been running for dev environments have 12GB RAM, 8vCPUs and 30GB disk. I've also done some tests with 16GB RAM and that worked well too. Have yet to find an upper limit. Another (unrelated) test we've done is on overprovisioning memory. We were able to run 200 VMs (all running Vite dev server where a file was changed every second) with 2GB RAM per VM, on a node with 128GB RAM. Because we were mapping the memory files on disk directly to the VM, the VM would automatically "swap" the memory back to the memory file when it had memory pressure. The bottleneck here was CPU.
scarface74 4y ago
AWS’s serverless Docker solution - Fargate - based on Firecracker supports up to 30GB of RAM and 4 vCPUs. Unrelated TIL: AWS Fargate has supported Windows since last October. I work at AWS and “specialize” in serverless and I didn’t know that.
- easton 4y ago
  I have to imagine that Fargate on Windows doesn't use Firecracker though, right? Firecracker needs kernel level changes to work properly, and the open source version doesn't let you run anything but Linux.
  - scarface74 4y ago
    I have no idea how it works under the hood. Knowing what I know about Firecracker from watching the publicly available videos, I was shocked and thought it would never happen. On the other hand, CodeBuild has supported Windows containers for years and at least CodeBuild for Linux is based on Fargate, so the service team figured something out. (I had to figure out how to word that. I can’t say “they figured it out” since I work for the same company. But I couldn’t say “we” since I’m so far removed from any service team in the consulting department that it would be disingenuous)
evandrofisico 4y ago
The "micro" in microvms is less about size and more about resources. A typical virtual machine under Xen or KVM (para)virtualizes a lot of hardware and emulates a lot of devices, so that the operating system sees it as a normal machine. The microVM emulates the minimal possible set of devices needed to run, such as disks and network devices, and in the specific case of firecracker, through the use of the virtio model. So it can theoretically use huge amounts of memory of a large vCPU count and still be a microvm.

CompuIves 4y ago

Thank you! Right now our main use case is cloning development environments so we can provide a fresh running dev env for every branch and PR. However there are many other interesting applications, like speeding up CI jobs with VMs that start from a snapshot. I'll make sure we write about the other topics as well. For the network, we run the VM in its own network namespace on the host, and we give every VM the same IP. We then use an iptable rule to rewrite every incoming and outgoing packet to the IP that the host has assigned for the VM.

nhoughto 4y ago
Yeah the CI case is really interesting, it’s generally reproducible and declarative so a good fit that way, and time waiting for things to start is a big deal in CI. Another use case I was thinking of was stateful compilers like scala where warming up the compiler is expensive, often a CI task too.

rad_gruchalski 4y ago

Regarding turning Dockerfiles into a MicroVM: https://gruchalski.com/posts/2021-03-23-introducing-firebuild/ https://gruchalski.com/posts/2021-03-23-introducing-firebuil..., on GitHub: https://github.com/combust-labs/firebuild https://github.com/combust-labs/firebuild. This could get you started. Plenty of moving parts in that problem. Many root OS’s, many inits, … Difficult to pull this off by one person without any particular reason so I kinda suspended the project but who knows, seems like people want it so might be a good idea to reboot it. Disclaimer: I’m the author.

silasb 4y ago
I think fly.io does something like this as well?

shriphani 4y ago

The bit about turning a dockerfile into a rootfs. A docker image is just a tarball of tarballs. We do something like this: - you can dump the image using `docker save <name>`. - you can then get a list of the tarballs in this image by extracting this tarball and reading the file `manifest.json`; `Config` -> `Layers` will give you a list of tarballs (see undocker for how to do this: https://github.com/larsks/undocker https://github.com/larsks/undocker) - Untar these in a directory and use linux tools to convert this dir to a rootfs.