10 ms·

We clone a running VM in 2 seconds

414 points by mrkurt 4y ago

sytelus 4y ago
TLDR; At fundamental level, the idea is that most VMs don't change much to their large images so you can lazily only work with diffs. The copy-on-write allows blocks of your new file points to blocks of original and maintain diff of blocks that get changed. Overall great article!
kosolam 4y ago
Yep. Fc is awesome
- CompuIves 4y ago
  Firecracker is one of the most exciting technologies I've seen in a while. It felt like a whole world opened up after learning about its capabilities.
  - jacobgorm 4y ago
    TBH having worked on live- and self-migration on top of Xen, as well as on MicroVMs at Bromium, the startup that coined the term, this feels like history repeating, just with Rust instead of C. I was once told by one of the Bromium founders that they tried to sell the MicroVM idea to AWS, but the AWS guys just took everything they learned in the meeting as inspiration and built Firecracker instead.
mjb 4y ago
It's always great to see different ways folks are using Firecracker. Snapshot-and-restore is a particularly cool capability, especially if you solve the data movement problems (like these folks have). One challenge to clone-and-restore that they don't talk about here is making sure that clones don't behave too similarly (like returning the same cryptographic random numbers). We wrote a paper about that a while back (https://arxiv.org/abs/2102.12892 https://arxiv.org/abs/2102.12892), and the Linux kernel community has been doing some great work in that area recently too.
- CompuIves 4y ago
  Thanks for sharing the paper, that is super interesting. This is indeed a challenge. The fact that we run development workloads instead of production workloads makes this easier for our use case. We do some rehydration, and when forking across team/organization boundaries we don't clone if there are secrets set. But for production workloads this would not be ready yet. Once uniqueness has been solved though, VM cloning would become a real solution for serverless hosting (and many of other cases), exciting prospect!
- yjftsjthsd-h 4y ago
  Does firecracker not support virtio rng? I won't comment on other uniqueness issues, but I would naively expect that you can fix random number generation by outsourcing it to the host. Or does Linux not pull from the provided rng on every use, resulting in a gap right after restore where your per-VM rng isn't unique? I suppose you could fix that by making the VM kernel aware that it was just restored? And now I see why it's not trivial:P
  - cperciva 4y ago
    Nope. Official advice is "use RDRAND".
    - kevincox 4y ago
      Which does avoid this problem as long as you are using it directly (not just as a seed).
- boulos 4y ago
  I'm surprised "MAC" never appears in the paper. Does Linux happily pickup a new MAC "on wake-up"?
  - NavinF 4y ago
    What do MAC addresses have to do with rng? Are you thinking of the old style UUIDs that used the machines's MAC address and system time? What a terrible idea that was.
notsapiensatall 4y ago
Very cool, snapshotting has come a long way in recent years. I don't see anything about graphics in the article - could this approach also be used to clone a VM running a desktop window manager like Gnome or KDE? Or would that rely on GPU memory which is not included in the dumps?
- CompuIves 4y ago
  I don't believe Firecracker currently supports GPU (latest what I saw about it is here: https://github.com/firecracker-microvm/firecracker/issues/1179 https://github.com/firecracker-microvm/firecracker/issues/11...). But I wouldn't be surprised if there's another MicroVM manager that would support GPU + snapshotting.
nhoughto 4y ago
great to hear more details about snapshot/restore in the wild, plenty about firecracker but seemingly much less about this exciting feature in real usecases. looking forward to the unwritten details / future posts too, particularly: - How to handle network and IP duplicates on cloned VMs and - Turning a Dockerfile into a rootfs for the MicroVM (quickly)
- nhoughto 4y ago
  also interested in the upper limit of a micro vm, like how big can it get? 64gb memory? not really micro any more and maybe a traditional VM would be a better fit.
  - CompuIves 4y ago
    The biggest VM we've been running for dev environments have 12GB RAM, 8vCPUs and 30GB disk. I've also done some tests with 16GB RAM and that worked well too. Have yet to find an upper limit. Another (unrelated) test we've done is on overprovisioning memory. We were able to run 200 VMs (all running Vite dev server where a file was changed every second) with 2GB RAM per VM, on a node with 128GB RAM. Because we were mapping the memory files on disk directly to the VM, the VM would automatically "swap" the memory back to the memory file when it had memory pressure. The bottleneck here was CPU.
  - scarface74 4y ago
    AWS’s serverless Docker solution - Fargate - based on Firecracker supports up to 30GB of RAM and 4 vCPUs. Unrelated TIL: AWS Fargate has supported Windows since last October. I work at AWS and “specialize” in serverless and I didn’t know that.
    - easton 4y ago
      I have to imagine that Fargate on Windows doesn't use Firecracker though, right? Firecracker needs kernel level changes to work properly, and the open source version doesn't let you run anything but Linux.
conradev 4y ago
When QEMU saves a snapshot, it tries to be "smart" about memory, only saving the memory in use[1]. This trades off CPU at snapshot time for I/O at transfer time. How compatible is Firecracker's virtual memory subsystem with doing something like that? [1] https://github.com/qemu/qemu/blob/7dd9d7e0bd29abf590d1ac235c0a00606ef81153/migration/ram.c#L2599-L2616 https://github.com/qemu/qemu/blob/7dd9d7e0bd29abf590d1ac235c...
- conradev 4y ago
  To answer my own question, Firecracker supports the `track_dirty_pages` and `enable_diff_snapshots` flags, which allow for incremental snapshotting: https://github.com/firecracker-microvm/firecracker/blob/main/docs/snapshotting/snapshot-support.md#creating-diff-snapshots https://github.com/firecracker-microvm/firecracker/blob/main...
- CompuIves 4y ago
  Firecracker keeps a bitmap of which pages have been dirtied (it's a flag you can turn on), so you can make incremental snapshots of only the changed pages (more here: https://github.com/firecracker-microvm/firecracker/blob/main/docs/snapshotting/snapshot-support.md https://github.com/firecracker-microvm/firecracker/blob/main...). In our case we changed Firecracker to use a shared mmap instead of an private mmap, so in our case the dirtied pages were synced back automatically to the backing memory file. The main reason for this was to reduce IO on snapshot time. I'm also looking at other ways we can do this, because using a shared mmap fragments the underlying xfs fs pretty fast. Maybe we can batch writes more instead of writing single pages.
  - liuliu 4y ago
    Can these just be shared memory all the way through or in your case, the persistence to disk at fork time is important?
    - CompuIves 4y ago
      It could be shared memory all the way through, but the memory of the original VM should become read-only once a cloned VM starts reading from it. So then both VMs (the original VM and the new VM) should put their writes in a new CoW layer. Using XFS with CoW has been the easiest way to enable this, but if there's a way that we can do this purely in-memory, that would be even faster. That said, for hibernation we would still have to persist to disk, but timing is less important there.
nstart 4y ago
Tangential to the topic: I look forward to the day that fast snapshotting and snapshot restoring becomes a thing for all VPS providers like hetzner, digital ocean, and vultr (and all the others). Especially if a machine is snapshotted, restored, snapshotted again, restored and the cycle continues. Even if what’s stored doesn’t get much larger the subsequent snapshot+restore processes take a little longer each time. Each provider has different timelines with vultr saying it can take up to 60 minutes for a snapshot to restore. My use case is similar but different to code sandbox. I use a beefy remote machine for development and to keep costs low I fire it up and tear it down on demand and pay only for the hours the machine was up. It works fine for me but I just wish snapshotting+restoring was faster on these services. That would make it perfect.
- cperciva 4y ago
  Why not use EC2? I find it works great as a development environment like this; starting a stopped instance takes < 5 seconds.
  - nstart 4y ago
    Because you have to pay for stopped instances. That’s why I snapshot and restore. It’s the same with all VPS providers (at least the reliable ones I know)
    - cperciva 4y ago
      You're only paying for their disks. Admittedly it's more expensive than snapshot-and-restore, but it's much cheaper than leaving the instance running.
      - nstart 4y ago
        Yeesht. I had never tried this out with AWS so I had just completely made assumptions about their stopped instance pricing. Thanks for the correction. At the same time, the pricing is rough. Instances with similar configs cost anywhere between 3-7x the price. My current bill of between 2 to 3 USD a month would go up to about 15-25 USD. In absolute terms that's not big but over the months that is going to add up. Still. Food for thought for me. Thanks again :)
cperciva 4y ago
Not to diminish this work, but I think it's worth noting that it's increasingly possibly to launch new VMs extremely quickly too. FreeBSD/Firecracker can reach userland in 33 ms, and the OSv unikernel boots in under 10 ms. I think increasingly we'll see Firecracker used with EC2-like setups of "create a disk image with everything preinstalled and then boot it" rather than using snapshots of running (suspended) VMs.
- vlovich123 4y ago
  I think you may be ignoring the aspect of cloning the codebase and handling writes transparently and then being able to quickly clone/snapshot that VM.
  - cperciva 4y ago
    Cloning the codebase is what I'm getting at with preparing a disk image.
    - CompuIves 4y ago
      I'm very eager to see more developments in the fresh start times! The main reason why snapshotting became interesting for us, is because we're running development servers defined by our users. A development server could take a long time to start, sometimes minutes. So even if we can start the VM fast, the most important speedup for us is on the user code that we cannot control.
      - visarga 4y ago
        Say the user code initiates a download, what happens if we clone during the run of the operation? Will the clone be able to finish the download? The opposite case - say the user code binds to an IP:port to run a service. Will the clone try to step over the parent, binding to a port that is already taken?
        CompuIves 4y ago
londons_explore 4y ago
Beware that such techniques slow down memory access for the newly running VM quite substantially. For example, a simple memset() call across a few gigabytes of RAM inside the VM might slow down a factor of 1000x after a VM clone like this. Both 'parent' and 'child' VM's see the slowdowns. Some types of garbage collector also see really substantial slowdowns. It was a deal-breaker for my project where I did similar sorts of things.
- CompuIves 4y ago
  Could you share why you're seeing such a slowdown? I've mainly experienced a slowdown when loading pages for the first time, since new pages need to be page faulted into memory from disk first, but after those pages have been loaded into memory I don't experience any slowdown compared to fresh starts. That said, that's only based on how I've experienced it with the type of workloads that we run on the VMs.
  - londons_explore 4y ago
    It's the page fault+copy latency, together with some secondary effects from the page tables being updated (seems to briefly halt all cores). The actual copying of a page of RAM is almost free compared to the time spent in all the kernel code for a page fault. If your RAM is file backed, you end up spending lots of time in the filesystem code too - I used anonymous mappings which really helped there, and called clone() on the VM process to keep them shared. I suspect if you use huge pages you might see lots of the impact vanish, but obviously that has other downsides.
    - CompuIves 4y ago
      Right, that makes sense. Once the memory page is in memory it should be fast though. We use shared mapping right now, and practically the pages stay in memory during the lifetime of a VM once they've been loaded, but we need to do more testing when there's more memory pressure. I've been looking at huge pages recently, I'm going to do some more testing with transparent huge pages today and see if it changes performance. Unfortunately we cannot use reserved huge pages because that doesn't work with shared mmap on say an XFS FS. Another idea is to make clones use the same memory base layer of their parent, then the pages are already prefaulted and it would deduplicate overall memory usage. Many things to discover still..
aetherspawn 4y ago
There's something seriously wrong with the page on Firefox. The whole browser locks up for a few seconds when you scroll up/down. Couldn't possibly read the article.
- mrintegrity 4y ago
  Works fine for me, firefox on ubuntu (not installed with snap). I use ublock origin and it seems to have blocked a few external url's on that page so probably it's some bad js slowing you down.
  - brabel 4y ago
    Fine for me as well, on MacOS.
- pistoriusp 4y ago
  Has the same issue on my Pixel using Chrome. Felt like j was scrolling through slime.
nikita 4y ago
Did you guys think about live migrations? https://github.com/cloud-hypervisor/cloud-hypervisor https://github.com/cloud-hypervisor/cloud-hypervisor seems to support it and it shares a good amount of code with firecracker.
- CompuIves 4y ago
  Yes! We're watching cloud-hypervisor as well, it might also be more suitable for our use cases. Yes, we did look at live migrations since there's a lot written about it and it's the closest to cloning a running VM. Lots of development in that space!
ShroudedNight 4y ago
This sounds a lot like SnowFlock[1], a U of T project to fork Xen VMs from ~12 years ago. Are they related? Or is this an independent re-discovery of the same principles? [1] http://www.cs.toronto.edu/~brudno/public/pdf/lagar2009snowflock.pdf http://www.cs.toronto.edu/~brudno/public/pdf/lagar2009snowfl...
- rwmj 4y ago
  Qemu can also do this (and fast boot these days), there's nothing new about this.
- michaelmior 4y ago
  Came here to say this. I was a grad student at U of T working with some of the original SnowFlock authors back in 2009. (Although I had no hand in the development of SnowFlock itself, I did contribute to some of the follow up work.) Skimming through the article, it looks like the high level idea is the same. The major difference seems to be that SnowFlock would start a proprietary server which is responsible for sending memory pages over the network on demand whenever the clone reads them. Some follow up work also added several different prefetching strategies to improve the performance of the cloned VMs while they were still fetching remote memory. SnowFlock was really targeted at compute-heavy applications. The idea was that you could mostly set up your application in the single VM, clone it, and then after cloning, it could be fairly easy to configure the clones to continue working on the problem in parallel. My Masters thesis made use of SnowFlock to clone relational databases on demand. https://www.researchgate.net/publication/221351958_FlurryDB_A_dynamically_scalable_relational_database_with_virtual_machine_cloning https://www.researchgate.net/publication/221351958_FlurryDB_...
nikita 4y ago
This is very cool! This technology will open the door to preview environments in CI/CD. If you also have a versioned filesystem you can efficiently create lots of snapshots that store VM images differentially you can introduce a branchable/versioned environments for the whole backend and tie it to the repo commit hash.
- CompuIves 4y ago
  Exactly! We're looking at the second use case, to have a VM running tied to the git history. Right now we do this on a branch level, so you could say "connect to branch X on microservice Y" from another microservice, and you can test APIs quickly. There's a lot of new things this enables!
kgeist 4y ago
The "fork" feature pauses the current VM for cloning - does it mean that your environment can have unpredictable pauses because you never know when someone will press "Fork" and how often?
- CompuIves 4y ago
  Yes it's a good point, this is one of the reasons that we wanted the fork time to be low. If we keep it low enough, the connection won't break and other users won't notice it. That said, for some things (like terminal access) it's impossible to hide it. Practically, 99% of the forks will be done from the `main`/`master` branch of the repo, which is read-only for everyone on the team. So the mini-pause isn't breaking in those cases.
Ambberses 4y ago
[flagged]
mlejva 4y ago
This is a great post! Love that CEO goes into such technical details. Looking forward to read about networking. That I think is technically also interesting and has been a challenge for us for a bit. Coming to the VMs and lower level topics like kernel, or Linux networking has been really fun for me. Weirdly, things feel much simpler the lower you go for some reason. Probably less abstraction? A bit of self less promo. We are using Firecracker to create interactive onboarding for devs. We did one for Prisma https://prisma.usedevbook.com https://prisma.usedevbook.com We start a Firecracker clone when you visit the website. Everything you do happens in your Firecracker VM. You have access to the terminal and can play around with Prisma.
tatoalo 4y ago
Slightly OT, but I have a question on AWS Lambda(s) since they run on Firecracker MicripoVM(s): have somebody found a way to reduce cold-starts times? Online I was only able to find that the way to go seems to be to produce an heartbeat on a rolling schedule, wanted to look into this.
- santiagobasulto 4y ago
  The solution for cold starts is usually keeping them warm :) In Lambdas, you can schedule a cloudwatch event similar to the heartbeat you've mentioned.
  - devoutsalsa 4y ago
    Dumb question. How is keeping a lambda warm different than just running a VM? Does the warm lambda instance respond quickly, and when it starts getting saturated, then additional Lambdas come online via a cold start process?
    - santiagobasulto 4y ago
      I use Zappa, it just schedules a frequent execution of the lambda: https://github.com/zappa/Zappa#keeping-the-server-warm https://github.com/zappa/Zappa#keeping-the-server-warm
      - vegardx 4y ago
        But what devoutsalsa said is true, you will still get a cold start when a request comes in while the already "warm" lambda is processing another request. This is one of the gotchas that not everyone seem to understand about these cloud functions. Sure, you'll scale quite horizontally, but just one executor at the time. If you have a long cold start, this can significantly increase latency. Some providers let you pay for concurrency, so that you can scale out more quickly.
- JoshTriplett 4y ago
  What runtime are you using? A custom runtime can get cold start times in milliseconds, as long as you're not loading a large language runtime or a container. Try a custom runtime that has only a single statically linked binary in it and nothing else.
dark-star 4y ago
We cloned VMware vSphere VMs in under 5 seconds 4 years ago. Relatively easy with proper storage integration and things like refclone copies on the storage. Problem is the VM takes twice that time to boot so it's not as impressive ;-) (yes, it's a different idea to the OP but still pretty neat)
- AlphaSite 4y ago
  Is this seperate from vmfork? https://blogs.vmware.com/euc/2016/02/horizon-7-view-instant-clone-technology-linked-clone-just-in-time-desktop.html https://blogs.vmware.com/euc/2016/02/horizon-7-view-instant-...
- deleted 4y ago
  [deleted]
txutxu 4y ago
Not so new age... but in IRC, the evalbot in #bash, did use something similar with QEMU: > Initially, Qemu is booted and its state is saved. On each evaluated command, this state is loaded (giving a usable shell in less than one second), a command is fed on stdin and the output read on stdout. https://www.vidarholen.net/contents/evalbot/ https://www.vidarholen.net/contents/evalbot/
sluongng 4y ago
I think this skipped out on the downside of using Firecracker: the host underneath needs to be either Baremetal(AWS) or VM with nested virt support(GCP). This creates additional complexity to manage such a setup in production. Moreover, since both nested-virt VM / baremetal both comes at a very high spec by default, the economic would only make sense if you are at a scale that could saturate(or over-saturated like the article hinted at) these resources. Very interesting blog post nevertheless. Looking forward to read more!
- CompuIves 4y ago
  You're completely right. Right now we're hosting the VMs on bare-metal at Hetzner, and we're looking at OVH or Contabo for hosting in the US.
kashyapc 4y ago
As the side-note in the article mentions, the core idea of "copy-on-write" has been around for ages. In context of virtualization, check out QEMU's "qcow2" format and its notion of "backing files" and "overlays". A quick example to take offline, instantaneous "disk snapshots" (QEMU can do this for live VMs too): Let's assume you already have a disk image of a clean Linux distro, let's call it _base.raw_. Then you can create instantaneous "snapshot"[1] this way: $> qemu-img create -f qcow2 -b ./base.raw -F raw overlay1.qcow2 [The "-F raw" is specifying the file format of the backing file; this is a good practice to explicitly mention this when creating overlay files;] Once you do this, and boot the VM with overlay1.qcow2, all the new guest writes will go to overlay1.qcow2. And whenever the guest need to refer to some old data it is copied over from the backing file, base.raw into the overlay1.qcow2 file. This lets you take a a backup of the base image, or make more "snapshots" (overlays) based on it. To take an instantaneous disk snapshot while the guest is running, refer to the docs here[2]. [1] The term "snapshot" here a bit of a misnomer, it is actually called an "overlay" — because the overlay file "refers" to its backing file, which becomes read-only once you create the overlay. [2] https://libvirt.org/kbase/live_full_disk_backup.html https://libvirt.org/kbase/live_full_disk_backup.html
kretaceous 4y ago
I know nothing about VMs or filesystems but I absolutely enjoyed this article. The language was very clear and easy to follow. Would be following the blog from now on. I have a question about the copy-on-write example involving VM A and VM B. It says t VM B will directly use all the data from VM A and for any change, it copies the block, writes into it and reads from it after this. But what if, say, block 2 is changed by VM A and was never written to by VM B? Wouldn't VM B read the changed block 2? Clearly, it doesn't happen cause a fork is a copy, but an explanation of how this is tackled is appreciated!
- birdman3131 4y ago
  Vm A also does copy on write. So VM B is still seeing the unmodified block.
  - kretaceous 4y ago
    Hi, not sure I understand. What if VM A is a new VM? What happens to the block after copy-on-write? Just destroyed?
    - CompuIves 4y ago
      So let's say VM A is already running, and it's cloned to VM B. When that happens, we freeze the memory of VM A, and link it to VM B. For both VM A and VM B, any new write will be done to a new layer.
      - kretaceous 4y ago
        Thanks for replying. So the logic is to check if VM A has a new fork. If yes, then start CoW to a new layer of blocks, and leave the current layer to be linked with VM B. If no, just don't use CoW. I hope I got it right!
        CompuIves 4y ago
        Yep, that's exactly it. So the moment the fork happens, we create both a new layer for VM A and VM B, so they can both use the same base layer.
mkl95 4y ago
Let's say you wanted to test something quickly. You could 1) Clone your VM in 1.5s as described in the article 2) Clone your database in a few seconds with Database Lab Engine [1] 3) Something else? [1] https://postgres.ai/products/how-it-works https://postgres.ai/products/how-it-works
- CompuIves 4y ago
  That would work! You could even run the Postgres inside the VM, and the data inside the DB will be cloned as well between clones.
bluelightning2k 4y ago
CodeSandbox is one of the most impressive engineering teams I'm aware of For most of us who are consumers only of these more fundamental infrastructure projects, there's something deeply satisfying about seeing people push these boundaries (very appropriate for HN too). Fly is another similar team/blog
- parhamn 4y ago
  Not to take credit away, but interestingly enough, both of those heavily rely on Firecracker VM, which certainly solves a huge “fundamental” infrastructure problem.
solarkraft 4y ago
I'd like to see some of these techniques put into a directly usable project. This stuff seems a lot like where the VM world is heading, yet doing these things is still a lot harder than it'd need to be.
alexellisuk 4y ago
This is really cool. I've also been working with Firecracker, but for isolated CI runners with Docker and KinD/K3s support. Starting with GitHub Actions [1] I've also had interest in making OpenFaaS use pause/resume from Gatsby.js who wanted to reduce their hosting costs. The main challenges were around the networking - if you use CNI and the Go SDK [2] then restores simply don't work. Not sure if you're working with netlink and IMAP directly to get around it? My question is how are you guaranteeing uniqueness, or do you only clone snapshots for a single tenant? [3] [1] https://github.com/self-actuated/actuated https://github.com/self-actuated/actuated [2] https://github.com/firecracker-microvm/firecracker-go-sdk https://github.com/firecracker-microvm/firecracker-go-sdk [3] https://github.com/firecracker-microvm/firecracker/blob/main/docs/snapshotting/snapshot-support.md#snapshot-security-and-uniqueness https://github.com/firecracker-microvm/firecracker/blob/main...

414 points by mrkurt 4y ago

TLDR; At fundamental level, the idea is that most VMs don't change much to their large images so you can lazily only work with diffs. The copy-on-write allows blocks of your new file points to blocks of original and maintain diff of blocks that get changed. Overall great article!

kosolam 4y ago

Yep. Fc is awesome

CompuIves 4y ago
Firecracker is one of the most exciting technologies I've seen in a while. It felt like a whole world opened up after learning about its capabilities.
- jacobgorm 4y ago
  TBH having worked on live- and self-migration on top of Xen, as well as on MicroVMs at Bromium, the startup that coined the term, this feels like history repeating, just with Rust instead of C. I was once told by one of the Bromium founders that they tried to sell the MicroVM idea to AWS, but the AWS guys just took everything they learned in the meeting as inspiration and built Firecracker instead.

mjb 4y ago

It's always great to see different ways folks are using Firecracker. Snapshot-and-restore is a particularly cool capability, especially if you solve the data movement problems (like these folks have). One challenge to clone-and-restore that they don't talk about here is making sure that clones don't behave too similarly (like returning the same cryptographic random numbers). We wrote a paper about that a while back (https://arxiv.org/abs/2102.12892 https://arxiv.org/abs/2102.12892), and the Linux kernel community has been doing some great work in that area recently too.

CompuIves 4y ago
Thanks for sharing the paper, that is super interesting. This is indeed a challenge. The fact that we run development workloads instead of production workloads makes this easier for our use case. We do some rehydration, and when forking across team/organization boundaries we don't clone if there are secrets set. But for production workloads this would not be ready yet. Once uniqueness has been solved though, VM cloning would become a real solution for serverless hosting (and many of other cases), exciting prospect!
yjftsjthsd-h 4y ago
Does firecracker not support virtio rng? I won't comment on other uniqueness issues, but I would naively expect that you can fix random number generation by outsourcing it to the host. Or does Linux not pull from the provided rng on every use, resulting in a gap right after restore where your per-VM rng isn't unique? I suppose you could fix that by making the VM kernel aware that it was just restored? And now I see why it's not trivial:P
- cperciva 4y ago
  Nope. Official advice is "use RDRAND".
  - kevincox 4y ago
    Which does avoid this problem as long as you are using it directly (not just as a seed).
boulos 4y ago
I'm surprised "MAC" never appears in the paper. Does Linux happily pickup a new MAC "on wake-up"?
- NavinF 4y ago
  What do MAC addresses have to do with rng? Are you thinking of the old style UUIDs that used the machines's MAC address and system time? What a terrible idea that was.

notsapiensatall 4y ago

Very cool, snapshotting has come a long way in recent years. I don't see anything about graphics in the article - could this approach also be used to clone a VM running a desktop window manager like Gnome or KDE? Or would that rely on GPU memory which is not included in the dumps?

CompuIves 4y ago
I don't believe Firecracker currently supports GPU (latest what I saw about it is here: https://github.com/firecracker-microvm/firecracker/issues/1179 https://github.com/firecracker-microvm/firecracker/issues/11...). But I wouldn't be surprised if there's another MicroVM manager that would support GPU + snapshotting.

nhoughto 4y ago

great to hear more details about snapshot/restore in the wild, plenty about firecracker but seemingly much less about this exciting feature in real usecases. looking forward to the unwritten details / future posts too, particularly: - How to handle network and IP duplicates on cloned VMs and - Turning a Dockerfile into a rootfs for the MicroVM (quickly)

nhoughto 4y ago
also interested in the upper limit of a micro vm, like how big can it get? 64gb memory? not really micro any more and maybe a traditional VM would be a better fit.
- CompuIves 4y ago
  The biggest VM we've been running for dev environments have 12GB RAM, 8vCPUs and 30GB disk. I've also done some tests with 16GB RAM and that worked well too. Have yet to find an upper limit. Another (unrelated) test we've done is on overprovisioning memory. We were able to run 200 VMs (all running Vite dev server where a file was changed every second) with 2GB RAM per VM, on a node with 128GB RAM. Because we were mapping the memory files on disk directly to the VM, the VM would automatically "swap" the memory back to the memory file when it had memory pressure. The bottleneck here was CPU.
- scarface74 4y ago
  AWS’s serverless Docker solution - Fargate - based on Firecracker supports up to 30GB of RAM and 4 vCPUs. Unrelated TIL: AWS Fargate has supported Windows since last October. I work at AWS and “specialize” in serverless and I didn’t know that.
  - easton 4y ago
    I have to imagine that Fargate on Windows doesn't use Firecracker though, right? Firecracker needs kernel level changes to work properly, and the open source version doesn't let you run anything but Linux.

conradev 4y ago

When QEMU saves a snapshot, it tries to be "smart" about memory, only saving the memory in use[1]. This trades off CPU at snapshot time for I/O at transfer time. How compatible is Firecracker's virtual memory subsystem with doing something like that? [1] https://github.com/qemu/qemu/blob/7dd9d7e0bd29abf590d1ac235c0a00606ef81153/migration/ram.c#L2599-L2616 https://github.com/qemu/qemu/blob/7dd9d7e0bd29abf590d1ac235c...

conradev 4y ago
To answer my own question, Firecracker supports the `track_dirty_pages` and `enable_diff_snapshots` flags, which allow for incremental snapshotting: https://github.com/firecracker-microvm/firecracker/blob/main/docs/snapshotting/snapshot-support.md#creating-diff-snapshots https://github.com/firecracker-microvm/firecracker/blob/main...
CompuIves 4y ago
Firecracker keeps a bitmap of which pages have been dirtied (it's a flag you can turn on), so you can make incremental snapshots of only the changed pages (more here: https://github.com/firecracker-microvm/firecracker/blob/main/docs/snapshotting/snapshot-support.md https://github.com/firecracker-microvm/firecracker/blob/main...). In our case we changed Firecracker to use a shared mmap instead of an private mmap, so in our case the dirtied pages were synced back automatically to the backing memory file. The main reason for this was to reduce IO on snapshot time. I'm also looking at other ways we can do this, because using a shared mmap fragments the underlying xfs fs pretty fast. Maybe we can batch writes more instead of writing single pages.
- liuliu 4y ago
  Can these just be shared memory all the way through or in your case, the persistence to disk at fork time is important?
  - CompuIves 4y ago
    It could be shared memory all the way through, but the memory of the original VM should become read-only once a cloned VM starts reading from it. So then both VMs (the original VM and the new VM) should put their writes in a new CoW layer. Using XFS with CoW has been the easiest way to enable this, but if there's a way that we can do this purely in-memory, that would be even faster. That said, for hibernation we would still have to persist to disk, but timing is less important there.

nstart 4y ago

Tangential to the topic: I look forward to the day that fast snapshotting and snapshot restoring becomes a thing for all VPS providers like hetzner, digital ocean, and vultr (and all the others). Especially if a machine is snapshotted, restored, snapshotted again, restored and the cycle continues. Even if what’s stored doesn’t get much larger the subsequent snapshot+restore processes take a little longer each time. Each provider has different timelines with vultr saying it can take up to 60 minutes for a snapshot to restore. My use case is similar but different to code sandbox. I use a beefy remote machine for development and to keep costs low I fire it up and tear it down on demand and pay only for the hours the machine was up. It works fine for me but I just wish snapshotting+restoring was faster on these services. That would make it perfect.

cperciva 4y ago
Why not use EC2? I find it works great as a development environment like this; starting a stopped instance takes < 5 seconds.
- nstart 4y ago
  Because you have to pay for stopped instances. That’s why I snapshot and restore. It’s the same with all VPS providers (at least the reliable ones I know)
  - cperciva 4y ago
    You're only paying for their disks. Admittedly it's more expensive than snapshot-and-restore, but it's much cheaper than leaving the instance running.
    - nstart 4y ago
      Yeesht. I had never tried this out with AWS so I had just completely made assumptions about their stopped instance pricing. Thanks for the correction. At the same time, the pricing is rough. Instances with similar configs cost anywhere between 3-7x the price. My current bill of between 2 to 3 USD a month would go up to about 15-25 USD. In absolute terms that's not big but over the months that is going to add up. Still. Food for thought for me. Thanks again :)

cperciva 4y ago

Not to diminish this work, but I think it's worth noting that it's increasingly possibly to launch new VMs extremely quickly too. FreeBSD/Firecracker can reach userland in 33 ms, and the OSv unikernel boots in under 10 ms. I think increasingly we'll see Firecracker used with EC2-like setups of "create a disk image with everything preinstalled and then boot it" rather than using snapshots of running (suspended) VMs.

vlovich123 4y ago
I think you may be ignoring the aspect of cloning the codebase and handling writes transparently and then being able to quickly clone/snapshot that VM.
- cperciva 4y ago
  Cloning the codebase is what I'm getting at with preparing a disk image.
  - CompuIves 4y ago
    I'm very eager to see more developments in the fresh start times! The main reason why snapshotting became interesting for us, is because we're running development servers defined by our users. A development server could take a long time to start, sometimes minutes. So even if we can start the VM fast, the most important speedup for us is on the user code that we cannot control.
    - visarga 4y ago
      Say the user code initiates a download, what happens if we clone during the run of the operation? Will the clone be able to finish the download? The opposite case - say the user code binds to an IP:port to run a service. Will the clone try to step over the parent, binding to a port that is already taken?
      - CompuIves 4y ago

londons_explore 4y ago

Beware that such techniques slow down memory access for the newly running VM quite substantially. For example, a simple memset() call across a few gigabytes of RAM inside the VM might slow down a factor of 1000x after a VM clone like this. Both 'parent' and 'child' VM's see the slowdowns. Some types of garbage collector also see really substantial slowdowns. It was a deal-breaker for my project where I did similar sorts of things.

CompuIves 4y ago
Could you share why you're seeing such a slowdown? I've mainly experienced a slowdown when loading pages for the first time, since new pages need to be page faulted into memory from disk first, but after those pages have been loaded into memory I don't experience any slowdown compared to fresh starts. That said, that's only based on how I've experienced it with the type of workloads that we run on the VMs.
- londons_explore 4y ago
  It's the page fault+copy latency, together with some secondary effects from the page tables being updated (seems to briefly halt all cores). The actual copying of a page of RAM is almost free compared to the time spent in all the kernel code for a page fault. If your RAM is file backed, you end up spending lots of time in the filesystem code too - I used anonymous mappings which really helped there, and called clone() on the VM process to keep them shared. I suspect if you use huge pages you might see lots of the impact vanish, but obviously that has other downsides.
  - CompuIves 4y ago
    Right, that makes sense. Once the memory page is in memory it should be fast though. We use shared mapping right now, and practically the pages stay in memory during the lifetime of a VM once they've been loaded, but we need to do more testing when there's more memory pressure. I've been looking at huge pages recently, I'm going to do some more testing with transparent huge pages today and see if it changes performance. Unfortunately we cannot use reserved huge pages because that doesn't work with shared mmap on say an XFS FS. Another idea is to make clones use the same memory base layer of their parent, then the pages are already prefaulted and it would deduplicate overall memory usage. Many things to discover still..

aetherspawn 4y ago

There's something seriously wrong with the page on Firefox. The whole browser locks up for a few seconds when you scroll up/down. Couldn't possibly read the article.

mrintegrity 4y ago
Works fine for me, firefox on ubuntu (not installed with snap). I use ublock origin and it seems to have blocked a few external url's on that page so probably it's some bad js slowing you down.
- brabel 4y ago
  Fine for me as well, on MacOS.
pistoriusp 4y ago
Has the same issue on my Pixel using Chrome. Felt like j was scrolling through slime.

nikita 4y ago

Did you guys think about live migrations? https://github.com/cloud-hypervisor/cloud-hypervisor https://github.com/cloud-hypervisor/cloud-hypervisor seems to support it and it shares a good amount of code with firecracker.

CompuIves 4y ago
Yes! We're watching cloud-hypervisor as well, it might also be more suitable for our use cases. Yes, we did look at live migrations since there's a lot written about it and it's the closest to cloning a running VM. Lots of development in that space!

ShroudedNight 4y ago

This sounds a lot like SnowFlock[1], a U of T project to fork Xen VMs from ~12 years ago. Are they related? Or is this an independent re-discovery of the same principles? [1] http://www.cs.toronto.edu/~brudno/public/pdf/lagar2009snowflock.pdf http://www.cs.toronto.edu/~brudno/public/pdf/lagar2009snowfl...

rwmj 4y ago
Qemu can also do this (and fast boot these days), there's nothing new about this.
michaelmior 4y ago
Came here to say this. I was a grad student at U of T working with some of the original SnowFlock authors back in 2009. (Although I had no hand in the development of SnowFlock itself, I did contribute to some of the follow up work.) Skimming through the article, it looks like the high level idea is the same. The major difference seems to be that SnowFlock would start a proprietary server which is responsible for sending memory pages over the network on demand whenever the clone reads them. Some follow up work also added several different prefetching strategies to improve the performance of the cloned VMs while they were still fetching remote memory. SnowFlock was really targeted at compute-heavy applications. The idea was that you could mostly set up your application in the single VM, clone it, and then after cloning, it could be fairly easy to configure the clones to continue working on the problem in parallel. My Masters thesis made use of SnowFlock to clone relational databases on demand. https://www.researchgate.net/publication/221351958_FlurryDB_A_dynamically_scalable_relational_database_with_virtual_machine_cloning https://www.researchgate.net/publication/221351958_FlurryDB_...

nikita 4y ago

This is very cool! This technology will open the door to preview environments in CI/CD. If you also have a versioned filesystem you can efficiently create lots of snapshots that store VM images differentially you can introduce a branchable/versioned environments for the whole backend and tie it to the repo commit hash.

CompuIves 4y ago
Exactly! We're looking at the second use case, to have a VM running tied to the git history. Right now we do this on a branch level, so you could say "connect to branch X on microservice Y" from another microservice, and you can test APIs quickly. There's a lot of new things this enables!

kgeist 4y ago

The "fork" feature pauses the current VM for cloning - does it mean that your environment can have unpredictable pauses because you never know when someone will press "Fork" and how often?

CompuIves 4y ago
Yes it's a good point, this is one of the reasons that we wanted the fork time to be low. If we keep it low enough, the connection won't break and other users won't notice it. That said, for some things (like terminal access) it's impossible to hide it. Practically, 99% of the forks will be done from the `main`/`master` branch of the repo, which is read-only for everyone on the team. So the mini-pause isn't breaking in those cases.

Ambberses 4y ago

[flagged]

mlejva 4y ago

This is a great post! Love that CEO goes into such technical details. Looking forward to read about networking. That I think is technically also interesting and has been a challenge for us for a bit. Coming to the VMs and lower level topics like kernel, or Linux networking has been really fun for me. Weirdly, things feel much simpler the lower you go for some reason. Probably less abstraction? A bit of self less promo. We are using Firecracker to create interactive onboarding for devs. We did one for Prisma https://prisma.usedevbook.com https://prisma.usedevbook.com We start a Firecracker clone when you visit the website. Everything you do happens in your Firecracker VM. You have access to the terminal and can play around with Prisma.

tatoalo 4y ago

Slightly OT, but I have a question on AWS Lambda(s) since they run on Firecracker MicripoVM(s): have somebody found a way to reduce cold-starts times? Online I was only able to find that the way to go seems to be to produce an heartbeat on a rolling schedule, wanted to look into this.

santiagobasulto 4y ago
The solution for cold starts is usually keeping them warm :) In Lambdas, you can schedule a cloudwatch event similar to the heartbeat you've mentioned.
- devoutsalsa 4y ago
  Dumb question. How is keeping a lambda warm different than just running a VM? Does the warm lambda instance respond quickly, and when it starts getting saturated, then additional Lambdas come online via a cold start process?
  - santiagobasulto 4y ago
    I use Zappa, it just schedules a frequent execution of the lambda: https://github.com/zappa/Zappa#keeping-the-server-warm https://github.com/zappa/Zappa#keeping-the-server-warm
    - vegardx 4y ago
      But what devoutsalsa said is true, you will still get a cold start when a request comes in while the already "warm" lambda is processing another request. This is one of the gotchas that not everyone seem to understand about these cloud functions. Sure, you'll scale quite horizontally, but just one executor at the time. If you have a long cold start, this can significantly increase latency. Some providers let you pay for concurrency, so that you can scale out more quickly.
JoshTriplett 4y ago
What runtime are you using? A custom runtime can get cold start times in milliseconds, as long as you're not loading a large language runtime or a container. Try a custom runtime that has only a single statically linked binary in it and nothing else.

dark-star 4y ago

We cloned VMware vSphere VMs in under 5 seconds 4 years ago. Relatively easy with proper storage integration and things like refclone copies on the storage. Problem is the VM takes twice that time to boot so it's not as impressive ;-) (yes, it's a different idea to the OP but still pretty neat)

AlphaSite 4y ago
Is this seperate from vmfork? https://blogs.vmware.com/euc/2016/02/horizon-7-view-instant-clone-technology-linked-clone-just-in-time-desktop.html https://blogs.vmware.com/euc/2016/02/horizon-7-view-instant-...
deleted 4y ago
[deleted]

txutxu 4y ago

Not so new age... but in IRC, the evalbot in #bash, did use something similar with QEMU: > Initially, Qemu is booted and its state is saved. On each evaluated command, this state is loaded (giving a usable shell in less than one second), a command is fed on stdin and the output read on stdout. https://www.vidarholen.net/contents/evalbot/ https://www.vidarholen.net/contents/evalbot/

sluongng 4y ago

I think this skipped out on the downside of using Firecracker: the host underneath needs to be either Baremetal(AWS) or VM with nested virt support(GCP). This creates additional complexity to manage such a setup in production. Moreover, since both nested-virt VM / baremetal both comes at a very high spec by default, the economic would only make sense if you are at a scale that could saturate(or over-saturated like the article hinted at) these resources. Very interesting blog post nevertheless. Looking forward to read more!

CompuIves 4y ago
You're completely right. Right now we're hosting the VMs on bare-metal at Hetzner, and we're looking at OVH or Contabo for hosting in the US.

kashyapc 4y ago

As the side-note in the article mentions, the core idea of "copy-on-write" has been around for ages. In context of virtualization, check out QEMU's "qcow2" format and its notion of "backing files" and "overlays". A quick example to take offline, instantaneous "disk snapshots" (QEMU can do this for live VMs too): Let's assume you already have a disk image of a clean Linux distro, let's call it _base.raw_. Then you can create instantaneous "snapshot"[1] this way: $> qemu-img create -f qcow2 -b ./base.raw -F raw overlay1.qcow2 [The "-F raw" is specifying the file format of the backing file; this is a good practice to explicitly mention this when creating overlay files;] Once you do this, and boot the VM with overlay1.qcow2, all the new guest writes will go to overlay1.qcow2. And whenever the guest need to refer to some old data it is copied over from the backing file, base.raw into the overlay1.qcow2 file. This lets you take a a backup of the base image, or make more "snapshots" (overlays) based on it. To take an instantaneous disk snapshot while the guest is running, refer to the docs here[2]. [1] The term "snapshot" here a bit of a misnomer, it is actually called an "overlay" — because the overlay file "refers" to its backing file, which becomes read-only once you create the overlay. [2] https://libvirt.org/kbase/live_full_disk_backup.html https://libvirt.org/kbase/live_full_disk_backup.html

kretaceous 4y ago

I know nothing about VMs or filesystems but I absolutely enjoyed this article. The language was very clear and easy to follow. Would be following the blog from now on. I have a question about the copy-on-write example involving VM A and VM B. It says t VM B will directly use all the data from VM A and for any change, it copies the block, writes into it and reads from it after this. But what if, say, block 2 is changed by VM A and was never written to by VM B? Wouldn't VM B read the changed block 2? Clearly, it doesn't happen cause a fork is a copy, but an explanation of how this is tackled is appreciated!

birdman3131 4y ago
Vm A also does copy on write. So VM B is still seeing the unmodified block.
- kretaceous 4y ago
  Hi, not sure I understand. What if VM A is a new VM? What happens to the block after copy-on-write? Just destroyed?
  - CompuIves 4y ago
    So let's say VM A is already running, and it's cloned to VM B. When that happens, we freeze the memory of VM A, and link it to VM B. For both VM A and VM B, any new write will be done to a new layer.
    - kretaceous 4y ago
      Thanks for replying. So the logic is to check if VM A has a new fork. If yes, then start CoW to a new layer of blocks, and leave the current layer to be linked with VM B. If no, just don't use CoW. I hope I got it right!
      - CompuIves 4y ago
        Yep, that's exactly it. So the moment the fork happens, we create both a new layer for VM A and VM B, so they can both use the same base layer.

mkl95 4y ago

Let's say you wanted to test something quickly. You could 1) Clone your VM in 1.5s as described in the article 2) Clone your database in a few seconds with Database Lab Engine [1] 3) Something else? [1] https://postgres.ai/products/how-it-works https://postgres.ai/products/how-it-works

CompuIves 4y ago
That would work! You could even run the Postgres inside the VM, and the data inside the DB will be cloned as well between clones.

bluelightning2k 4y ago

CodeSandbox is one of the most impressive engineering teams I'm aware of For most of us who are consumers only of these more fundamental infrastructure projects, there's something deeply satisfying about seeing people push these boundaries (very appropriate for HN too). Fly is another similar team/blog

parhamn 4y ago
Not to take credit away, but interestingly enough, both of those heavily rely on Firecracker VM, which certainly solves a huge “fundamental” infrastructure problem.

solarkraft 4y ago

I'd like to see some of these techniques put into a directly usable project. This stuff seems a lot like where the VM world is heading, yet doing these things is still a lot harder than it'd need to be.

alexellisuk 4y ago

This is really cool. I've also been working with Firecracker, but for isolated CI runners with Docker and KinD/K3s support. Starting with GitHub Actions [1] I've also had interest in making OpenFaaS use pause/resume from Gatsby.js who wanted to reduce their hosting costs. The main challenges were around the networking - if you use CNI and the Go SDK [2] then restores simply don't work. Not sure if you're working with netlink and IMAP directly to get around it? My question is how are you guaranteeing uniqueness, or do you only clone snapshots for a single tenant? [3] [1] https://github.com/self-actuated/actuated https://github.com/self-actuated/actuated [2] https://github.com/firecracker-microvm/firecracker-go-sdk https://github.com/firecracker-microvm/firecracker-go-sdk [3] https://github.com/firecracker-microvm/firecracker/blob/main/docs/snapshotting/snapshot-support.md#snapshot-security-and-uniqueness https://github.com/firecracker-microvm/firecracker/blob/main...