4 ms·

It's always great to see different ways folks are using Firecracker. Snapshot-and-restore is a particularly cool capability, especially if you solve the data mo

by mjb 4y ago

It's always great to see different ways folks are using Firecracker. Snapshot-and-restore is a particularly cool capability, especially if you solve the data movement problems (like these folks have). One challenge to clone-and-restore that they don't talk about here is making sure that clones don't behave too similarly (like returning the same cryptographic random numbers). We wrote a paper about that a while back (https://arxiv.org/abs/2102.12892 https://arxiv.org/abs/2102.12892), and the Linux kernel community has been doing some great work in that area recently too.

CompuIves 4y ago
Thanks for sharing the paper, that is super interesting. This is indeed a challenge. The fact that we run development workloads instead of production workloads makes this easier for our use case. We do some rehydration, and when forking across team/organization boundaries we don't clone if there are secrets set. But for production workloads this would not be ready yet. Once uniqueness has been solved though, VM cloning would become a real solution for serverless hosting (and many of other cases), exciting prospect!
yjftsjthsd-h 4y ago
Does firecracker not support virtio rng? I won't comment on other uniqueness issues, but I would naively expect that you can fix random number generation by outsourcing it to the host. Or does Linux not pull from the provided rng on every use, resulting in a gap right after restore where your per-VM rng isn't unique? I suppose you could fix that by making the VM kernel aware that it was just restored? And now I see why it's not trivial:P
- cperciva 4y ago
  Nope. Official advice is "use RDRAND".
  - kevincox 4y ago
    Which does avoid this problem as long as you are using it directly (not just as a seed).
boulos 4y ago
I'm surprised "MAC" never appears in the paper. Does Linux happily pickup a new MAC "on wake-up"?
- NavinF 4y ago
  What do MAC addresses have to do with rng? Are you thinking of the old style UUIDs that used the machines's MAC address and system time? What a terrible idea that was.
  - flumpcakes 4y ago
    I assume not reusing a MAC address falls into the same bucket of "make sure the VMs are not too similar" rather than anything specific to random number generation.
    - lathiat 4y ago
      Hit that exact bug with a customer at work in libvirt. Two machines booted at approximately the same time generated a VM with the same Mac. Due to very poor choices of random seed using the boot time and PID and xoring which made that even less random. Details here: https://bugs.launchpad.net/bugs/1710341 https://bugs.launchpad.net/bugs/1710341 It was since fixed though I never updated the bug. src/util/virrandom.c:virRandomOnceInit seeds the random number generator using this formula: unsigned int seed = time(NULL) ^ getpid(); This seems to be a popular method after a quick google but it's easy to see how this can be problematic. The time is only in seconds, and during boot of a relatively identical system these numbers are both likely to be relatively similar across multiple systems which is quite likely in cloud-like environments. Secondly, by using bitwise OR only a small difference is created and if the 1st or 2nd MSB of the pid or time are 0 then it would be easy to have colliding values. Though problematic from basic logic, I also tested this with a small test program trying 67,921 unique combinations of time() and pid() which produced only 5,693 random seeds using PID range 6799-6810 and time() range 1502484340 to 1502489999.
- JoshTriplett 4y ago
  You could hotplug different hardware, but having a unique MAC address isn't very important if you're on a virtual network where you only talk to the host to get your traffic routed. A unique MAC is only important if you put the cloned VMs on the same network segment.

by mjb 4y ago

CompuIves 4y ago

Thanks for sharing the paper, that is super interesting. This is indeed a challenge. The fact that we run development workloads instead of production workloads makes this easier for our use case. We do some rehydration, and when forking across team/organization boundaries we don't clone if there are secrets set. But for production workloads this would not be ready yet. Once uniqueness has been solved though, VM cloning would become a real solution for serverless hosting (and many of other cases), exciting prospect!

yjftsjthsd-h 4y ago

Does firecracker not support virtio rng? I won't comment on other uniqueness issues, but I would naively expect that you can fix random number generation by outsourcing it to the host. Or does Linux not pull from the provided rng on every use, resulting in a gap right after restore where your per-VM rng isn't unique? I suppose you could fix that by making the VM kernel aware that it was just restored? And now I see why it's not trivial:P

cperciva 4y ago
Nope. Official advice is "use RDRAND".
- kevincox 4y ago
  Which does avoid this problem as long as you are using it directly (not just as a seed).

boulos 4y ago

I'm surprised "MAC" never appears in the paper. Does Linux happily pickup a new MAC "on wake-up"?

NavinF 4y ago
What do MAC addresses have to do with rng? Are you thinking of the old style UUIDs that used the machines's MAC address and system time? What a terrible idea that was.
- flumpcakes 4y ago
  I assume not reusing a MAC address falls into the same bucket of "make sure the VMs are not too similar" rather than anything specific to random number generation.
  - lathiat 4y ago
    Hit that exact bug with a customer at work in libvirt. Two machines booted at approximately the same time generated a VM with the same Mac. Due to very poor choices of random seed using the boot time and PID and xoring which made that even less random. Details here: https://bugs.launchpad.net/bugs/1710341 https://bugs.launchpad.net/bugs/1710341 It was since fixed though I never updated the bug. src/util/virrandom.c:virRandomOnceInit seeds the random number generator using this formula: unsigned int seed = time(NULL) ^ getpid(); This seems to be a popular method after a quick google but it's easy to see how this can be problematic. The time is only in seconds, and during boot of a relatively identical system these numbers are both likely to be relatively similar across multiple systems which is quite likely in cloud-like environments. Secondly, by using bitwise OR only a small difference is created and if the 1st or 2nd MSB of the pid or time are 0 then it would be easy to have colliding values. Though problematic from basic logic, I also tested this with a small test program trying 67,921 unique combinations of time() and pid() which produced only 5,693 random seeds using PID range 6799-6810 and time() range 1502484340 to 1502489999.
JoshTriplett 4y ago
You could hotplug different hardware, but having a unique MAC address isn't very important if you're on a virtual network where you only talk to the host to get your traffic routed. A unique MAC is only important if you put the cloned VMs on the same network segment.