4 ms·

Hypothetical scenario for Nvidia licensing of fast-start microVMs: 1. Instead of blocking VM start for license validation, convert that step into non-blocking

Hypothetical scenario for Nvidia licensing of fast-start microVMs: 1. Instead of blocking VM start for license validation, convert that step into non-blocking async submission of usage telemetry, allowing every VM to start instantly. For PoC purposes, Nvidia's existing stack could be binary patched to proxy the license request to a script that isn't blocking VM start, pending step 2 negotiation. 2. Reconcile aggregate vGPU usage telemetry from Nvidia Fly-wide license server (Step 1) with aggregate vGPU usage reports from Fly's orchestration/control plane, which already has that data for VM usage accounting. In theory, Fly orchestration has more awareness of vGPU guest workload context than Nvidia's VM-start gatekeeping license agent, so there might be mutual interest in trading instant VM start for async workload analytics.

tptacek 2y ago
Sure. We also could have virtualized CUDA ourselves, used MIG on the host-side, and done a proxy PCI passthrough driver thing in Cloud Hypervisor. I think we could have gotten it to work. But it would have been a huge lift. I'm glad we didn't try.
- transpute 2y ago
  > proxy PCI passthrough driver thing Do you mean vCS [1], which is already integrated and licensed by KVM/RedHat/Nutanix, Xen/Citrix and VMware? It's distinct from SR-IOV, PCI passthrough, vGPU-for-VDI, and MIG. [1] https://blogs.nvidia.com/blog/virtualcomputeserver-expands-vgpu-portfolio/ https://blogs.nvidia.com/blog/virtualcomputeserver-expands-v...
  - tptacek 2y ago
    I assume anything we did to make MIG work for us would have been custom.
    - transpute 2y ago
      Going back to the blog post: > Alternatively, we could have used a conventional hypervisor. Nvidia suggested VMware (heh). But they could have gotten things working had we used QEMU. We like QEMU fine, and could have talked ourselves into a security story for it, but the whole point of Fly Machines is that they take milliseconds to start. Someone could implement virtio-cuda (there are PoCs on github [1] [2]), but it would be a huge maintenance burden. It should really be done by Nvidia, in lockstep with CUDA extensions. Nvidia vCS makes use of licensed GPGPU emulation code in the VM device model, which is QEMU in the case of KVM and Xen. Cloud Hypervisor doesn't use QEMU, it has its own (Rust?) device model, https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/device_model.md https://github.com/cloud-hypervisor/cloud-hypervisor/blob/ma... So the question is, how to reuse Nvidia's proprietary GPGPU emulation code from QEMU, with Cloud Hypervisor? C and Rust are not friends. Can a Firecracker or Cloud Hypervisor VM use QEMU only for GPGPU emulation, alongside the existing device model, without impacting millisecond launch speed? Could an emulated vGPGPU be hotplugged after VM launch? There has been some design/PoC work for QEMU disaggregation [3][4] of emulation functions into separate processes. It might be possible to apply similar techniques so that Cloud Hypervisor's device model (in Rust) process could run alongside a QEMU GPGPU emulator (in C) process, with some coordination by KVM. If this approach is feasible, the architecture and code changes should be broadly useful to upstream for long-term support and maintenance, rather than custom to Fly. The custom code would be the GPGPU emulator, which is already maintained by Nvidia and running within QEMU on RedHat, Nutanix, etc. It would also advance the state of the art in security isolation and access control of emulated devices used by VMs. [1] https://github.com/coldfunction/qCUDA https://github.com/coldfunction/qCUDA [2] https://github.com/juniorprincewang/virtio-cuda-module https://github.com/juniorprincewang/virtio-cuda-module [3] https://www.qemu.org/docs/master/devel/multi-process.html https://www.qemu.org/docs/master/devel/multi-process.html [4] https://wiki.qemu.org/Features/MultiProcessQEMU https://wiki.qemu.org/Features/MultiProcessQEMU
      - ignoramous 2y ago
        > Someone could implement virtio-cuda (there are PoCs on github [1][2] Any company (let alone Fly) doing this won't go against Nvidia Enterprise T&C? > how to reuse Nvidia's proprietary GPGPU emulation code from QEMU If it has been contributed to QEMU, it isn't GPL/LGPL? > Could an emulated vGPGPU be hotplugged after VM launch gVisor instead bounces ioctls back and forth between "guest" and host. Sounds like a nice, lightweight (even if limited & sandbox-busting) approach, too. Unsure if it mitigates the need for the "licensing dance" tptacek mentioned above, but I reckon the security posture of such a setup is unacceptable for Fly. https://gvisor.dev/docs/user_guide/gpu/ https://gvisor.dev/docs/user_guide/gpu/ > would also advance the state of the art in security isolation and access control of emulated devices used by VMs I hope I'm not talking to DeepSeek / DeepResearch (:

tptacek 2y ago

Sure. We also could have virtualized CUDA ourselves, used MIG on the host-side, and done a proxy PCI passthrough driver thing in Cloud Hypervisor. I think we could have gotten it to work. But it would have been a huge lift. I'm glad we didn't try.

transpute 2y ago
> proxy PCI passthrough driver thing Do you mean vCS [1], which is already integrated and licensed by KVM/RedHat/Nutanix, Xen/Citrix and VMware? It's distinct from SR-IOV, PCI passthrough, vGPU-for-VDI, and MIG. [1] https://blogs.nvidia.com/blog/virtualcomputeserver-expands-vgpu-portfolio/ https://blogs.nvidia.com/blog/virtualcomputeserver-expands-v...
- tptacek 2y ago
  I assume anything we did to make MIG work for us would have been custom.
  - transpute 2y ago
    Going back to the blog post: > Alternatively, we could have used a conventional hypervisor. Nvidia suggested VMware (heh). But they could have gotten things working had we used QEMU. We like QEMU fine, and could have talked ourselves into a security story for it, but the whole point of Fly Machines is that they take milliseconds to start. Someone could implement virtio-cuda (there are PoCs on github [1] [2]), but it would be a huge maintenance burden. It should really be done by Nvidia, in lockstep with CUDA extensions. Nvidia vCS makes use of licensed GPGPU emulation code in the VM device model, which is QEMU in the case of KVM and Xen. Cloud Hypervisor doesn't use QEMU, it has its own (Rust?) device model, https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/device_model.md https://github.com/cloud-hypervisor/cloud-hypervisor/blob/ma... So the question is, how to reuse Nvidia's proprietary GPGPU emulation code from QEMU, with Cloud Hypervisor? C and Rust are not friends. Can a Firecracker or Cloud Hypervisor VM use QEMU only for GPGPU emulation, alongside the existing device model, without impacting millisecond launch speed? Could an emulated vGPGPU be hotplugged after VM launch? There has been some design/PoC work for QEMU disaggregation [3][4] of emulation functions into separate processes. It might be possible to apply similar techniques so that Cloud Hypervisor's device model (in Rust) process could run alongside a QEMU GPGPU emulator (in C) process, with some coordination by KVM. If this approach is feasible, the architecture and code changes should be broadly useful to upstream for long-term support and maintenance, rather than custom to Fly. The custom code would be the GPGPU emulator, which is already maintained by Nvidia and running within QEMU on RedHat, Nutanix, etc. It would also advance the state of the art in security isolation and access control of emulated devices used by VMs. [1] https://github.com/coldfunction/qCUDA https://github.com/coldfunction/qCUDA [2] https://github.com/juniorprincewang/virtio-cuda-module https://github.com/juniorprincewang/virtio-cuda-module [3] https://www.qemu.org/docs/master/devel/multi-process.html https://www.qemu.org/docs/master/devel/multi-process.html [4] https://wiki.qemu.org/Features/MultiProcessQEMU https://wiki.qemu.org/Features/MultiProcessQEMU
    - ignoramous 2y ago
      > Someone could implement virtio-cuda (there are PoCs on github [1][2] Any company (let alone Fly) doing this won't go against Nvidia Enterprise T&C? > how to reuse Nvidia's proprietary GPGPU emulation code from QEMU If it has been contributed to QEMU, it isn't GPL/LGPL? > Could an emulated vGPGPU be hotplugged after VM launch gVisor instead bounces ioctls back and forth between "guest" and host. Sounds like a nice, lightweight (even if limited & sandbox-busting) approach, too. Unsure if it mitigates the need for the "licensing dance" tptacek mentioned above, but I reckon the security posture of such a setup is unacceptable for Fly. https://gvisor.dev/docs/user_guide/gpu/ https://gvisor.dev/docs/user_guide/gpu/ > would also advance the state of the art in security isolation and access control of emulated devices used by VMs I hope I'm not talking to DeepSeek / DeepResearch (: