6 ms·

> We burned months trying (and ultimately failing) to get Nvidia’s host drivers working to map virtualized GPUs into Intel Cloud Hypervisor... We think there’s

> We burned months trying (and ultimately failing) to get Nvidia’s host drivers working to map virtualized GPUs into Intel Cloud Hypervisor... We think there’s probably a market for users doing lightweight ML work getting tiny GPUs. This is what Nvidia MIG does, slicing a big GPU into arbitrarily small virtual GPUs. But for fully-virtualized workloads, it’s not baked; we can’t use it. Near as we can tell, MIG gives you a UUID to talk to the host driver, not a PCI device. Apparently this is technically possible, if you can find the right person at Nvidia to talk about vGPU licensing and magic incantations. Hopefully someone reading this HN front page story can make the introduction.

tptacek 2y ago
We (I) spent a lot of time talking to several different teams at Nvidia about this. We were able to get VFIO vGPUs to the point where guest libraries would recognize them, but the process fell apart in the guest/host licensing dance, and we weren't really OK with the idea that there'd be a phone-home licensing dance every time a Fly Machine started. Unlike GPU enablement at GCP or AWS, the core DX of a Fly Machine is that it stop/start very quickly; think of it as a midpoint in the design space between Lambda and Fargate. This is what we're talking about when we say it's hard to fit GPUs into our DX.
- transpute 2y ago
  > phone-home licensing dance every time a Fly Machine started To userspace Nvidia license server (a) in each host, (b) for entire Fly cloud, or (c) over WAN to Nvidia cloud?
  - tptacek 2y ago
    IIRC, (a) and (c), which I think sort of implies (b)? Really what we'd have wanted to do would have been to give Fly Machines MIG slices. But to the best of my understanding, MIG is paravirtualized; it doesn't give you SR-IOV-style PCI addresses for the slices, but rather a reference Nvidia's userland libraries pass to the kernel driver, which is a dance you can't do across VM boundaries unless your hypervisor does it deliberately.
    - transpute 2y ago
      Hypothetical scenario for Nvidia licensing of fast-start microVMs: 1. Instead of blocking VM start for license validation, convert that step into non-blocking async submission of usage telemetry, allowing every VM to start instantly. For PoC purposes, Nvidia's existing stack could be binary patched to proxy the license request to a script that isn't blocking VM start, pending step 2 negotiation. 2. Reconcile aggregate vGPU usage telemetry from Nvidia Fly-wide license server (Step 1) with aggregate vGPU usage reports from Fly's orchestration/control plane, which already has that data for VM usage accounting. In theory, Fly orchestration has more awareness of vGPU guest workload context than Nvidia's VM-start gatekeeping license agent, so there might be mutual interest in trading instant VM start for async workload analytics.
      - tptacek 2y ago
        Sure. We also could have virtualized CUDA ourselves, used MIG on the host-side, and done a proxy PCI passthrough driver thing in Cloud Hypervisor. I think we could have gotten it to work. But it would have been a huge lift. I'm glad we didn't try.
        transpute 2y ago
        > proxy PCI passthrough driver thing Do you mean vCS [1], which is already integrated and licensed by KVM/RedHat/Nutanix, Xen/Citrix and VMware? It's distinct from SR-IOV, PCI passthrough, vGPU-for-VDI, and MIG. [1] https://blogs.nvidia.com/blog/virtualcomputeserver-expands-vgpu-portfolio/ https://blogs.nvidia.com/blog/virtualcomputeserver-expands-v...
        tptacek 2y ago
        I assume anything we did to make MIG work for us would have been custom.
        transpute 2y ago
        Going back to the blog post: > Alternatively, we could have used a conventional hypervisor. Nvidia suggested VMware (heh). But they could have gotten things working had we used QEMU. We like QEMU fine, and could have talked ourselves into a security story for it, but the whole point of Fly Machines is that they take milliseconds to start. Someone could implement virtio-cuda (there are PoCs on github [1] [2]), but it would be a huge maintenance burden. It should really be done by Nvidia, in lockstep with CUDA extensions. Nvidia vCS makes use of licensed GPGPU emulation code in the VM device model, which is QEMU in the case of KVM and Xen. Cloud Hypervisor doesn't use QEMU, it has its own (Rust?) device model, https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/device_model.md https://github.com/cloud-hypervisor/cloud-hypervisor/blob/ma... So the question is, how to reuse Nvidia's proprietary GPGPU emulation code from QEMU, with Cloud Hypervisor? C and Rust are not friends. Can a Firecracker or Cloud Hypervisor VM use QEMU only for GPGPU emulation, alongside the existing device model, without impacting millisecond launch speed? Could an emulated vGPGPU be hotplugged after VM launch? There has been some design/PoC work for QEMU disaggregation [3][4] of emulation functions into separate processes. It might be possible to apply similar techniques so that Cloud Hypervisor's device model (in Rust) process could run alongside a QEMU GPGPU emulator (in C) process, with some coordination by KVM. If this approach is feasible, the architecture and code changes should be broadly useful to upstream for long-term support and maintenance, rather than custom to Fly. The custom code would be the GPGPU emulator, which is already maintained by Nvidia and running within QEMU on RedHat, Nutanix, etc. It would also advance the state of the art in security isolation and access control of emulated devices used by VMs. [1] https://github.com/coldfunction/qCUDA https://github.com/coldfunction/qCUDA [2] https://github.com/juniorprincewang/virtio-cuda-module https://github.com/juniorprincewang/virtio-cuda-module [3] https://www.qemu.org/docs/master/devel/multi-process.html https://www.qemu.org/docs/master/devel/multi-process.html [4] https://wiki.qemu.org/Features/MultiProcessQEMU https://wiki.qemu.org/Features/MultiProcessQEMU
- Aeolun 2y ago
  Is it GPU’s, or is it Nvidia GPU’s?
  - tptacek 2y ago
    The distinction is irrelevant to our customers.
    - tuna74 2y ago
      Not to the people reading your article.
intelVISA 2y ago
It is quite possible, bit surprised a minor systems detour would burn months but alas. This inspires me to figure out how to sell it, sounds expensive...

tptacek 2y ago

We (I) spent a lot of time talking to several different teams at Nvidia about this. We were able to get VFIO vGPUs to the point where guest libraries would recognize them, but the process fell apart in the guest/host licensing dance, and we weren't really OK with the idea that there'd be a phone-home licensing dance every time a Fly Machine started. Unlike GPU enablement at GCP or AWS, the core DX of a Fly Machine is that it stop/start very quickly; think of it as a midpoint in the design space between Lambda and Fargate. This is what we're talking about when we say it's hard to fit GPUs into our DX.

transpute 2y ago
> phone-home licensing dance every time a Fly Machine started To userspace Nvidia license server (a) in each host, (b) for entire Fly cloud, or (c) over WAN to Nvidia cloud?
- tptacek 2y ago
  IIRC, (a) and (c), which I think sort of implies (b)? Really what we'd have wanted to do would have been to give Fly Machines MIG slices. But to the best of my understanding, MIG is paravirtualized; it doesn't give you SR-IOV-style PCI addresses for the slices, but rather a reference Nvidia's userland libraries pass to the kernel driver, which is a dance you can't do across VM boundaries unless your hypervisor does it deliberately.
  - transpute 2y ago
    Hypothetical scenario for Nvidia licensing of fast-start microVMs: 1. Instead of blocking VM start for license validation, convert that step into non-blocking async submission of usage telemetry, allowing every VM to start instantly. For PoC purposes, Nvidia's existing stack could be binary patched to proxy the license request to a script that isn't blocking VM start, pending step 2 negotiation. 2. Reconcile aggregate vGPU usage telemetry from Nvidia Fly-wide license server (Step 1) with aggregate vGPU usage reports from Fly's orchestration/control plane, which already has that data for VM usage accounting. In theory, Fly orchestration has more awareness of vGPU guest workload context than Nvidia's VM-start gatekeeping license agent, so there might be mutual interest in trading instant VM start for async workload analytics.
    - tptacek 2y ago
      Sure. We also could have virtualized CUDA ourselves, used MIG on the host-side, and done a proxy PCI passthrough driver thing in Cloud Hypervisor. I think we could have gotten it to work. But it would have been a huge lift. I'm glad we didn't try.
      - transpute 2y ago
        > proxy PCI passthrough driver thing Do you mean vCS [1], which is already integrated and licensed by KVM/RedHat/Nutanix, Xen/Citrix and VMware? It's distinct from SR-IOV, PCI passthrough, vGPU-for-VDI, and MIG. [1] https://blogs.nvidia.com/blog/virtualcomputeserver-expands-vgpu-portfolio/ https://blogs.nvidia.com/blog/virtualcomputeserver-expands-v...
        tptacek 2y ago
        I assume anything we did to make MIG work for us would have been custom.
        transpute 2y ago
        Going back to the blog post: > Alternatively, we could have used a conventional hypervisor. Nvidia suggested VMware (heh). But they could have gotten things working had we used QEMU. We like QEMU fine, and could have talked ourselves into a security story for it, but the whole point of Fly Machines is that they take milliseconds to start. Someone could implement virtio-cuda (there are PoCs on github [1] [2]), but it would be a huge maintenance burden. It should really be done by Nvidia, in lockstep with CUDA extensions. Nvidia vCS makes use of licensed GPGPU emulation code in the VM device model, which is QEMU in the case of KVM and Xen. Cloud Hypervisor doesn't use QEMU, it has its own (Rust?) device model, https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/device_model.md https://github.com/cloud-hypervisor/cloud-hypervisor/blob/ma... So the question is, how to reuse Nvidia's proprietary GPGPU emulation code from QEMU, with Cloud Hypervisor? C and Rust are not friends. Can a Firecracker or Cloud Hypervisor VM use QEMU only for GPGPU emulation, alongside the existing device model, without impacting millisecond launch speed? Could an emulated vGPGPU be hotplugged after VM launch? There has been some design/PoC work for QEMU disaggregation [3][4] of emulation functions into separate processes. It might be possible to apply similar techniques so that Cloud Hypervisor's device model (in Rust) process could run alongside a QEMU GPGPU emulator (in C) process, with some coordination by KVM. If this approach is feasible, the architecture and code changes should be broadly useful to upstream for long-term support and maintenance, rather than custom to Fly. The custom code would be the GPGPU emulator, which is already maintained by Nvidia and running within QEMU on RedHat, Nutanix, etc. It would also advance the state of the art in security isolation and access control of emulated devices used by VMs. [1] https://github.com/coldfunction/qCUDA https://github.com/coldfunction/qCUDA [2] https://github.com/juniorprincewang/virtio-cuda-module https://github.com/juniorprincewang/virtio-cuda-module [3] https://www.qemu.org/docs/master/devel/multi-process.html https://www.qemu.org/docs/master/devel/multi-process.html [4] https://wiki.qemu.org/Features/MultiProcessQEMU https://wiki.qemu.org/Features/MultiProcessQEMU
Aeolun 2y ago
Is it GPU’s, or is it Nvidia GPU’s?
- tptacek 2y ago
  The distinction is irrelevant to our customers.
  - tuna74 2y ago
    Not to the people reading your article.

intelVISA 2y ago

It is quite possible, bit surprised a minor systems detour would burn months but alas. This inspires me to figure out how to sell it, sounds expensive...