4 ms·
We wrote all sorts of stuff this week and this is what gets to the front page. :P
by tptacek 2y ago
We wrote all sorts of stuff this week and this is what gets to the front page. :P
- Rzor 2y agoSign of the times.
- ethbr1 2y agoSounds like you might have been wrong in spending time on the other stuff. ;) Who even knows what the customer is ever going to want? Pivot. Pivot. Pivot. PS: And pouring one out for the engineering hours that went into shipping GPUs. Sometimes it's a fine product, but just doesn't fit.
- transpute 2y ago> We burned months trying (and ultimately failing) to get Nvidia’s host drivers working to map virtualized GPUs into Intel Cloud Hypervisor... We think there’s probably a market for users doing lightweight ML work getting tiny GPUs. This is what Nvidia MIG does, slicing a big GPU into arbitrarily small virtual GPUs. But for fully-virtualized workloads, it’s not baked; we can’t use it. Near as we can tell, MIG gives you a UUID to talk to the host driver, not a PCI device. Apparently this is technically possible, if you can find the right person at Nvidia to talk about vGPU licensing and magic incantations. Hopefully someone reading this HN front page story can make the introduction.
- tptacek 2y agoWe (I) spent a lot of time talking to several different teams at Nvidia about this. We were able to get VFIO vGPUs to the point where guest libraries would recognize them, but the process fell apart in the guest/host licensing dance, and we weren't really OK with the idea that there'd be a phone-home licensing dance every time a Fly Machine started. Unlike GPU enablement at GCP or AWS, the core DX of a Fly Machine is that it stop/start very quickly; think of it as a midpoint in the design space between Lambda and Fargate. This is what we're talking about when we say it's hard to fit GPUs into our DX.
- transpute 2y ago> phone-home licensing dance every time a Fly Machine started To userspace Nvidia license server (a) in each host, (b) for entire Fly cloud, or (c) over WAN to Nvidia cloud?
- tptacek 2y agoIIRC, (a) and (c), which I think sort of implies (b)? Really what we'd have wanted to do would have been to give Fly Machines MIG slices. But to the best of my understanding, MIG is paravirtualized; it doesn't give you SR-IOV-style PCI addresses for the slices, but rather a reference Nvidia's userland libraries pass to the kernel driver, which is a dance you can't do across VM boundaries unless your hypervisor does it deliberately.
- transpute 2y agoHypothetical scenario for Nvidia licensing of fast-start microVMs: 1. Instead of blocking VM start for license validation, convert that step into non-blocking async submission of usage telemetry, allowing every VM to start instantly. For PoC purposes, Nvidia's existing stack could be binary patched to proxy the license request to a script that isn't blocking VM start, pending step 2 negotiation. 2. Reconcile aggregate vGPU usage telemetry from Nvidia Fly-wide license server (Step 1) with aggregate vGPU usage reports from Fly's orchestration/control plane, which already has that data for VM usage accounting. In theory, Fly orchestration has more awareness of vGPU guest workload context than Nvidia's VM-start gatekeeping license agent, so there might be mutual interest in trading instant VM start for async workload analytics.
- tptacek 2y agoSure. We also could have virtualized CUDA ourselves, used MIG on the host-side, and done a proxy PCI passthrough driver thing in Cloud Hypervisor. I think we could have gotten it to work. But it would have been a huge lift. I'm glad we didn't try.
- intelVISA 2y agoIt is quite possible, bit surprised a minor systems detour would burn months but alas. This inspires me to figure out how to sell it, sounds expensive...
- fuddle 2y agoYou just wrote a blog post about whats needed for a top HN post. You should have known it would do well :) https://fly.io/blog/a-blog-if-kept/ https://fly.io/blog/a-blog-if-kept/
- nicoburns 2y agoFWIW, I'd consider this good publicity. I have no use for GPUs in the cloud (at least not at the prices that they're available at (in general, not just on fly)). So if fly is moving there effort towards things I actually need (like managed databases) then that's going to increase my confidence in them as a platform quite a bit.
- narag 2y agoEmpathy. Such a seemingly good idea. Maybe just a little ahead of its time.