5 ms·

Great questions! To clarify the demo, we were ssh'd into a linux machine with no GPU. Going into more details for how this works, we intercept communication be

by bmodel 2y ago

Great questions! To clarify the demo, we were ssh'd into a linux machine with no GPU. Going into more details for how this works, we intercept communication between the CPU and the GPU so only GPU code and commands are sent across the network to a GPU that we are hosting. This way we are able to virtualize a remote GPU and make your computer think it's directly attached to that GPU. We are not copying your CPU code and running it on our machines. The CPU code runs entirely on your instance (meaning no files need to be copied over or packages installed on the GPU machine). One of the benefits of this approach is that you can easily scale to a more / less powerful GPU without needing to setup a new server.

billconan 2y ago
does this mean you have a customized/dummy kernel gpu driver? will that cause system instability, say, if the network suddenly dropped?
- bmodel 2y ago
  We are not writing any kernel drivers, this runs entirely in userspace (this won't result in a crowdstrike level crash haha). Given that, if the network suddenly dropped then only the process using the GPU would fail.
  - ZeroCool2u 2y ago
    How do you do that exactly? Are you using eBPF or something else? Also, for my ML workloads the most common bottleneck is GPU VRAM <-> RAM copies. Doesn't this dramatically increase latency? Or is it more like it increases latency on first data transfer, but as long as you dump everything into VRAM all at once at the beginning you're fine? I'd expect this wouldn't play super well with stuff like PyTorch data loaders, but would be curious to hear how you've faired when testing.
    - bmodel 2y ago
      We intercept api calls and use our own implementation to forward them to a remote machine. No eBPF (which I believe need to run in the kernel). As for latency, we've done a lot of work to minimize that as much as possible. You can see the performance we get running inference on BERT from huggingface here: https://youtu.be/qsOBFQZtsFM?t=64 https://youtu.be/qsOBFQZtsFM?t=64. It's still slower than local (mainly for training workloads) but not by as much as you'd expect. We're aiming to reach near parity in the next few months!
      - samstave 2y ago
        When you release a self-host version, what would be really neat would be to see it across HFT focused NICs that have huge TCP buffers... https://www.arista.com/assets/data/pdf/HFT/HFTTradingNetworkPrimer.pdf https://www.arista.com/assets/data/pdf/HFT/HFTTradingNetwork... Basically taking into account the large buffers and super-time-sensitive nature of HFT networking optimizations, I wonder if your TCP<-->GPU might benefit from both the HW and the learnings of NFT stylings?
      - ZeroCool2u 2y ago
        Got it. eBPF module run as part of the kernel, but they're still user space programs. I would would consider using a larger model for demonstrating inference performance as I have 7B models deployed on CPU at work, but GPU is still important training BERT size models.

by bmodel 2y ago

billconan 2y ago

does this mean you have a customized/dummy kernel gpu driver? will that cause system instability, say, if the network suddenly dropped?

bmodel 2y ago
We are not writing any kernel drivers, this runs entirely in userspace (this won't result in a crowdstrike level crash haha). Given that, if the network suddenly dropped then only the process using the GPU would fail.
- ZeroCool2u 2y ago
  How do you do that exactly? Are you using eBPF or something else? Also, for my ML workloads the most common bottleneck is GPU VRAM <-> RAM copies. Doesn't this dramatically increase latency? Or is it more like it increases latency on first data transfer, but as long as you dump everything into VRAM all at once at the beginning you're fine? I'd expect this wouldn't play super well with stuff like PyTorch data loaders, but would be curious to hear how you've faired when testing.
  - bmodel 2y ago
    We intercept api calls and use our own implementation to forward them to a remote machine. No eBPF (which I believe need to run in the kernel). As for latency, we've done a lot of work to minimize that as much as possible. You can see the performance we get running inference on BERT from huggingface here: https://youtu.be/qsOBFQZtsFM?t=64 https://youtu.be/qsOBFQZtsFM?t=64. It's still slower than local (mainly for training workloads) but not by as much as you'd expect. We're aiming to reach near parity in the next few months!
    - samstave 2y ago
      When you release a self-host version, what would be really neat would be to see it across HFT focused NICs that have huge TCP buffers... https://www.arista.com/assets/data/pdf/HFT/HFTTradingNetworkPrimer.pdf https://www.arista.com/assets/data/pdf/HFT/HFTTradingNetwork... Basically taking into account the large buffers and super-time-sensitive nature of HFT networking optimizations, I wonder if your TCP<-->GPU might benefit from both the HW and the learnings of NFT stylings?
    - ZeroCool2u 2y ago
      Got it. eBPF module run as part of the kernel, but they're still user space programs. I would would consider using a larger model for demonstrating inference performance as I have 7B models deployed on CPU at work, but GPU is still important training BERT size models.