3 ms·

We are not writing any kernel drivers, this runs entirely in userspace (this won't result in a crowdstrike level crash haha). Given that, if the network sudden

by bmodel 2y ago

We are not writing any kernel drivers, this runs entirely in userspace (this won't result in a crowdstrike level crash haha). Given that, if the network suddenly dropped then only the process using the GPU would fail.

ZeroCool2u 2y ago
How do you do that exactly? Are you using eBPF or something else? Also, for my ML workloads the most common bottleneck is GPU VRAM <-> RAM copies. Doesn't this dramatically increase latency? Or is it more like it increases latency on first data transfer, but as long as you dump everything into VRAM all at once at the beginning you're fine? I'd expect this wouldn't play super well with stuff like PyTorch data loaders, but would be curious to hear how you've faired when testing.
- bmodel 2y ago
  We intercept api calls and use our own implementation to forward them to a remote machine. No eBPF (which I believe need to run in the kernel). As for latency, we've done a lot of work to minimize that as much as possible. You can see the performance we get running inference on BERT from huggingface here: https://youtu.be/qsOBFQZtsFM?t=64 https://youtu.be/qsOBFQZtsFM?t=64. It's still slower than local (mainly for training workloads) but not by as much as you'd expect. We're aiming to reach near parity in the next few months!
  - samstave 2y ago
    When you release a self-host version, what would be really neat would be to see it across HFT focused NICs that have huge TCP buffers... https://www.arista.com/assets/data/pdf/HFT/HFTTradingNetworkPrimer.pdf https://www.arista.com/assets/data/pdf/HFT/HFTTradingNetwork... Basically taking into account the large buffers and super-time-sensitive nature of HFT networking optimizations, I wonder if your TCP<-->GPU might benefit from both the HW and the learnings of NFT stylings?
  - ZeroCool2u 2y ago
    Got it. eBPF module run as part of the kernel, but they're still user space programs. I would would consider using a larger model for demonstrating inference performance as I have 7B models deployed on CPU at work, but GPU is still important training BERT size models.

by bmodel 2y ago

ZeroCool2u 2y ago

How do you do that exactly? Are you using eBPF or something else? Also, for my ML workloads the most common bottleneck is GPU VRAM <-> RAM copies. Doesn't this dramatically increase latency? Or is it more like it increases latency on first data transfer, but as long as you dump everything into VRAM all at once at the beginning you're fine? I'd expect this wouldn't play super well with stuff like PyTorch data loaders, but would be curious to hear how you've faired when testing.

bmodel 2y ago
We intercept api calls and use our own implementation to forward them to a remote machine. No eBPF (which I believe need to run in the kernel). As for latency, we've done a lot of work to minimize that as much as possible. You can see the performance we get running inference on BERT from huggingface here: https://youtu.be/qsOBFQZtsFM?t=64 https://youtu.be/qsOBFQZtsFM?t=64. It's still slower than local (mainly for training workloads) but not by as much as you'd expect. We're aiming to reach near parity in the next few months!
- samstave 2y ago
  When you release a self-host version, what would be really neat would be to see it across HFT focused NICs that have huge TCP buffers... https://www.arista.com/assets/data/pdf/HFT/HFTTradingNetworkPrimer.pdf https://www.arista.com/assets/data/pdf/HFT/HFTTradingNetwork... Basically taking into account the large buffers and super-time-sensitive nature of HFT networking optimizations, I wonder if your TCP<-->GPU might benefit from both the HW and the learnings of NFT stylings?
- ZeroCool2u 2y ago
  Got it. eBPF module run as part of the kernel, but they're still user space programs. I would would consider using a larger model for demonstrating inference performance as I have 7B models deployed on CPU at work, but GPU is still important training BERT size models.