6 ms·

The race to build a distributed GPU runtime

79 points by jonbaer 1y ago

varelse 1y ago
[dead]
KaiserPro 1y ago
One thing thats not addressed here is that the bigger you scale your shared memory cluster the closer to 100% chance that one node fucks up and corrupts your global memory space. Currently the fastest way to get data from node a to node b is to RDMA it. which means that any node can inject anything into your memory space. I'm not really sure how Theseus guards against that.
- buildbot 1y ago
  I’m not sure any system prevents RDMA from ruining your day :( Back in grad school I remember we did something fairly simple but clearly illegal and wedged the machine so bad the out of band management also went down!
  - KaiserPro 1y ago
    > wedged the machine so bad the out of band management also went down! Now thats living the dream of a shared cluster! This is hazy now, but I do remember a massive outage of a lustre cluster, which I think was because there was a dodgy node injecting crap into everyone's memory space via the old lustre fast filesystem kernel driver. I think they switched to NFS export nodes after that. (for the render farm and desktops at least.)
jauntywundrkind 1y ago
Lot of hype, but man does Voltron Data keep blowing me away with what they bring out. Mad respect. > There’s a strong argument to be made that RAPIDS cuDF/RAPIDS libcudf drives NVIDIA’s CUDA-X Data Processing stack, from ETL (NVTabular) and SQL (BlazingSQL) to MLOps/security (Morpheus) and Spark acceleration (cuDF-Java). Yeah this seems like the core indeed, libcudf. Focus here is on TCP & GPUDirect (Nvidia's pci-p2p, letting for example RDMA without CPU involvement across a full GPU -> NIC -> switch -> nic -> GPU happen). Personally it feels super dangerous to just trust Nvidia on all of this, to just buy the solution available. Pytorch nicely sees this somewhat, adopted & took over Facebook/Meta's gloo project, which wraps a lot of the rdma efforts. But man there's just so so many steps ahead that Theseus is here with figuring out & planning what to do with these capabilities, these ultra efficient links, figuring out how to not need to use them if possible! The coordination problems keep growing in computing. I think of RISC-V with its arbitrary vector-based alternative to conventional x86 simd, going from a specific instruction for each particular operation to instructions being parameterized over different data lengths & types. https://github.com/pytorch/gloo https://github.com/pytorch/gloo I'd really like to see a concerted effort to around Ultra Ethernet emerge, fast. Hardware isnt really available, and it's going to start out being absurdly expensive. But Ultra Ethernet looks like a lovely mix of collision-less credit-based Infiniband RDMA and Ethernet, with lots of other niceties (transport security). Deployments just starting (AMD Pensando Pollara 400 at Oracle). I worry that without broader availability & interest, without mass saturation, AI is going to stay stuck on libcudf forever; getting hardware out there & getting software stacos using it is a chicken & egg problem that big players need to spend real effort accelerating UET or else. https://www.tomshardware.com/networking/amd-deploys-its-first-ultra-ethernet-ready-network-card-pensando-pollara-provides-up-to-400-gbps-performance https://www.tomshardware.com/networking/amd-deploys-its-firs...
- latchkey 1y ago
  Our MI300x boxes have had 8x400G Thor2 RDMA working great for a year now.
anonymousDan 1y ago
Can someone tell me if the challenges the article describes and indeed the frameworks they mention are mostly relevant for training or also for inference?
- benreesman 1y ago
  The fast interconnect between nodes has aaplications in inference at scale (big KV caches and other semi-durable state, multi-node tensor parallelism on mega models). But this article in particular is emphasizing extreme performance ambitions for columnar data processing with hardware acceleration. Relevant to many ML training scenarios, but also other kinds of massive MapReduce-style (or at least scale) workloads. There are lots of applications of "magic massive petabyte plus DataFrame" (which is not I think solved in the general case).
smj-edison 1y ago
This reminds me a lot of Seymour Cray's two maxims of supercomputing: get the data where it needs to be when it needs to be there, and get the heat out. Still seems to apply today!
- CamperBob2 1y ago
  Calls to mind his other famous quote, "Would you rather plow your field with two strong oxen or 1024 chickens?" How about ten billion chickens?
  - smj-edison 1y ago
    Yeah. I feel like he's still partially vindicated with things like the dragonfly topology, as a lot of problems don't nicely map onto a 2D or 3D topology (so longest distance is still the limiting factor). But the chicken approach certainly scales better, and I feel like since Cray's time there are more local-aware algorithms around.
RachelF 1y ago
I find it odd that given the billions of dollars involved, no competitor has managed to replicate the functions of CUDA. Is it that hard to do, or is the software lock-in so great?
- deleted 1y ago
  [deleted]
- ronsor 1y ago
  The problem is that CUDA is tightly integrated with NVIDIA hardware. You don't just have to replicate CUDA (which is a lot of tedious work at best), but you also need the hardware to run your "I can't believe it's not CUDA"
- shmerl 1y ago
  A better question is why there is no stronger push for a nicer GPU language that's not tied to any particular GPU and serves any purpose of GPU usage (whether it's compute or graphics). I mean efforts like rust-gpu: https://github.com/Rust-GPU/rust-gpu/ https://github.com/Rust-GPU/rust-gpu/ Combine such language with Vulkan (using Rust as well) and why would you need CUDA?
  - bee_rider 1y ago
    I think Intel Fortran has some ability to offload to their GPUs now. And Nvidia has some stuff to run(?) CUDA from Fortran. Probably just needs a couple short decades of refinement…
    - pjmlp 1y ago
      One of the reasons CUDA won over OpenCL, was that NVidia, contrary to Khronos, saw a value in helping those HPC researchers move their Fortran code into the GPU. Hence they bought PGI, and improved their compiler. Intel eventually did the same with Open API (which isn't plain OpenCL, rather an extension with Intel goodies). I was on a Khronos webminar where the panel showed disbelief why anyone would care about Fortran, oh well.
JonChesterfield 1y ago
The underlying problem here is real and legitimately difficult. Shunting data around a cluster (ideally as parts of it fall over) to minimise overall time, in an application independent fashion, is a definable dataflow problem and also a serious discrete optimisation challenge. The more compute you spend on trying to work out where to move the data around, the less you have left over for the application. Also tricky working out what the data access patterns seem to be. Very like choosing how much of the runtime budget to spend on a JIT compiler. This _should_ breakdown as running optimised programs on their runtime makes things worse and running less-carefully-structured ones makes things better, where many programs out there turn out to be either quite naive or obsessively optimised for an architecture that hasn't existed for decades. I'd expect this runtime to be difficult to build but with high value on success. Interesting project, thanks for posting it.
up2isomorphism 1y ago
As of today GPU is just too expensive for data processing. The direction they took makes it a very hard sell.
tucnak 1y ago
I don't understand, doesn't kauldron[0] already exist? [0] https://github.com/google-research/kauldron https://github.com/google-research/kauldron
danielz4tp5 1y ago
[dead]

79 points by jonbaer 1y ago

[dead]

One thing thats not addressed here is that the bigger you scale your shared memory cluster the closer to 100% chance that one node fucks up and corrupts your global memory space. Currently the fastest way to get data from node a to node b is to RDMA it. which means that any node can inject anything into your memory space. I'm not really sure how Theseus guards against that.

buildbot 1y ago
I’m not sure any system prevents RDMA from ruining your day :( Back in grad school I remember we did something fairly simple but clearly illegal and wedged the machine so bad the out of band management also went down!
- KaiserPro 1y ago
  > wedged the machine so bad the out of band management also went down! Now thats living the dream of a shared cluster! This is hazy now, but I do remember a massive outage of a lustre cluster, which I think was because there was a dodgy node injecting crap into everyone's memory space via the old lustre fast filesystem kernel driver. I think they switched to NFS export nodes after that. (for the render farm and desktops at least.)

jauntywundrkind 1y ago

Lot of hype, but man does Voltron Data keep blowing me away with what they bring out. Mad respect. > There’s a strong argument to be made that RAPIDS cuDF/RAPIDS libcudf drives NVIDIA’s CUDA-X Data Processing stack, from ETL (NVTabular) and SQL (BlazingSQL) to MLOps/security (Morpheus) and Spark acceleration (cuDF-Java). Yeah this seems like the core indeed, libcudf. Focus here is on TCP & GPUDirect (Nvidia's pci-p2p, letting for example RDMA without CPU involvement across a full GPU -> NIC -> switch -> nic -> GPU happen). Personally it feels super dangerous to just trust Nvidia on all of this, to just buy the solution available. Pytorch nicely sees this somewhat, adopted & took over Facebook/Meta's gloo project, which wraps a lot of the rdma efforts. But man there's just so so many steps ahead that Theseus is here with figuring out & planning what to do with these capabilities, these ultra efficient links, figuring out how to not need to use them if possible! The coordination problems keep growing in computing. I think of RISC-V with its arbitrary vector-based alternative to conventional x86 simd, going from a specific instruction for each particular operation to instructions being parameterized over different data lengths & types. https://github.com/pytorch/gloo https://github.com/pytorch/gloo I'd really like to see a concerted effort to around Ultra Ethernet emerge, fast. Hardware isnt really available, and it's going to start out being absurdly expensive. But Ultra Ethernet looks like a lovely mix of collision-less credit-based Infiniband RDMA and Ethernet, with lots of other niceties (transport security). Deployments just starting (AMD Pensando Pollara 400 at Oracle). I worry that without broader availability & interest, without mass saturation, AI is going to stay stuck on libcudf forever; getting hardware out there & getting software stacos using it is a chicken & egg problem that big players need to spend real effort accelerating UET or else. https://www.tomshardware.com/networking/amd-deploys-its-first-ultra-ethernet-ready-network-card-pensando-pollara-provides-up-to-400-gbps-performance https://www.tomshardware.com/networking/amd-deploys-its-firs...

latchkey 1y ago
Our MI300x boxes have had 8x400G Thor2 RDMA working great for a year now.

anonymousDan 1y ago

Can someone tell me if the challenges the article describes and indeed the frameworks they mention are mostly relevant for training or also for inference?

benreesman 1y ago
The fast interconnect between nodes has aaplications in inference at scale (big KV caches and other semi-durable state, multi-node tensor parallelism on mega models). But this article in particular is emphasizing extreme performance ambitions for columnar data processing with hardware acceleration. Relevant to many ML training scenarios, but also other kinds of massive MapReduce-style (or at least scale) workloads. There are lots of applications of "magic massive petabyte plus DataFrame" (which is not I think solved in the general case).

smj-edison 1y ago

This reminds me a lot of Seymour Cray's two maxims of supercomputing: get the data where it needs to be when it needs to be there, and get the heat out. Still seems to apply today!

CamperBob2 1y ago
Calls to mind his other famous quote, "Would you rather plow your field with two strong oxen or 1024 chickens?" How about ten billion chickens?
- smj-edison 1y ago
  Yeah. I feel like he's still partially vindicated with things like the dragonfly topology, as a lot of problems don't nicely map onto a 2D or 3D topology (so longest distance is still the limiting factor). But the chicken approach certainly scales better, and I feel like since Cray's time there are more local-aware algorithms around.

RachelF 1y ago

I find it odd that given the billions of dollars involved, no competitor has managed to replicate the functions of CUDA. Is it that hard to do, or is the software lock-in so great?

deleted 1y ago
[deleted]
ronsor 1y ago
The problem is that CUDA is tightly integrated with NVIDIA hardware. You don't just have to replicate CUDA (which is a lot of tedious work at best), but you also need the hardware to run your "I can't believe it's not CUDA"
shmerl 1y ago
A better question is why there is no stronger push for a nicer GPU language that's not tied to any particular GPU and serves any purpose of GPU usage (whether it's compute or graphics). I mean efforts like rust-gpu: https://github.com/Rust-GPU/rust-gpu/ https://github.com/Rust-GPU/rust-gpu/ Combine such language with Vulkan (using Rust as well) and why would you need CUDA?
- bee_rider 1y ago
  I think Intel Fortran has some ability to offload to their GPUs now. And Nvidia has some stuff to run(?) CUDA from Fortran. Probably just needs a couple short decades of refinement…
  - pjmlp 1y ago
    One of the reasons CUDA won over OpenCL, was that NVidia, contrary to Khronos, saw a value in helping those HPC researchers move their Fortran code into the GPU. Hence they bought PGI, and improved their compiler. Intel eventually did the same with Open API (which isn't plain OpenCL, rather an extension with Intel goodies). I was on a Khronos webminar where the panel showed disbelief why anyone would care about Fortran, oh well.

JonChesterfield 1y ago

The underlying problem here is real and legitimately difficult. Shunting data around a cluster (ideally as parts of it fall over) to minimise overall time, in an application independent fashion, is a definable dataflow problem and also a serious discrete optimisation challenge. The more compute you spend on trying to work out where to move the data around, the less you have left over for the application. Also tricky working out what the data access patterns seem to be. Very like choosing how much of the runtime budget to spend on a JIT compiler. This _should_ breakdown as running optimised programs on their runtime makes things worse and running less-carefully-structured ones makes things better, where many programs out there turn out to be either quite naive or obsessively optimised for an architecture that hasn't existed for decades. I'd expect this runtime to be difficult to build but with high value on success. Interesting project, thanks for posting it.

up2isomorphism 1y ago

As of today GPU is just too expensive for data processing. The direction they took makes it a very hard sell.

tucnak 1y ago

I don't understand, doesn't kauldron[0] already exist? [0] https://github.com/google-research/kauldron https://github.com/google-research/kauldron

danielz4tp5 1y ago

[dead]