4 ms·

Exactly. Not sure why it would be better to run Rust on Nvidia GPUs compared to actual CUDA code. I get the idea of added abstraction, but do think it becomes

Exactly. Not sure why it would be better to run Rust on Nvidia GPUs compared to actual CUDA code. I get the idea of added abstraction, but do think it becomes a bit jack-of-all-tradesey.

MuffinFlavored 1y ago
> Exactly. Not sure why it would be better to run Rust on Nvidia GPUs compared to actual CUDA code. You get to pull no_std Rust crates and they go to GPU instead of having to convert them to C++
rbanffy 1y ago
I think the idea is to allow developers to write a single implementation and have a portable binary that can run on any kind of hardware. We do that all the time - there are lots of code that chooses optimal code paths depending on runtime environment or which ISA extensions are available.
- kookamamie 1y ago
  Sure. The performance-purist in me would be very doubtful about the result's optimality, though.
  - littlestymaar 1y ago
    The performance purist don't use Cuda either though (that's why Deepseek used PTX directly). Everything is an abstraction and choosing the right level of abstraction for your usecase is a tradeoff between your engineering capacities and your performance needs.
    - LowLevelMahn 1y ago
      this Rust demo also uses PTX directly During the build, build.rs uses rustc_codegen_nvvm to compile the GPU kernel to PTX. The resulting PTX is embedded into the CPU binary as static data. The host code is compiled normally.
      - LegNeato 1y ago
        To be more technically correct, we compile to NVVM IR and then use NVIDIA's NVVM to convert it to PTX.
      - saagarjha 1y ago
        That’s not really the same thing; it compiles through PTX rather than using inline assembly.
        LegNeato 1y ago
        FYI, you can drop down into ptx if need be: https://github.com/Rust-GPU/Rust-CUDA/blob/aa7e61512788cc70200005eb318b7441a97fad6d/crates/cuda_std/src/mem.rs#L40 https://github.com/Rust-GPU/Rust-CUDA/blob/aa7e61512788cc702...
    - brandonpelfrey 1y ago
      The issue in my mind is that this doesn’t seem to include any of the critical library functionality specific eg to NVIDIA cards, think reduction operations across threads in a warp and similar. Some of those don’t exist in all hardware architectures. We may get to a point where everything could be written in one language but actually leveraging the hardware correctly still requires a bunch of different implementations, ones for each target architecture. The fact that different hardware has different features is a good thing.
      - rbanffy 1y ago
        The features missing hardware support can fall back to software implementations. In any case, ideally, the level of abstraction would be higher, with little application logic requiring GPU architecture awareness.
- pjmlp 1y ago
  Without the tooling though. Commendable effort, however just like people forget languages are ecosystems, they tend to forget APIs are ecosystems as well.
the__alchemist 1y ago
I think the sweet spot is: If your program is written in rust, use an abstraction like Cudarc to send and receive data from the GPU. Write normal CUDA kernels.
Ar-Curunir 1y ago
Because folks like to program in Rust, not CUDA
- tucnak 1y ago
  "Folks" as-in Rust stans, whom know very little about CUDA and what makes it nice in the first place, sure, but is there demand for Rust ports amongst actual CUDA programmers? I think not.
  - LegNeato 1y ago
    FYI, rust-cuda outputs nvvm so it can integrate with the existing cuda ecosystem. We aren't suggesting rewriting everything in Rust. Check the repo for crates that allow using existing stuff like cudnn and cuBLAS.
    - tucnak 1y ago
      I take it you're the maintainer. Firstly, congrats on the work done, for the open source people are a small crowd, and determination of Rust teams here is commendable. On the other hand, I'm struggling to see the unique value proposition. What is your motivation with Rust-GPU? Graphics or general-purpose computing? If it's the latter, at least from my POV, I would struggle to justify going up against a daunting umbrella project like this; in view of it likely culminating in layers upon layers of abstraction. Is the long-term goal here to have fun writing a bit of Rust, or upsetting the entrenched status quo of highly-concurrent GPU programming? There's a saying that goes along like "pleasing all is a lot like pleasing none," and intuitively I would guess it should apply here.
      - LegNeato 1y ago
        Thanks! I'm personally focused on compute, while other contributors are focused on graphics. I believe GPUs are the future of computing. I think the tooling, languages, and ecosystems of GPUs are very bad compared to CPUs. Partially because they are newer, partially because they are different, and partially because for some reason the expectations are so low. So I intend to upset the status quo.
JayEquilibria 1y ago
Good stuff. I have been thinking of learning Rust because of people here even though CUDA is what I care about. My abstractions though are probably best served by Pytorch and Julia so Rust is just a waste of time, FOR ME.

Exactly. Not sure why it would be better to run Rust on Nvidia GPUs compared to actual CUDA code. I get the idea of added abstraction, but do think it becomes a bit jack-of-all-tradesey.

MuffinFlavored 1y ago

> Exactly. Not sure why it would be better to run Rust on Nvidia GPUs compared to actual CUDA code. You get to pull no_std Rust crates and they go to GPU instead of having to convert them to C++

rbanffy 1y ago

I think the idea is to allow developers to write a single implementation and have a portable binary that can run on any kind of hardware. We do that all the time - there are lots of code that chooses optimal code paths depending on runtime environment or which ISA extensions are available.

kookamamie 1y ago
Sure. The performance-purist in me would be very doubtful about the result's optimality, though.
- littlestymaar 1y ago
  The performance purist don't use Cuda either though (that's why Deepseek used PTX directly). Everything is an abstraction and choosing the right level of abstraction for your usecase is a tradeoff between your engineering capacities and your performance needs.
  - LowLevelMahn 1y ago
    this Rust demo also uses PTX directly During the build, build.rs uses rustc_codegen_nvvm to compile the GPU kernel to PTX. The resulting PTX is embedded into the CPU binary as static data. The host code is compiled normally.
    - LegNeato 1y ago
      To be more technically correct, we compile to NVVM IR and then use NVIDIA's NVVM to convert it to PTX.
    - saagarjha 1y ago
      That’s not really the same thing; it compiles through PTX rather than using inline assembly.
      - LegNeato 1y ago
        FYI, you can drop down into ptx if need be: https://github.com/Rust-GPU/Rust-CUDA/blob/aa7e61512788cc70200005eb318b7441a97fad6d/crates/cuda_std/src/mem.rs#L40 https://github.com/Rust-GPU/Rust-CUDA/blob/aa7e61512788cc702...
  - brandonpelfrey 1y ago
    The issue in my mind is that this doesn’t seem to include any of the critical library functionality specific eg to NVIDIA cards, think reduction operations across threads in a warp and similar. Some of those don’t exist in all hardware architectures. We may get to a point where everything could be written in one language but actually leveraging the hardware correctly still requires a bunch of different implementations, ones for each target architecture. The fact that different hardware has different features is a good thing.
    - rbanffy 1y ago
      The features missing hardware support can fall back to software implementations. In any case, ideally, the level of abstraction would be higher, with little application logic requiring GPU architecture awareness.
pjmlp 1y ago
Without the tooling though. Commendable effort, however just like people forget languages are ecosystems, they tend to forget APIs are ecosystems as well.

the__alchemist 1y ago

I think the sweet spot is: If your program is written in rust, use an abstraction like Cudarc to send and receive data from the GPU. Write normal CUDA kernels.

Ar-Curunir 1y ago

Because folks like to program in Rust, not CUDA

tucnak 1y ago
"Folks" as-in Rust stans, whom know very little about CUDA and what makes it nice in the first place, sure, but is there demand for Rust ports amongst actual CUDA programmers? I think not.
- LegNeato 1y ago
  FYI, rust-cuda outputs nvvm so it can integrate with the existing cuda ecosystem. We aren't suggesting rewriting everything in Rust. Check the repo for crates that allow using existing stuff like cudnn and cuBLAS.
  - tucnak 1y ago
    I take it you're the maintainer. Firstly, congrats on the work done, for the open source people are a small crowd, and determination of Rust teams here is commendable. On the other hand, I'm struggling to see the unique value proposition. What is your motivation with Rust-GPU? Graphics or general-purpose computing? If it's the latter, at least from my POV, I would struggle to justify going up against a daunting umbrella project like this; in view of it likely culminating in layers upon layers of abstraction. Is the long-term goal here to have fun writing a bit of Rust, or upsetting the entrenched status quo of highly-concurrent GPU programming? There's a saying that goes along like "pleasing all is a lot like pleasing none," and intuitively I would guess it should apply here.
    - LegNeato 1y ago
      Thanks! I'm personally focused on compute, while other contributors are focused on graphics. I believe GPUs are the future of computing. I think the tooling, languages, and ecosystems of GPUs are very bad compared to CPUs. Partially because they are newer, partially because they are different, and partially because for some reason the expectations are so low. So I intend to upset the status quo.

JayEquilibria 1y ago

Good stuff. I have been thinking of learning Rust because of people here even though CUDA is what I care about. My abstractions though are probably best served by Pytorch and Julia so Rust is just a waste of time, FOR ME.