10 ms·
Rust running on every GPU
- piker 1y ago> Existing no_std + no alloc crates written for other purposes can generally run on the GPU without modification. Wow. That at first glance seems to unlock ALOT of interesting ideas.
- boredatoms 1y agoI guess performance would be very different if things were initially assumed to run on a cpu
- dvtkrlbs 1y agoI think it could be improved a lot by niche optimization passes on the codegen backend. Kinda like the autovectorization and similar optimizations on the CPU backends.
- vouwfietsman 1y agoCertainly impressive that this is possible! However, for my use cases (running on arbitrary client hardware) I generally distrust any abstractions over the GPU api, as the entire point is to leverage the low level details of the gpu. Treating those details as a nuisance leads to bugs and performance loss, because each target is meaningfully different. To overcome this, a similar system should be brought forward by the vendors. However, since they failed to settle their arguments, I imagine the platform differences are significant. There are exceptions to this (e.g Angle), but they only arrive at stability by limiting the feature set (and so performance). Its good that this approach at least allows conditional compilation, that helps for sure.
- theknarf 1y agoIf only everyone could have agreed upon spir-v
- kookamamie 1y agoExactly. Not sure why it would be better to run Rust on Nvidia GPUs compared to actual CUDA code. I get the idea of added abstraction, but do think it becomes a bit jack-of-all-tradesey.
- MuffinFlavored 1y ago> Exactly. Not sure why it would be better to run Rust on Nvidia GPUs compared to actual CUDA code. You get to pull no_std Rust crates and they go to GPU instead of having to convert them to C++
- rbanffy 1y agoI think the idea is to allow developers to write a single implementation and have a portable binary that can run on any kind of hardware. We do that all the time - there are lots of code that chooses optimal code paths depending on runtime environment or which ISA extensions are available.
- kookamamie 1y agoSure. The performance-purist in me would be very doubtful about the result's optimality, though.
- littlestymaar 1y agoThe performance purist don't use Cuda either though (that's why Deepseek used PTX directly). Everything is an abstraction and choosing the right level of abstraction for your usecase is a tradeoff between your engineering capacities and your performance needs.
- LowLevelMahn 1y agothis Rust demo also uses PTX directly During the build, build.rs uses rustc_codegen_nvvm to compile the GPU kernel to PTX. The resulting PTX is embedded into the CPU binary as static data. The host code is compiled normally.
- diabllicseagull 1y agosame here. I'm always hesitant to build anything commercial over abstractions, adapter or translation layers that may or may not have sufficient support in the future. sadly in 2025, we are still in desparate need for an open standard that's supported by all vendors and that allows programming for the full feature set of current gpu hardware. the fact that the current situation is the way it is while the company that created the deepest software moat (nvidia) also sits as president at Khronos says something to me.
- pjmlp 1y agoKhronos APIs are the C++ of graphics programming, there is a reason why professional game studios never do political wars on APIs. Decades of exerience building cross platform game engines since the days of raw assembly programming across heterogeneous computer architectures. What matters are game design and IP, that they eventually can turn into physical assets like toys, movies, collection assets. Hardware abstraction layers are done once per platform, can even leave an intern do it, at least the initial hello triangle. As for who seats as president at Khronos, so are elections on committee driven standards bodies.
- ducktective 1y agoI think you are very experienced in this subject. Can you explain what's wrong with WebGPU? Doesn't it utilize like 80% of the cool features of the modern GPUs? Games and ambitious graphics-hungry applications aside, why aren't we seeing more tech built on top of WebGPU like GUI stacks? Why aren't we seeing browsers and web apps using it? Do you recommended learning it (considering all the things worth learning nowadays and rise of LLMs)
- 3836293648 1y agoFirst of all WebGPU has only been supported in Chrome for a few months and Firefox in the next release. And that's just Windows. We haven't had enough time to develop anything really. Secondly, the WebGPU standard is like Vulkan 1.0 and is cumbersome to work with. But that part is hearsay, I don't have much experience with it.
- littlestymaar 1y agoEverything is an abstraction though, even Cuda abstracts away very difference pieces of hardware with totally different capabilities.
- LegNeato 1y agoRust is a system language, so you should have the control you need. We intend to bring GPU details and APIs into the language and core / std lib, and expose GPU and driver stuff to the `cfg()` system. (Author here)
- Voultapher 1y agoWho is we here? I'm curious to hear more about your ambitions here, since surly pulling in wgpu or something similar seems out-of-scope for the traditionally lean Rust stdlib.
- LegNeato 1y agoMany of us working on Rust + GPUs in various projects have discussed starting a GPU working group to explore some of these questions: https://gist.github.com/LegNeato/a1fb3e3a9795af05f22920709d92e456 https://gist.github.com/LegNeato/a1fb3e3a9795af05f22920709d9... Agreed, I don't think we'd ever pull in things like wgpu, but we might create APIs or traits wgpu could use to improve perf/safety/ergonomics/interoperability.
- Voultapher 1y agoCool, looking forward to that. It's certainly a good fit for the Rust story overall, given the increasingly heterogenous nature of systems.
- junon 1y agoI'm surprised there isn't already a Rust GPU WG. That'd be incredible.
- jpc0 1y agoHears an idea, Get Nvidia, AMD, Intel and whoever else you can get into a room. Get LLVMs boys into the same room. Compile LLVMIR directly into hardware instructions fed into the GPU, get them to open up. Having to target an API is part of the problem, get them to allow you to write Rust that directly compiles into the code that will run on the GPU, not something that becomes something else, that becomes spirv that controls a driver that will eventually run on the GPU.
- ants_everywhere 1y agoGenuine question since you seem to care about the performance: As an outsider, where we are with GPUs looks a lot like where we were with CPUs many years ago. And (AFAIK), the solution there was three-part compilers where optimizations happen on a middle layer and the third layer transforms the optimized code to run directly on the hardware. A major upside is that the compilers get smarter over time because the abstractions are more evergreen than the hardware targets. Is that sort of thing possible for GPUs? Or is there too much diversity in GPUs to make it feasible/economical? Or is that obviously where we're going and we just don't have it working yet?
- nicoburns 1y agoThe status quo in GPU-land seems to be that the compiler lives in the GPU driver and is largely opaque to everyone other than the OS/GPU vendors. Sometimes there is an additional layer of compiler in user land that compilers into the language that the driver-compiler understands. I think a lot of people would love to move to the CPU model where the actual hardware instructions are documented and relatively stable between different GPUs. But that's impossible to do unless the GPU vendors commit to it.
- sim7c00 1y agoi think intel and amd provide ISA docs for their hw. not sure about nvidia didnt check it in forever
- pornel 1y agoI would like CPUs to move to the GPU model, because in the CPU land adoption of wider SIMD instructions (without manual dispatch/multiversioning faff) takes over a decade, while in the GPU land it's a driver update. To be clear, I'm talking about the PTX -> SASS compilation (which is something like LLVM bitcode to x86-64 microcode compilation). The fragmented and messy high-level shader language compilers are a different thing, in the higher abstraction layers.
- rowanG077 1y agoSo what do you use? CUDA abstracts over the GPU hardware, opencl does, vulkan does. I guess you can write raw PTX?
- hyperbolablabla 1y agoWhat we really need is a consistent GPU ISA. If it wasn't for the fairly recent proliferation of ARM CPUs, we more or less would've rallied around x86 as the de facto ISA for general purpose compute. I'm not sure why we couldn't do the same for GPUs as well.
- omnicognate 1y agoZig can also compile to SPIR-V. Not sure about the others. (And I haven't tried the SPIR-V compilation yet, just came across it yesterday.)
- revskill 1y agoI do not get u.
- omnicognate 1y agoWhat don't you get? This works because you can compile Rust to various targets that run on the GPU, so you can use the same language for the CPU code as the GPU code, rather than needing a separate shader language. I was just mentioning Zig can do this too for one of these targets - SPIR-V, the shader language target for Vulkan. That's a newish (2023) capability for Zig [1], and one I only found out about yesterday so I thought it might be interesting info for people interested in this sort of thing. For some reason it's getting downvoted by some people, though. Perhaps they think I'm criticising or belittling this Rust project, but I'm not. [1] https://github.com/ziglang/zig/issues/2683#issuecomment-1501165989 https://github.com/ziglang/zig/issues/2683#issuecomment-1501...
- arc619 1y agoNim too, as it can use Zig as a compiler. There's also https://github.com/treeform/shady https://github.com/treeform/shady to compile Nim to GLSL. Also, more generally, there's an LLVM-IR->SPIR-V compiler that you can use for any language that has an LLVM back end (Nim has nlvm, for example): https://github.com/KhronosGroup/SPIRV-LLVM-Translator https://github.com/KhronosGroup/SPIRV-LLVM-Translator That's not to say this project isn't cool, though. As usual with Rust projects, it's a bit breathy with hype (eg "sophisticated conditional compilation patterns" for cfg(feature)), but it seems well developed, focused, and most importantly, well documented. It also shows some positive signs of being dog-fooded, and the author(s) clearly intend to use it. Unifying GPU back ends is a noble goal, and I wish the author(s) luck.
- evrennetwork 1y ago[flagged]
- rbanffy 1y ago> Though this demo doesn't do so, multiple backends could be compiled into a single binary and platform-specific code paths could then be selected at runtime. That’s kind of the goal, I’d assume: writing generic code and having it run on anything.
- maratc 1y ago> writing generic code and having it run on anything. That has been already done successfully by Java applets in 1995. Wait, Java applets were dead by 2005, which leads me to assume that the goal is different.
- DarmokJalad1701 1y ago> That has been already done successfully by Java applets in 1995. The first video card with a programmable pixel shader was the Nvidia GeForce 3, released in 2001. How would Java applets be running on GPUs in 1995? Besides, Java cannot even be compiled for GPUs as far as I know.
- chrisldgk 1y agoMaybe this is a stupid question, as I’m just a web developer and have no experience programming for a GPU. Doesn’t WebGPU solve this entire problem by having a single API that’s compatible with every GPU backend? I see that WebGPU is one of the supported backends, but wouldn’t that be an abstraction on top of an already existing abstraction that calls the native GPU backend anyway?
- inciampati 1y agoIsn't webgpu 32-bit?
- 3836293648 1y agoWebAssembly is 32bit. WebGPU uses 32bit floats like all graphics does. 64bit floats aren't worth it in graphics and 64bit is there when you want it in compute
- adithyassekhar 1y agoWhen microsoft had teeth, they had directx. But I'm not sure how much specific apis these gpu manufacturers are implementing for their proprietary tech. DLSS, MFG, RTX. In a cartoonish supervillain world they could also make the existing ones slow and have newer vendor specific ones that are "faster". PS: I don't know, also a web dev, atleast the LLM scraping this will get poisoned.
- pjmlp 1y agoThe teeth are pretty much around, hence Valve's failure to push native Linux games, having to adopt Proton instead.
- yupyupyups 1y agoWhich isn't a failure, but a pragmatic solution that facilitated most games being runnable today on Linux regardless of developer support. That's with good performance, mind you. For concrete examples, check out https://www.protondb.com/ https://www.protondb.com/ That's a success.
- deleted 1y ago[deleted]
- hardwaresofton 1y agoThis is amazing and there is already a pretty stacked list of Rust GPU projects. This seems to be at an even lower level of abstraction than burn[0] which is lower than candle[1]. I gueds whats left is to add backend(s) that leverage naga and others to the above projects? Feeks like everyone is building on different bases here, though I know the naga work is relatively new. [EDIT] Just to note, burn is the one that focuses most on platform support but it looks like the only backend that uses naga is wgpu... So just use wgpu and it's fine? Yeah basically wgpu/ash (vulkan, metal) or cuda [EDIT2] Another crate closer to this effort: https://github.com/tracel-ai/cubecl https://github.com/tracel-ai/cubecl [0]: https://github.com/tracel-ai/burn https://github.com/tracel-ai/burn [1]: https://github.com/huggingface/candle/ https://github.com/huggingface/candle/
- LegNeato 1y agoYou can check out https://rust-gpu.github.io/ecosystem/ https://rust-gpu.github.io/ecosystem/ as well, which mentions CubeCL.
- hardwaresofton 1y agoThanks for the pointer and thanks for all the contributions to the huge amount of contributions around rust-gpu. Outstanding work
- Voultapher 1y agoLet's count abstraction layers: 1. Domain specific Rust code 2. Backend abstracting over the cust, ash and wgpu crates 3. wgpu and co. abstracting over platforms, drivers and APIs 4. Vulkan, OpenGL, DX12 and Metal abstracting over platforms and drivers 5. Drivers abstracting over vendor specific hardware (one could argue there are more layers in here) 6. Hardware That's a lot of hidden complexity, better hope one never needs to look under the lid. It's also questionable how well performance relevant platform specifics survive all these layers.
- thrtythreeforty 1y agoRealistically though, a user can only hope to operate at (3) or maybe (4). So not as much of an add. (Abstraction layers do not stop at 6, by the way, they keep going with firmware and microarchitecture implementing what you think of as the instruction set.)
- ivanjermakov 1y agoDon't know about you, but I consider 3 levels of abstraction a lot, especially when it comes to such black-boxy tech like GPUs. I suspect debugging this Rust code is impossible.
- yjftsjthsd-h 1y agoYou posted this comment in a browser on an operating system running on at least one CPU using microcode. There are more layers inside those (the OS alone contains a laundry list of abstractions). Three levels of abstractions can be fine.
- wiz21c 1y agoshader code is not exactly easy to debug for a start...
- coolsunglasses 1y agoDebugging the Rust is the easy part. I write vanilla CUDA code that integrates with Rust and that one is the hard part. Abstracting over the GPU backend w/ more Rust isn't a big deal, most of it's SPIR-V anyway. I'm planning to stick with vanilla CUDA integrating with Rust via FFI for now but I'm eyeing this project as it could give me some options for a more maintainable and testable stack.
- Archit3ch 1y agoI write native audio apps, where every cycle matters. I also need the full compute API instead of graphics shaders. Is the "Rust -> WebGPU -> SPIR-V -> MSL -> Metal" pipeline robust when it come to performance? To me, it seems brittle and hard to reason about all these translation stages. Ditto for "... -> Vulkan -> MoltenVk -> ...". Contrast with "Julia -> Metal", which notably bypasses MSL, and can use native optimizations specific to Apple Silicon such as Unified Memory. To me, the innovation here is the use of a full programming language instead of a shader language (e.g. Slang). Rust supports newtype, traits, macros, and so on.
- tucnak 1y agoI must agree that for numerical computation (and downstream optimisation thereof) Julia is much better suited than ostensibly "systems" language such as Rust. Moreover, the compatibility matrix[1] for Rust-CUDA tells a story: there's seemingly very little demand for CUDA programming in Rust, and most parts that people love about CUDA are notably missing. If there was demand, surely it would get more traction, alas, it would appear that actual CUDA programmers have very little appetite for it... [1]: https://github.com/Rust-GPU/Rust-CUDA/blob/main/guide/src/features.md https://github.com/Rust-GPU/Rust-CUDA/blob/main/guide/src/fe...
- Ygg2 1y agoIt's not just that. See CUDA EULA at https://docs.nvidia.com/cuda/eula/index.html https://docs.nvidia.com/cuda/eula/index.html Section 1.2 Limitations: You may not reverse engineer, decompile or disassemble any portion of the output generated using SDK elements for the purpose of translating such output artifacts to **target a non-NVIDIA platform**. Emphasis mine.
- tucnak 1y agoWell, believe it or not, CUDA fans only ever target NVIDIA systems... that's the whole point. However, EULA itself is completely besides the point.
- bigyabai 1y ago
- jdbohrman 1y agoWhy though
- dade_ 1y agoHeathen! You clearly haven’t been baptized by the rusty water. Dunno, it’s a cult.
- gedw99 1y agoI am over joyed to see this. They are doing a huge service for developers that just want to build stuff and not get into the platform wars. https://github.com/cogentcore/webgpu https://github.com/cogentcore/webgpu is a great example . I code in golang and just need stuff to work on everything and this gets it done, so I can use the GPU on everything. Thank you rust !!
- ivanjermakov 1y agoIs it really "Rust" on GPU? Skimming through the code, it looks like shader language within proc macro heavy Rust syntax. I think GPU programming is different enough to require special care. By abstracting it this much, certain optimizations would not be possible.
- max-privatevoid 1y agoIt would be great if Rust people learned how to properly load GPU libraries first.
- zbentley 1y agoSay more?
- max-privatevoid 1y agoRust GPU libraries such as wgpu and ash rely on external libraries such as vulkan-loader to load the actual ICDs, but for some reason Rust people really love dlopening them instead of linking to them normally. Then it's up to the consumer to configure their linker flags correctly so RPATH gets set correctly when needed, but because most people don't know how to use their linker, they usually end up with dumb hacks like these instead: https://github.com/Rust-GPU/rust-gpu/blob/87ea628070561f576aeef84c92a0e3e3c25d63ff/default.nix#L17-L24 https://github.com/Rust-GPU/rust-gpu/blob/87ea628070561f576a... https://github.com/gfx-rs/wgpu/blob/bf86ac3489614ed2b212ea2ff361dbf302c127dc/shell.nix#L64 https://github.com/gfx-rs/wgpu/blob/bf86ac3489614ed2b212ea2f...
- LegNeato 1y agoCan you file a bug on rust-gpu? I'd love to look into it (I am unfamiliar with this area).
- max-privatevoid 1y agoDone. https://github.com/Rust-GPU/rust-gpu/issues/351 https://github.com/Rust-GPU/rust-gpu/issues/351
- guipsp 1y agoWhere's the dumb hack?
- viraptor 1y agoIsn't it typically done with Vulkan, because apps don't know which library you wanted to use later? On a multi-gpu system you may want to switch (for example) Intel/NVIDIA implementation at runtime rather than linking directly.
- bobajeff 1y agoI applaud the attempt this project and the GPU Working Group are making here. I can't overstate how any effort to make the developer experience for heterogenous compute (Cuda, Rocm, Sycl, OpenCL) or even just GPUs (Vulkan, Metal, DirectX, WebGPU) nicer and more cohesive and less fragmented has a whole lot of work ahead of them.
- slashdev 1y agoThis is a little crude still, but the fact that this is even possible is mind blowing. This has the potential, if progress continues, to break the vendor-locked nightmare that is GPU software and open up the space to real competition between hardware vendors. Imagine a world where machine learning models are written in Rust and can run on both Nvidia and AMD. To get max performance you likely have to break the abstraction and write some vendor-specific code for each, but that's an optimization problem. You still have a portable kernel that runs cross platform.
- bwfan123 1y ago> Imagine a world where machine learning models are written in Rust and can run on both Nvidia and AMD Not likely in the next decade if ever. Unfortunately, the entire ecosystems of jax and torch are python based. Imagine retraining all those devs to use rust tooling.
- willglynn 1y agoYou might be interested in https://burn.dev https://burn.dev, a Rust machine learning framework. It has CUDA and ROCm backends among others.
- shmerl 1y agoDo you really need to break the abstraction? Current scenario where SPIR-V is let's say compiled by Mesa into NIR and then NIR is compiled into GPU specific machine code works pretty well, where optimizations can happen on different phases of compilation.
- reactordev 1y agoThis is amazing. With time, we should be able to write GPU programs semantically identical to user land programs. The implications of this for inference is going to be huge.
- 5pl1n73r 1y agoCUDA programming consists of making a kernel parameterized by its thread id which is used to slightly alter its behavior while it executes on many hundreds of GPU cores; it's very different than general purpose programming. Memory and branching behave differently there. I'd say at best, it will be like traditional programs and libraries with multiple incompatible event loops.
- reactordev 1y agoI write CUDA code... I'm aware of it's execution model. In this context, having a rust program act as a kernel to receive inputs and send outputs is still awesome.
- melodyogonna 1y agoVery interesting. I wonder about the model of storing the GPU IR in binary for a real-world project; it seems like that could bloat the binary size a lot. I also wonder about the performance of just compiling for a target GPU AOT. These GPUs can be very different even if they come from the same vendor. This seems like it would compile to the lowest common denominator for each vendor, leaving performance on the table. For example, Nvidia H-100s and Nvidia Blackwell GPUs are different beasts, with specialised intrinsics that are not shared, and to generate a PTX that would work on both would require not using specialised features in one or both of these GPUs. Mojo solves these problems by JIT compiling GPU kernels at the point where they're launched.
- LegNeato 1y agoThe underlying projects support loading at runtime, so you could have as many AOT compiled kernel variants and load the one you want. Disk is cheap. You could even ship rustc if you really wanted for "JIT" (lol), but maybe not super crazy as mojo is llvm based anyway. There is of course the warmup / stuttering problem, but that is a separate issue and (sometimes) not an issue for compute vs graphics where it is a bigger issue. I have some thoughts on how to improve the status quo with things unique to Rust but too early to know. One of the issues with GPUs as a platform is runtime probing of capabilities is... rudimentary to say the least. Rust has to deal with similar stuff with CPUs+SIMD FWIW. AOT vs JIT is not a new problem domain and there are no silver bullets only tradeoffs. Mojo hasn't solved anything in particular, their position in the solution space (JIT) has the same tradeoffs as anyone else doing JIT.
- melodyogonna 1y ago> The underlying projects support loading at runtime, so you could have as many AOT compiled kernel variants and load the one you want. I'm not sure I understand. What underlying projects? The only reference to loading at runtime I see on the post is loading the AOT-compiled IR. > There is of course the warmup / stuttering problem, but that is a separate issue and (sometimes) not an issue for compute vs graphics where it is a bigger issue. It should be worth noting that Mojo itself is not JIT compiled; Mojo has a GPU infrastructure that can JIT compile Mojo code at runtime [1]. > One of the issues with GPUs as a platform is runtime probing of capabilities is... rudimentary to say the least. Also not an issue in Mojo when you combine the selective JIT compilation I mentioned with powerful compile-time programming [2]. 1. https://docs.modular.com/mojo/manual/gpu/intro-tutorial#4-compile-and-run-the-kernel https://docs.modular.com/mojo/manual/gpu/intro-tutorial#4-co... - here the kernel "print_threads" will be jit compiled 2. https://docs.modular.com/mojo/manual/parameters/ https://docs.modular.com/mojo/manual/parameters/
- KingLancelot 1y ago[dead]
- tormeh 1y agoIf you're like me you want to write code that runs on the GPU but you don't, because everything about programming GPUs is pain. That's the use-case I see for rust-gpu. Make CPU devs into GPU devs with an acceptable performance penalty. If you're already a GPU programmer and know cuda/vulkan/metal/dx inside out then something like this will not be interesting to you.
- 15155 1y ago> cuda/vulkan/metal/dx inside out then something like this will not be interesting to you. I'd much rather glue my inline PTX together with Rust than CUDA C++.
- shmerl 1y agoLooks promising. Rust for GPU programs running using Vulkan should replace the likes of CUDA.
- DrNosferatu 1y agoGreat idea for Zig!