5 ms·
Alternative(s) to run CUDA on non-Nvidia hardware
- pjmlp 3mo agoMost of these "alternatives" focus on CUDA C++, and overlook what actually makes CUDA interesting. Already in 2020, https://developer.nvidia.com/blog/cuda-refresher-the-gpu-computing-ecosystem https://developer.nvidia.com/blog/cuda-refresher-the-gpu-com...
- msond 3mo agoWe're actually targeting all of it, and not just CUDA C++.
- pjmlp 3mo agoIncluding stuff like Fortran, Haskell, Java, .NET via PTX, Python JIT, IDE tooling integration with major IDEs, graphical GPU debugging and profiling, libraries and co? Then I guess all the best.
- zorked 3mo agoThis post has some serious peanut-gallery vibes.
- pjmlp 3mo agoPeanut-gallery is happily using CUDA, and needs actual sound reasons to move.
- account42 3mo agoThen the peanut gallery has nothing to complain when Nvidia jacks up prices.
- pjmlp 3mo agoDo you see me complaining? Here is a tip, you don't always need to suffer from FOMO and get the very latest model card. In fact, contrary to the competition, one can play with CUDA even on laptops, go figure.
- anon291 3mo ago> In fact, contrary to the competition, one can play with CUDA even on laptops, go figure. This is the part people don't get. You can program cuda anywhere on any Nvidia card, unlike other companies' chips you don't need a data center gpu to have full programmability. It's been this way for over a decade
- pjmlp 3mo agoYes, this is how they get students hooked on CUDA.
- anon291 3mo agoThey don't though. Money for ai is pretty much free in America to anyone who can demonstrate a modicum of competence in the field. The only people who are without access are students or hobbyists really.
- embedding-shape 3mo agoAmbitious but neat, good luck if nothing else :) If you were to guess, when do you think your Nsight Compute alternative might be ready with your own toolchain?
- msond 3mo agoA guess would be some time next year — since our public launch our focus has generally been on API coverage and increasingly recently, on performance. While performance improvements will always remain a target, we're soon at full coverage of the core CUDA APIs and will be shifting an increasing amount of effort towards developer tooling.
- ychen306 3mo agoHow do you deal with target-specific inline asm like tcgen05.mma?
- msond 3mo agoWe haven’t yet released support for tcgen05, but we’ll deal with it the same way we deal with other inline PTX: parsing it and converting it to target-appropriate instructions together with the rest of the program. This is something we’ve done already for the hopper-class tensorcore instructions, and the blackwell ones will map similarly, though likely with a kernel launch involved.
- mschuetz 3mo ago> Ease of programming and a giant leap in performance is one of the key reasons for the CUDA platform’s widespread adoption This, so much. Other platforms continue to ignore developer UX, but it's one of the main things that get's new users onboard and keeps old users around.
- maxloh 3mo agoThere is also ZLUDA, which is open source and works on pre-compiled binaries. https://github.com/vosen/ZLUDA https://github.com/vosen/ZLUDA
- DiabloD3 3mo agoIts easier to just get rid of your legacy code entirely and use Vulkan for compute, or have your compiler emit SPIR-V directly. No reason to tie yourself to Nvidia's moat.
- dannecodez 3mo ago[dead]
- swerner 3mo agoUnfortunately, Vulkan Compute doesn’t to all the things that OpenCL, SYCL, HIP or CUDA do.
- binsquare 3mo agoYep, there are inference stacks where it just does not work without cuda in any meaningful performance
- DiabloD3 3mo agoWeird, since the most used open source inference engine is faster on Vulkan on platforms that offer multiple options, with the sole exception being Nvidia, due to poor Nvidia driver quality (which I am forced to assume is intentional, Nvidia wishes to maintain their moat after all).
- inigyou 3mo agoThere's nothing stopping any of us from writing a better Nvidia driver btw. LLMs are very helpful with reverse engineering.
- HelloNurse 3mo agoBeing fast and being as easy to program as CUDA are two different things.
- mschuetz 3mo agoA couple of years ago I evaluated both Vulkan and Cuda as a choice for future projects. I couldnt get anything done after a week in Vulkan, but had the test prototype project working after just a day in Cuda. Needless to say, I'd never ever pick Vulkan for any project after that experience. It's just way to needlessly overengineered and bloated.
- lulzx 3mo agoI have been trying for cuda -> metal, to run it on mac, https://github.com/lulzx/cuda-metal https://github.com/lulzx/cuda-metal
- luciana1u 3mo ago[flagged]
- msond 3mo agoActually we launched in 2024 and the last message in our discord is definitely not that: https://discord.gg/KNpgGbTc38 https://discord.gg/KNpgGbTc38
- woctordho 3mo agoThere's nothing wrong to run CUDA on non-Nvidia hardware. CUDA has an interface that is reasonably well-designed, well-documented/reverse-engineered, and battle-tested for decades. What we need is not to invent another interface just under the name of 'open standard', but to implement the same interface. ROCm is exactly doing this, and so are other hardware SDKs such as MooreThread and Alibaba T-Head.
- rfv6723 3mo agoThat sounds nice on paper, but you’re assuming Nvidia wants to play fair. Nvidia is never going to share future microarchitecture secrets, so the moment they drop a new chip and update the compiler, everyone playing the compatibility game has to start from scratch.
- jingpostmedia 3mo ago[flagged]
- LogicFailsMe 3mo agoSomeone needs to stand up a benchmark suite for ROCM, this, and everyone else attempting it to really get the ball rolling here. SemiAnalysis could have a blast with this.
- zdw 3mo agoA YouTuber automated ROCm version testing for Strix Halo/R9700 + LLM workloads: https://kyuz0.github.io/amd-strix-halo-toolboxes/ https://kyuz0.github.io/amd-strix-halo-toolboxes/ https://kyuz0.github.io/amd-r9700-ai-toolboxes/ https://kyuz0.github.io/amd-r9700-ai-toolboxes/
- superkuh 3mo agoThe difference between ROCm and CUDA is that when a consumer GPU is released by nvidia it's supported for CUDA for about a decade (1xxx series cards just dropped last year). When a consumer GPU is released by AMD it's not supported by ROCm till about a year after release and then it's supported for about 3-4 years. With the RX 580 there were only 3.7 years after release before ROCm support was pulled. I bought mine a couple years after release and so only had about a year and a half of ROCm. Never again. Things might be different in enterprise but for consumer AMD GPU ROCm is a trap. It is a mayfly. Sure, you can try to run the cards unsupported but you're just multiplying the difficulty and maintainence burden. And nothing will just work.
- z0ltan 3mo ago[dead]
- puschkinfr 3mo agoIn this context AdaptiveCpp should also be mentioned. Started as a SYCL implementation, but recently-ish added a compiler for compiling a CUDA dialect to GPUs and CPUs from basically all vendors
- lumrn 3mo agoSYCL is probably the most up-to-date CUDA alternative for all intents and purposes, at least if one likes modern C++ style (and lambdas inside lambdas). Expose it as C and get bindings to any other language for relatively little effort as well since it’s just C++. With AdaptiveCpp you can also compile SYCL to CUDA so both ways work with the CUDA dialect (PCUDA). SYCL, as well as AdaptiveCpp, is a relatively active project though and has been for several years, feeding into the C++ standards committee work and is supported by several large organisations, including US national labs and several European universities. I suppose it’s worth keeping track of for people in related fields. I suppose it’s just really hard to beat the head start and ecosystem integration NVIDIA has with CUDA.
- dachworker 3mo agoWhy should I not just port my kernel to Triton? What's the appeal of Scale?
- noselasd 3mo agoYou can skip the porting part.
- asdaqopqkq 3mo agoaren't llms smart enough to directly write custom kernels for custom hardware from cuda code?
- u1hcw9nx 3mo agoAlternatives exist, but little demand outside hyperscalers and special uses. Neocloud customers just want plug-and-play CUDA. It works, it's tested, it adapts faster, and has known performance. Alternatives give no significant benefits. Things can change, but they are not changing now.
- tangsoupgallery 3mo ago[dead]
- cactusplant7374 3mo agoIsn't the future of the industry specialized chips like those that Broadcom and Cerebras are making? I don't know how much longer I can tolerate 50 tokens per second. It feels like the dial-up era.
- mailship 3mo ago[dead]
- woodrowbarlow 3mo agoi'm also interested in tenstorrent. they're building GPUs with cheap GDDR6 using a fast SRAM cache, and writing their own compiler stack (used instead of CUDA) that pipelines data to the SRAM ahead-of-time so you (in theory) never need to suffer the slow speed of GDDR6 for AI workloads. also they've got built-in SFP cages where the video ports would normally be.
- inigyou 3mo agoIs tenstorrent building GPUs now, not just tensor processors?
- woodrowbarlow 3mo agoyou're right, i misspoke by saying it's a GPU since it doesn't have video output. the just-in-time data pipelining wouldn't work for graphics workloads anyway.
- villgax 3mo ago@claude add this to the graveyard of wannabees
- HarHarVeryFunny 3mo agoThese efforts to support CUDA on non-Nvidia hardware seem to me misguided. If all you want is to be able to easily use non-NVidia hardware then high level tools like PyTorch already let you do that (and torch.compile uses Triton for target-specific optimization). OTOH if you want to be programming close to the metal to achieve top performance then you are probably not using CUDA in the first place, and using some CUDA translation layer on non-NVidia hardware would be an even worse idea.
- msond 3mo agoWe actually support NVIDIA hardware, too. In some benchmarks, SCALE beats nvcc, and we have compiler optimizations in the pipeline that will improve those numbers over time. > If all you want is to be able to easily use non-NVidia hardware then high level tools like PyTorch already let you do that Somewhat true, but, CUDA is significantly larger than PyTorch and there's more to Accelerated Computing than just those types of applications supported there. > OTOH if you want to be programming close to the metal to achieve top performance then you are probably not using CUDA in the first place, and using some CUDA translation layer on non-NVidia hardware would be an even worse idea. SOTA mlperf submissions use CUDA to achieve their high levels of performance. It's not a "translation layer", it's a native, ahead-of-time compiler that makes full use of the native hardware features. Here's an example of a feature (Shuffles) being compiled to take advantage of native hardware instructions, resulting in speedups: https://scale-lang.com/posts/2026-01-19-optimizing-cuda-shuffles https://scale-lang.com/posts/2026-01-19-optimizing-cuda-shuf...
- LogicFailsMe 3mo agoI've played with your compiler. It's not very robust yet to production code that leans into all the libraries in the ecosystem. Solve that and you've got a killer app IMO.
- msond 3mo agoWe'd love to know which libraries your production code falls over on! Feel free to ping us on discord (https://discord.com/invite/KNpgGbTc38 https://discord.com/invite/KNpgGbTc38) or you can find my e-mail in my profile if that's preferable
- alightsoul 3mo agoI fail to see how scale is not just another form of vendor lock in, given that their compiler is not open source. Every compiler used today except cuda's is open source. And Nvidia can get away with it because no one else cares about development experience
- mangoorangesoda 3mo ago[dead]
- leoh 3mo agoWhy is London having such a comeback right now in AI?
- fragmede 3mo agoProbably something to do with Deepmind being headquartered in London.