4 ms·

That's really interesting. I have no experience writing anything that involves GPUs/TPUs, but over the years I've consistently read that CUDA is the "real moat

That's really interesting. I have no experience writing anything that involves GPUs/TPUs, but over the years I've consistently read that CUDA is the "real moat" of Nvidia, which I never totally believed, but the way you describe makes it seem like it's not actually a moat in the slightest. It just happens to be an ecosystem associated with hardware that is not only considered the gold standard but happens to be more open than potential competition. Could it be that Nvidia has been on top because none of the competition has actually tried kicking them where it hurts?

csomar 2mo ago
Software has always been the moat but for some reason it's always hamstrung by upper management. The latest of the frenzies being replacing sane (or whatever we have) of development practices with AI-slop. Management likes it because it removes software developers from the loop.
szundi 2mo ago
[dead]
calebkaiser 2mo ago
It's a totally reasonable question, and one that everyone asks when they're learning about CUDA. The frustrating answer to your last question is that lots of companies have shipped GPU dev environments that can theoretically be used instead of CUDA. AMD has ROCm, Apple has had a couple projects (OpenCL, Metal), Intel has some stuff, and there are newer efforts like TinyGrad + a generation of slightly higher level frameworks from AI companies, like Triton from OpenAI. The basic problem is that CUDA has become something of a Schelling point. If you want to train a model right now, the highest performance you can get is almost certainly on CUDA. From the basic general matrix multiply operation, to specific NN architectures, CUDA is going to have incredibly optimized implementations out of the box. And it's going to make multi-GPU training so much easier. And all the dependencies you build on (those layers you import from PyTorch or Transformers or whatever) are going to work optimally right away on CUDA. And that weird random repo that you found with a unique optimizer--it runs on CUDA too. And now the cool new implementation that you're about to release is also going to be built for CUDA. It's so tempting to think "Just write replacement software", but you also need to transition the entire ecosystem in large part to match CUDA's effectiveness, and you need to get comparable performance out of your chip/library combo as NVIDIA can get out of its cards with CUDA. There's a whole story here to how effective NVIDIA has been at navigating this. Very early on, they heavily prioritized PyTorch and TensorFlow, getting involved in the projects as much as they could and making sure they always ran best on CUDA. But the TLDR is that yes, you're right, another company could write a CUDA competitor. But actually replacing CUDA is a much larger task. I'm personally hopeful that with the rise of coding agents, we see more movement on this front with other projects moving into view. It will take some time for any ecosystem to start to emerge that can dislodge CUDA for researchers who don't want to dive that deep into the stack, but hopefully we start to see some momentum build.
compiler-guy 2mo ago
To paraphrase the apocryphal Winston Churchill quote about democracy: “CUDA is the worst development ecosystem in existence. Except for all the others.”
- lostlogin 2mo ago
  ‘Yes, I Am Drunk, But CUDA is Ugly. Tomorrow I Will Be Sober, And CUDA Will Still Be Ugly.’
elictronic 2mo ago
Retraining a large high paid user base is often a non-starter. To put this in perspective, Boeing’s eventual retraining costs for all the pilots for the 737Max was around 5 billion dollars. Looking at software more specifically the Linux foundation reported based on software dev salaries in 2008 it would be 1.4 billion to only write the Linux kernel. Up until about 2023 there wasn’t enough money involved to have any reason to make a real CUDA killer even if you could get it adopted.
- mandeepj 2mo ago
  Yet, Microsoft pulled it off with a new .NET Framework, and Apple with its new iOS SDK. There are many more examples besides those two.
  - compiler-guy 2mo ago
    Both of those were a vendor X electing to stop updating framework A in favor of framework B. That’s a high cost for vendor X’s users, but if their business depends on vendor X, they have no choice. Maybe they can switch to vendor Y at that point, but now you are switching both vendors and frameworks. Replacing CUDA with another framework has much lower motivation. That advantages of the new framework must cover the switching costs and the risk of such a switch. All while CUDA continues to evolve and allow access to additional features. Apple and Microsoft had something of a captive userbase. New vendor on the block trying to replace CUDA does not.
  - dragonwriter 2mo ago
    Neither of those are a competitor replacing another, they are a same-vendor replacement. This is easier, because the company whose established product you are trying to displace is cooperating (because its you!) not actively resisting.
  - eterm 2mo ago
    Microsoft pulled it off with dotnet, sort of, because they approached it like a completely new language, sold people on the benefits of it. And the people they were selling that to, ( It's free and open souce now! ), were a very different group to the market they left behind on .NET Framework, who are often still struggling to make the transition now. Had they actually killed off .NET Framework, it would have been a different story, much more like the VB6/VBA to VB.NET transition, which so few people bothered with that VB.NET died out, because if you had to retrain that much, you figured you might as well go to C# or a instead, or indeed a completely different language entirely. I briefly worked professionally on a VB.NET project, but outside that job I've never met anyone else who can say the same. I've met a few who went straight from VB6 to C# though.
kevstev 2mo ago
I think OP is overstating it a bit tbh. So Nvidia has the market for the hardware, which helps, but there is literally no alternative to CUDA. Nvidia keeps it a scalpel for skilled users, its not super easy to use, but unlocks orders of more magnitudes of power for the use cases it excels at vs CPUs. I don't have access to anything like it in the Apple ecosystem. AMD has had years to try and counter it, but just has not. Google is kinda trying to do an end run around it with TPUs but they are still niche high end stuff with limited availability. Its really just CUDA, and CUDA can be seen as somewhat akin to C for assembly used by Nvidia's gpus- In many ways a wrapper around the low level hardware that often has those details bleed through.
- YuechenLi 2mo ago
  I think there is a misconception here: CUDA is not even close to being C for assembly used by Nvidia's GPUs, PTX is, and it uses JIT to compile to Nvidia's GPU assembly, SASS. It honestly easier just to have LLMs write PTX directly than to go through CUDA C/C++ at all. Again, "CUDA" isn't a programming language, it stands for Compute Unified Device Architecture; "C/C++ for CUDA" are the high-level languages that compiles to PTX and then SASS as well CPU orchestration code via NVCC. And to be honest, pretty much everything you can do in CUDA C/C++, you can also do in HLSL/GLSL compiled to SPIR-V, as long as the Vulkan hardware extension is available.
  - kevstev 2mo ago
    My point with the c comparison is that it often feels like a very thin layer that still requires you to know a lot about what's going on under the hood. It was more to give a view in what it's like to work with the lib/api. Maybe I wasn't being precise enough with my language for this forum, and also my last hands on experience with it was roughly 6 years ago, maybe it's gotten better. But it was much less (and forgive the imprecision!) python/pytorch-like where you say hey take this big blob of data and just slice and dice it on your many cores, and more like ok, here is the data, let's cudamemcopy it in these size chunks over to the gpu itself, to be used by this block of threads and run these commands (kernel in cudaspeak) on it. Much more painstaking and micromanagey of the resources. Pytorch IMHO feels like a proper abstracted API that hides the details and lets you just unleash the fury at the cost of some efficiency, while the cuda api itself, similar to working with C, forces you to really think about the low level details. I have a heavy backend and systems development background, and while it wasn't really intimidating to me, it was like wow you really have to have a deep working knowledge of how these things work and it felt like a step back in time IMHO. I doubt that's going to satisfy you but I think it gives a clearer picture of what using cuda is like if you typically use higher level languages and haven't touched C since college.
cepp 2mo ago
One of the rarely-mentioned value adds Nvidia provides is nccl[1] which makes multi-node networking and topology essentially plug and play. The other players have since caught on [2][3] and are working hard to catch up but I'd say networking is a real moat. [1] https://developer.nvidia.com/nccl https://developer.nvidia.com/nccl [2] https://pytorch.org/blog/torchcomms/ https://pytorch.org/blog/torchcomms/ [3] https://rocm.docs.amd.com/projects/rccl/en/latest/ https://rocm.docs.amd.com/projects/rccl/en/latest/
dannyw 2mo ago
CUDA has many problems, but I would say less problems than ROCm, etc; even before considering the ecosystem and that more people (or open source projects) have already solved CUDA's problems for you. Ironically, there was an open source project that was making great progress on CUDA compatibility on AMD hardware. AMD hired the lead developer, and then he shut down the project.
- bri3d 2mo ago
  ZLUDA is still alive. AMD sponsored it and the project was briefly halted during a dispute with them, but it’s been making steady progress. It doesn’t really make sense for AMD themselves or most use cases, though; any compatibility shim just adds problems on top of problems, and for AMD, entrenching a competitors technology even more never really seemed like a great idea.

csomar 2mo ago

Software has always been the moat but for some reason it's always hamstrung by upper management. The latest of the frenzies being replacing sane (or whatever we have) of development practices with AI-slop. Management likes it because it removes software developers from the loop.

szundi 2mo ago

[dead]

calebkaiser 2mo ago

It's a totally reasonable question, and one that everyone asks when they're learning about CUDA. The frustrating answer to your last question is that lots of companies have shipped GPU dev environments that can theoretically be used instead of CUDA. AMD has ROCm, Apple has had a couple projects (OpenCL, Metal), Intel has some stuff, and there are newer efforts like TinyGrad + a generation of slightly higher level frameworks from AI companies, like Triton from OpenAI. The basic problem is that CUDA has become something of a Schelling point. If you want to train a model right now, the highest performance you can get is almost certainly on CUDA. From the basic general matrix multiply operation, to specific NN architectures, CUDA is going to have incredibly optimized implementations out of the box. And it's going to make multi-GPU training so much easier. And all the dependencies you build on (those layers you import from PyTorch or Transformers or whatever) are going to work optimally right away on CUDA. And that weird random repo that you found with a unique optimizer--it runs on CUDA too. And now the cool new implementation that you're about to release is also going to be built for CUDA. It's so tempting to think "Just write replacement software", but you also need to transition the entire ecosystem in large part to match CUDA's effectiveness, and you need to get comparable performance out of your chip/library combo as NVIDIA can get out of its cards with CUDA. There's a whole story here to how effective NVIDIA has been at navigating this. Very early on, they heavily prioritized PyTorch and TensorFlow, getting involved in the projects as much as they could and making sure they always ran best on CUDA. But the TLDR is that yes, you're right, another company could write a CUDA competitor. But actually replacing CUDA is a much larger task. I'm personally hopeful that with the rise of coding agents, we see more movement on this front with other projects moving into view. It will take some time for any ecosystem to start to emerge that can dislodge CUDA for researchers who don't want to dive that deep into the stack, but hopefully we start to see some momentum build.

compiler-guy 2mo ago

To paraphrase the apocryphal Winston Churchill quote about democracy: “CUDA is the worst development ecosystem in existence. Except for all the others.”

lostlogin 2mo ago
‘Yes, I Am Drunk, But CUDA is Ugly. Tomorrow I Will Be Sober, And CUDA Will Still Be Ugly.’

elictronic 2mo ago

Retraining a large high paid user base is often a non-starter. To put this in perspective, Boeing’s eventual retraining costs for all the pilots for the 737Max was around 5 billion dollars. Looking at software more specifically the Linux foundation reported based on software dev salaries in 2008 it would be 1.4 billion to only write the Linux kernel. Up until about 2023 there wasn’t enough money involved to have any reason to make a real CUDA killer even if you could get it adopted.

mandeepj 2mo ago
Yet, Microsoft pulled it off with a new .NET Framework, and Apple with its new iOS SDK. There are many more examples besides those two.
- compiler-guy 2mo ago
  Both of those were a vendor X electing to stop updating framework A in favor of framework B. That’s a high cost for vendor X’s users, but if their business depends on vendor X, they have no choice. Maybe they can switch to vendor Y at that point, but now you are switching both vendors and frameworks. Replacing CUDA with another framework has much lower motivation. That advantages of the new framework must cover the switching costs and the risk of such a switch. All while CUDA continues to evolve and allow access to additional features. Apple and Microsoft had something of a captive userbase. New vendor on the block trying to replace CUDA does not.
- dragonwriter 2mo ago
  Neither of those are a competitor replacing another, they are a same-vendor replacement. This is easier, because the company whose established product you are trying to displace is cooperating (because its you!) not actively resisting.
- eterm 2mo ago
  Microsoft pulled it off with dotnet, sort of, because they approached it like a completely new language, sold people on the benefits of it. And the people they were selling that to, ( It's free and open souce now! ), were a very different group to the market they left behind on .NET Framework, who are often still struggling to make the transition now. Had they actually killed off .NET Framework, it would have been a different story, much more like the VB6/VBA to VB.NET transition, which so few people bothered with that VB.NET died out, because if you had to retrain that much, you figured you might as well go to C# or a instead, or indeed a completely different language entirely. I briefly worked professionally on a VB.NET project, but outside that job I've never met anyone else who can say the same. I've met a few who went straight from VB6 to C# though.

kevstev 2mo ago

I think OP is overstating it a bit tbh. So Nvidia has the market for the hardware, which helps, but there is literally no alternative to CUDA. Nvidia keeps it a scalpel for skilled users, its not super easy to use, but unlocks orders of more magnitudes of power for the use cases it excels at vs CPUs. I don't have access to anything like it in the Apple ecosystem. AMD has had years to try and counter it, but just has not. Google is kinda trying to do an end run around it with TPUs but they are still niche high end stuff with limited availability. Its really just CUDA, and CUDA can be seen as somewhat akin to C for assembly used by Nvidia's gpus- In many ways a wrapper around the low level hardware that often has those details bleed through.

YuechenLi 2mo ago
I think there is a misconception here: CUDA is not even close to being C for assembly used by Nvidia's GPUs, PTX is, and it uses JIT to compile to Nvidia's GPU assembly, SASS. It honestly easier just to have LLMs write PTX directly than to go through CUDA C/C++ at all. Again, "CUDA" isn't a programming language, it stands for Compute Unified Device Architecture; "C/C++ for CUDA" are the high-level languages that compiles to PTX and then SASS as well CPU orchestration code via NVCC. And to be honest, pretty much everything you can do in CUDA C/C++, you can also do in HLSL/GLSL compiled to SPIR-V, as long as the Vulkan hardware extension is available.
- kevstev 2mo ago
  My point with the c comparison is that it often feels like a very thin layer that still requires you to know a lot about what's going on under the hood. It was more to give a view in what it's like to work with the lib/api. Maybe I wasn't being precise enough with my language for this forum, and also my last hands on experience with it was roughly 6 years ago, maybe it's gotten better. But it was much less (and forgive the imprecision!) python/pytorch-like where you say hey take this big blob of data and just slice and dice it on your many cores, and more like ok, here is the data, let's cudamemcopy it in these size chunks over to the gpu itself, to be used by this block of threads and run these commands (kernel in cudaspeak) on it. Much more painstaking and micromanagey of the resources. Pytorch IMHO feels like a proper abstracted API that hides the details and lets you just unleash the fury at the cost of some efficiency, while the cuda api itself, similar to working with C, forces you to really think about the low level details. I have a heavy backend and systems development background, and while it wasn't really intimidating to me, it was like wow you really have to have a deep working knowledge of how these things work and it felt like a step back in time IMHO. I doubt that's going to satisfy you but I think it gives a clearer picture of what using cuda is like if you typically use higher level languages and haven't touched C since college.

cepp 2mo ago

One of the rarely-mentioned value adds Nvidia provides is nccl[1] which makes multi-node networking and topology essentially plug and play. The other players have since caught on [2][3] and are working hard to catch up but I'd say networking is a real moat. [1] https://developer.nvidia.com/nccl https://developer.nvidia.com/nccl [2] https://pytorch.org/blog/torchcomms/ https://pytorch.org/blog/torchcomms/ [3] https://rocm.docs.amd.com/projects/rccl/en/latest/ https://rocm.docs.amd.com/projects/rccl/en/latest/

dannyw 2mo ago

CUDA has many problems, but I would say less problems than ROCm, etc; even before considering the ecosystem and that more people (or open source projects) have already solved CUDA's problems for you. Ironically, there was an open source project that was making great progress on CUDA compatibility on AMD hardware. AMD hired the lead developer, and then he shut down the project.

bri3d 2mo ago
ZLUDA is still alive. AMD sponsored it and the project was briefly halted during a dispute with them, but it’s been making steady progress. It doesn’t really make sense for AMD themselves or most use cases, though; any compatibility shim just adds problems on top of problems, and for AMD, entrenching a competitors technology even more never really seemed like a great idea.