7 ms·

How Nvidia’s CUDA Monopoly in Machine Learning Is Breaking

166 points by pella 4y ago

pella 4y ago
https://archive.ph/upoNg https://archive.ph/upoNg
- klelatti 4y ago
  You do know that this is a just a capture of the free part of the post? (ie most of it) Readers should at least visit the original site if they want to read the free part.
  - dylan522p 4y ago
    Thanks! Ya, the archive link is useless, it only captures the free part, and I think I am very generous with what I keep in the free section on the ad-free website.
dna_polymerase 4y ago
It is quite weird to talk about all the frameworks which are built on top of CUDA eventually, but not talking about Rocm or OpenCL. OpenAI's Triton compiles down to CUDA atm (if I read their github right), and only supports Nvidia GPUs. PyTorch 2.0's installation page only mentions CPU and CUDA targets, therefor it's effectively all Nvidia GPUs. While all the frameworks and abstractions could offer other back-ends in theory the story of anything ML related on the other big name in the industry, AMD, is still poor. If anybody looses business because of bad decisions it is AMD, not Nvidia, who lead the whole industry. I am not convinced that anything will change in the near future.
- westurner 4y ago
  From https://news.ycombinator.com/item?id=32904285 https://news.ycombinator.com/item?id=32904285 re: AMD ROcm, HIPIFY, : > AMD ROcm supports Pytorch, TensorFlow, MlOpen, rocBLAS on NVIDIA and AMD GPUs: https://rocmdocs.amd.com/en/latest/Deep_learning/Deep-learning.html https://rocmdocs.amd.com/en/latest/Deep_learning/Deep-learni... [...] > ROCm-Developer-Tools/HIPIFY https://github.com/ROCm-Developer-Tools/HIPIFY https://github.com/ROCm-Developer-Tools/HIPIFY : >> hipify-clang is a clang-based tool for translating CUDA sources into HIP sources. It translates CUDA source into an abstract syntax tree, which is traversed by transformation matchers. After applying all the matchers, the output HIP source is produced. [...] From https://github.com/RadeonOpenCompute/clang-ocl https://github.com/RadeonOpenCompute/clang-ocl : > RadeonOpenCompute/OpenCL compilation with clang compiler
  - westurner 4y ago
    A better overview from the docs: "Machine Learning and High Performance Computing Software Stack for AMD GPU" https://rocmdocs.amd.com/en/latest/Installation_Guide/Software-Stack-for-AMD-GPU.html https://rocmdocs.amd.com/en/latest/Installation_Guide/Softwa...
- defaultname 4y ago
  AMD's position on machine learning has been fascinating to watch. While the company has gone through some cash strapped years, investing even some marginal effort into the ML space would have paid off handsomely in data center sales. Instead they let nvidia run away with it, pushing out a half-baked ROCm, which bizarrely supports such a tiny subset of cards it is farcical: With nvidia devs with a standard consumer GPU are building solutions and testing models, and when it's time to push big staying with nvidia is just the easy choice, while AMD apparently tries to upsell for even marginal support. Apple has done more to see Apple Silicon supported in the ML space in almost no time than AMD did over the past decade. Apple obviously has more resources, but they don't have a 1000 person team working on this, I suspect, but instead it's a very commando, efficient operation. They got a small number of dedicated resources committed to bringing support and enabling the community, and it happened.
danuker 4y ago
Excellent. Die, nVidia. I will never buy another nVidia card. They have made the firmware only modifiable to themselves. This forbids you from slowing down the card, for instance to run at a lower temperature (and prolong its life). "Little hope of reclocking becoming available for GM20x and newer GPUs as firmware now needs to be signed by NVIDIA to have the necessary access." https://nouveau.freedesktop.org/ https://nouveau.freedesktop.org/
- c2h5oh 4y ago
  > They have made the firmware only modifiable to themselves The same is true for current generation of AMD cards :/ You can no longer edit voltage tables
  - danuker 4y ago
    Ah, well, only second-hand for me then. Thanks for letting me know. Hopefully my laptop will outlast these tyrannical practices. If it doesn't, I'll just skip the dedicated GPU entirely.
  - JonChesterfield 4y ago
    I didn't think that was the case. Do you have a reference? I don't believe the firmware build tools are public. If you want to have a go with a hex editor, I'm not aware of any encryption or code signing going on. Interested to hear if I've missed it though.
    - c2h5oh 4y ago
      https://www.hardwaretimes.com/amds-radeon-rx-7900-series-gpus-come-with-locked-power-tables-limiting-overclocking-potential/ https://www.hardwaretimes.com/amds-radeon-rx-7900-series-gpu...
      - JonChesterfield 4y ago
        Thanks for the link. I think that's a different thing - there's something involving changing settings in the windows registry via some vendor tool where an option as been removed from the GUI. There's a reference to that as an alternative to "flashing modded BIOSes" but no claim that the BIOS can't be modified. Do GPU's have a BIOS? I thought that was a (now legacy) x86 thing. Side note, the cookie popup box on that website was ridiculous and lacking the 'reject all' option, but the content is readable by disabling javascript.
Alifatisk 4y ago
Could we say that nVidia is slowing down innovation for their own profit?
- WithinReason 4y ago
  On the SW side, yes.
deleted 4y ago
[deleted]
- WithinReason 4y ago
  > Unfortunately, whether you use CUDA or not, NVIDIA hardware is still the only realistic performant option. > Maybe the move away from CUDA dependence for these frameworks will make other manufacturers step up competition for accelerating ML workloads. But they should have been doing that anyway... That is entirely down to SW, not the HW. For example consider these two 24GB cards: Nvidia 4090: 82.58 TFLOPS/$1,599 = 51 GFLOPS/$ AMD 7900 XTX: 61.42 TFLOPS/$999 = 61 GFLOPs/$
deepnet 4y ago
The bottleneck is memory bus bandwidth as current Neural Net scaling is parameter count over compute so most A100s only operate at 60% of capacity. Operator fusion ( function composition ) has helped but devs time is impacted by the ever growing list of fused operations. This is a great overview of what Pytorch 2.0, TorchInduction, and OpenAI's Triton potentially bring to the table. Namely compilation to LLVM bypassing CUDA, fusing operators and compiling graphs at a lower level allowing other arcitectures, kept out by CUDA's NVIDIA exclusivity, to potentially offer competeting compilers - other compilation toolchains are very early days though. Backward comptibility is vital and the article details that these innovations are succesfully tested against existing Neural Net code. NVIDIA still rules the roost in science and industry compute. In this rapidly growing market competition is healthy so this is a good thing for everyone - especially those differentiators like ARM competing on Watts per FLOP.
blagie 4y ago
I wish this were so, but I'm very, very sceptical. Everything I use is built to CUDA. Nothing actually works on non-NVidia hardware. "Actually works" is different from "theoretically works." A lot of what I use theoretically supports ROCm, but in practice, when you try to use it, you have enough bugs big and small, and things crash or don't work right. That means everything I build is locked to NVidia as well, and everything people build on top of my stuff will be locked to NVidia too. Trying to maintain compatibility is like making Wine run Windows apps natively. The lock-in is now very significant. Virtually no one else has shipped anything really stable and usable, and we're now a very long time into people coding and testing only on NVidia. I hear that higher-level APIs are being built, nominally intended to be cross-platform, but I suspect (1) those will go the same way (2) we'll want backwards-compatibility. I have code which uses old ML models, CuPy, or otherwise. That stuff is locked to NVidia. I'm buying NVidia. This means I'm only making patches to bugs impacting NVidia. Network effects. So long as most people buy NVidia, NVidia will have more engineering resources too. It's very hard for a smaller player to catch up or pass. See Intel for many decades. To break from NVidia would require either a gross failure in execution by NVidia or something fundamentally new. To break the monopoly, I could see new products which might deliver: - Very low latency - Support for conventional DDR in large quantity (so I can run GPT-3 models at home) - Much more general functionality (e.g. an open architecture, and integration with conventional Python to parallelize normal code) ... or other significant new features. However, I don't see a lot of stuff like that being worked on, let alone coming out.
- fragmede 4y ago
  Critically, there's also cost. The formula is $W Nvidia for N-flops vs $X ROCm for N-flops + $Y engineering cost of recoding. If $W > $X + $Y, then people will go with AMD.
  - sokoloff 4y ago
    That's roughly right only if the people paying $W vs $X and $Y are the same people. The people paying $X and the people paying $Y are often not the same people across large swaths of the GPU-buying ecosystem. ($W and $X also change over time and with volume of units shipped and volume shipped varies with how useful and widely applicable the cards are.) I'd love to get an AMD [6650XT-ish] for my teen's gaming PC. But they also do some programming with me and I slightly worry about "oh, your PC can't do CUDA, therefore can't do XYZ that's interesting to you". I probably will still just go AMD and then have us use rented GPUs if we want to do any CUDA demos together.
YetAnotherNick 4y ago
Almost the entirety of the post is not related to the title. The summary is of this is given in last: > OpenAI Triton only officially supports Nvidia GPUs today, but that is changing in the near future. Multiple other hardware vendors will be supported in the future, and this open-source project is gaining incredible steam. It's not even the first tool for allowing ML workloads in non NVidia cards. ROCm already exists. TF already supports M1/M2 GPU already. The entire point is that no cards come close to Nvidia on raw GPGPU performance, and this is not going to change in near future.
- atemerev 4y ago
  Have you actually tried to write something with ROCm compared to CUDA? For me, ROCm is very low-level, boilerplate-ridden and nearly unusable.
  - JonChesterfield 4y ago
    HIP is very similar to cuda. There's a perl string substitution thing that will try to convert one to the other. There is lower level code below cuda and below hip. In cuda, it's closed source and hidden. In rocm, you can follow the control flow all the way into the amdkfd kernel module. I like having the option to program at a lower level than cuda. When I need to write in hip, I like having all the source from the stack trace.
    - pjmlp 4y ago
      Except the small detail that CUDA is polyglot and quite a few runtimes target PTX. Those Perl scripts won't do nothing for them.
      - JonChesterfield 4y ago
        In which sense is cuda polyglot? It's a dialect of C++. Maybe you mean cuda the toolchain (driver libs, ptxas etc) as opposed to cuda the language. Shame they have the same name. Anything targeting PTX via LLVM should be able to target GCN by changing flags, though that may not be totally seamless. Things writing ptx assembly directly would be more awkward to retarget, there's no direct equivalent there.
atemerev 4y ago
A great report, which concludes with “everything we described here only supports CUDA backend for now, but it will change at some indeterminate point in the future”. And there’s a reason for that. OpenCL and particularly ROCm are much lower level and much less immediately usable compared to CUDA. AMD now has HIP, but HIP is “something that is almost CUDA-compatible, with no real improvements”, instead of actually doing something new and better. CUDA is an old API, and certainly has its rough edges, but I wonder why nobody has managed to do something genuinely _better_ in all these years. ROCm is so low-level and pedantic it is nearly unusable. A simple implementation of FFT takes about 150 lines of C++/CUDA and about 2000 lines in ROCm — so no, no competition at all.
- pjmlp 4y ago
  One of the best CUDA decisions was to go polyglot, and in that context, first class support for Fortran. The competition is still like "Fortran what??!" every time I happen to watch a Khronos or AMD talk on compute.
- washadjeffmad 4y ago
  I remember people saying the same of the Mantle/Vulkan API when compared to DirectX. Now it powers gaming on Linux and the Steam Deck, and now even Intel uses DXVK in their new Arc drivers. ROCm is a framework, and its existence likely has a bigger impact on the quality of CUDA and the good behavior of Jensen than we realize.
  - pjmlp 4y ago
    While being mostly ignored everywhere else. Enjoy your 2% market share.
- Hizonner 4y ago
  I know essentially nothing about any of these tools, but I get the impression that the report is arguing that the API for ML is really up at the Python layer, and ML programmers don't interact directly with the stuff you're talking about. If the new PyTorch funnels everything through abstractions that don't require you to implement as many primitives to port it onto a new backend, then it matters less than before if ROCm (or whatever...) is harder to use, because the person doing the port don't have to do as much with it, and the actual application programmer never sees it at all.
iopq 4y ago
> It's easier to debug Python, largely since it's interpreted. Uhh, it's easier to debug because it's not C++. Rust is fairly easy to debug and profile, despite being compiled
- lostmsu 4y ago
  C++ is easier to debug than Rust. Rust debuggers are very limited with expression evaluation. Specifically, trait calls are still impossible AFAIK.
  - pjmlp 4y ago
    While I don't have an installation around me now, I am sure I could do that in CodeLLDB/VSCode.
photonbeam 4y ago
Im curious, what stops AMD writing an api/abi compatibility layer for cuda?
- bno1 4y ago
  https://rocmdocs.amd.com/en/latest/Programming_Guides/HIP-porting-guide.html https://rocmdocs.amd.com/en/latest/Programming_Guides/HIP-po...
- JonChesterfield 4y ago
  Lawsuit risk from nvidia seems likely
- fxtentacle 4y ago
  My impression: arrogance The whole ROCm stuff is built around the idea that you push out a half-working, buggy, and slow mess as Open Source and then the community will take over and maintain it into a usable product. But that hasn't happened so far. Instead of building their own FOSS libraries, people are happy to use NVIDIA's proprietary libraries.
- mepian 4y ago
  The fate of ZLUDA points to the existence of some kind of legal obstacle: https://thenewstack.io/zluda-a-cuda-for-intel-gpus-needs-a-new-maintainer/ https://thenewstack.io/zluda-a-cuda-for-intel-gpus-needs-a-n...
fxtentacle 4y ago
"skipping Nvidia’s closed-source CUDA libraries, such as cuBLAS, in favor of open-source libraries, such as cutlass" Eww, so it'll be 90% slower? cuBLAS is insanely optimized. It's the reason why CUDA dominates the field even though one could in theory use a ROCm build of TensorFlow or PyTorch. But performance is abysmal without cuBLAS. So the entire article appears to hinge on a huge assumption: What if Free Open Source was equivalent to NVIDIA's top-tier priorietary library? (It's not)
- cjbgkagh 4y ago
  Cutlass is from NVidia and AFAIK gets pretty close to cuBLAS performance. I think the use case for cutlass is when you only need a few kernels and don’t want to pull in a huge cuBLAS dependency and are ok paying a small perf penalty for that. Then there are times when you need custom kernels that not available in cuBLAS and for that cutlass is about as fast as it gets.
  - dylan522p 4y ago
    cutlass is faster actually, in most cases where cutlass supports same stuff.
    - cjbgkagh 4y ago
      I presumed they’d fold cutlass improvements into cuBASE so cuBASE would be at least as fast. Maybe some compile time optimizations?
snird 4y ago
Shameless plug, I wrote a similar post a few months ago focusing more on the APIs side of things: https://www.the-investing-desk.com/nvidia-will-lose-its-grip-on-the-ai-industry/ https://www.the-investing-desk.com/nvidia-will-lose-its-grip...
singhrac 4y ago
In my opinion the only thing that would break the NVIDIA monopoly is if some other vendor (maybe Intel) sells a graphics card that breaks NVIDIA's consumer/datacenter divide (e.g. offers A100-size memory for consumers to purchase, ideally less than $1k), and starts working on 100% PyTorch compatibility with their software extension. No custom kernel patches, no Perl substitution, I want to (a) install a driver, (b) pip install a model, and (c) run it. The vast vast majority of ML researchers do not want to spend time fiddling with compiler flags. Give them something they can't do with NVIDIA, make the process not awful, and there's money to be found. At the very least, "winning" at AI will increase your revenue multiple...
- 7speter 4y ago
  Nvidia already does sort of break that divide, don’t they? By offering cuda on consumer cards, they offer a gateway to curious users, and something to get by with at home for professional users. I think intel was late to the game with Arc because they insisted on making a card that could compete with 80-90% of nvidias features on their first mass market try, and had to build much of that infrastructure from scratch.

166 points by pella 4y ago

pella 4y ago

https://archive.ph/upoNg https://archive.ph/upoNg

klelatti 4y ago
You do know that this is a just a capture of the free part of the post? (ie most of it) Readers should at least visit the original site if they want to read the free part.
- dylan522p 4y ago
  Thanks! Ya, the archive link is useless, it only captures the free part, and I think I am very generous with what I keep in the free section on the ad-free website.

dna_polymerase 4y ago

It is quite weird to talk about all the frameworks which are built on top of CUDA eventually, but not talking about Rocm or OpenCL. OpenAI's Triton compiles down to CUDA atm (if I read their github right), and only supports Nvidia GPUs. PyTorch 2.0's installation page only mentions CPU and CUDA targets, therefor it's effectively all Nvidia GPUs. While all the frameworks and abstractions could offer other back-ends in theory the story of anything ML related on the other big name in the industry, AMD, is still poor. If anybody looses business because of bad decisions it is AMD, not Nvidia, who lead the whole industry. I am not convinced that anything will change in the near future.

westurner 4y ago
From https://news.ycombinator.com/item?id=32904285 https://news.ycombinator.com/item?id=32904285 re: AMD ROcm, HIPIFY, : > AMD ROcm supports Pytorch, TensorFlow, MlOpen, rocBLAS on NVIDIA and AMD GPUs: https://rocmdocs.amd.com/en/latest/Deep_learning/Deep-learning.html https://rocmdocs.amd.com/en/latest/Deep_learning/Deep-learni... [...] > ROCm-Developer-Tools/HIPIFY https://github.com/ROCm-Developer-Tools/HIPIFY https://github.com/ROCm-Developer-Tools/HIPIFY : >> hipify-clang is a clang-based tool for translating CUDA sources into HIP sources. It translates CUDA source into an abstract syntax tree, which is traversed by transformation matchers. After applying all the matchers, the output HIP source is produced. [...] From https://github.com/RadeonOpenCompute/clang-ocl https://github.com/RadeonOpenCompute/clang-ocl : > RadeonOpenCompute/OpenCL compilation with clang compiler
- westurner 4y ago
  A better overview from the docs: "Machine Learning and High Performance Computing Software Stack for AMD GPU" https://rocmdocs.amd.com/en/latest/Installation_Guide/Software-Stack-for-AMD-GPU.html https://rocmdocs.amd.com/en/latest/Installation_Guide/Softwa...
defaultname 4y ago
AMD's position on machine learning has been fascinating to watch. While the company has gone through some cash strapped years, investing even some marginal effort into the ML space would have paid off handsomely in data center sales. Instead they let nvidia run away with it, pushing out a half-baked ROCm, which bizarrely supports such a tiny subset of cards it is farcical: With nvidia devs with a standard consumer GPU are building solutions and testing models, and when it's time to push big staying with nvidia is just the easy choice, while AMD apparently tries to upsell for even marginal support. Apple has done more to see Apple Silicon supported in the ML space in almost no time than AMD did over the past decade. Apple obviously has more resources, but they don't have a 1000 person team working on this, I suspect, but instead it's a very commando, efficient operation. They got a small number of dedicated resources committed to bringing support and enabling the community, and it happened.

danuker 4y ago

Excellent. Die, nVidia. I will never buy another nVidia card. They have made the firmware only modifiable to themselves. This forbids you from slowing down the card, for instance to run at a lower temperature (and prolong its life). "Little hope of reclocking becoming available for GM20x and newer GPUs as firmware now needs to be signed by NVIDIA to have the necessary access." https://nouveau.freedesktop.org/ https://nouveau.freedesktop.org/

c2h5oh 4y ago
> They have made the firmware only modifiable to themselves The same is true for current generation of AMD cards :/ You can no longer edit voltage tables
- danuker 4y ago
  Ah, well, only second-hand for me then. Thanks for letting me know. Hopefully my laptop will outlast these tyrannical practices. If it doesn't, I'll just skip the dedicated GPU entirely.
- JonChesterfield 4y ago
  I didn't think that was the case. Do you have a reference? I don't believe the firmware build tools are public. If you want to have a go with a hex editor, I'm not aware of any encryption or code signing going on. Interested to hear if I've missed it though.
  - c2h5oh 4y ago
    https://www.hardwaretimes.com/amds-radeon-rx-7900-series-gpus-come-with-locked-power-tables-limiting-overclocking-potential/ https://www.hardwaretimes.com/amds-radeon-rx-7900-series-gpu...
    - JonChesterfield 4y ago
      Thanks for the link. I think that's a different thing - there's something involving changing settings in the windows registry via some vendor tool where an option as been removed from the GUI. There's a reference to that as an alternative to "flashing modded BIOSes" but no claim that the BIOS can't be modified. Do GPU's have a BIOS? I thought that was a (now legacy) x86 thing. Side note, the cookie popup box on that website was ridiculous and lacking the 'reject all' option, but the content is readable by disabling javascript.

Alifatisk 4y ago

Could we say that nVidia is slowing down innovation for their own profit?

WithinReason 4y ago
On the SW side, yes.

deleted 4y ago

[deleted]

WithinReason 4y ago
> Unfortunately, whether you use CUDA or not, NVIDIA hardware is still the only realistic performant option. > Maybe the move away from CUDA dependence for these frameworks will make other manufacturers step up competition for accelerating ML workloads. But they should have been doing that anyway... That is entirely down to SW, not the HW. For example consider these two 24GB cards: Nvidia 4090: 82.58 TFLOPS/$1,599 = 51 GFLOPS/$ AMD 7900 XTX: 61.42 TFLOPS/$999 = 61 GFLOPs/$

deepnet 4y ago

The bottleneck is memory bus bandwidth as current Neural Net scaling is parameter count over compute so most A100s only operate at 60% of capacity. Operator fusion ( function composition ) has helped but devs time is impacted by the ever growing list of fused operations. This is a great overview of what Pytorch 2.0, TorchInduction, and OpenAI's Triton potentially bring to the table. Namely compilation to LLVM bypassing CUDA, fusing operators and compiling graphs at a lower level allowing other arcitectures, kept out by CUDA's NVIDIA exclusivity, to potentially offer competeting compilers - other compilation toolchains are very early days though. Backward comptibility is vital and the article details that these innovations are succesfully tested against existing Neural Net code. NVIDIA still rules the roost in science and industry compute. In this rapidly growing market competition is healthy so this is a good thing for everyone - especially those differentiators like ARM competing on Watts per FLOP.

blagie 4y ago

I wish this were so, but I'm very, very sceptical. Everything I use is built to CUDA. Nothing actually works on non-NVidia hardware. "Actually works" is different from "theoretically works." A lot of what I use theoretically supports ROCm, but in practice, when you try to use it, you have enough bugs big and small, and things crash or don't work right. That means everything I build is locked to NVidia as well, and everything people build on top of my stuff will be locked to NVidia too. Trying to maintain compatibility is like making Wine run Windows apps natively. The lock-in is now very significant. Virtually no one else has shipped anything really stable and usable, and we're now a very long time into people coding and testing only on NVidia. I hear that higher-level APIs are being built, nominally intended to be cross-platform, but I suspect (1) those will go the same way (2) we'll want backwards-compatibility. I have code which uses old ML models, CuPy, or otherwise. That stuff is locked to NVidia. I'm buying NVidia. This means I'm only making patches to bugs impacting NVidia. Network effects. So long as most people buy NVidia, NVidia will have more engineering resources too. It's very hard for a smaller player to catch up or pass. See Intel for many decades. To break from NVidia would require either a gross failure in execution by NVidia or something fundamentally new. To break the monopoly, I could see new products which might deliver: - Very low latency - Support for conventional DDR in large quantity (so I can run GPT-3 models at home) - Much more general functionality (e.g. an open architecture, and integration with conventional Python to parallelize normal code) ... or other significant new features. However, I don't see a lot of stuff like that being worked on, let alone coming out.

fragmede 4y ago
Critically, there's also cost. The formula is $W Nvidia for N-flops vs $X ROCm for N-flops + $Y engineering cost of recoding. If $W > $X + $Y, then people will go with AMD.
- sokoloff 4y ago
  That's roughly right only if the people paying $W vs $X and $Y are the same people. The people paying $X and the people paying $Y are often not the same people across large swaths of the GPU-buying ecosystem. ($W and $X also change over time and with volume of units shipped and volume shipped varies with how useful and widely applicable the cards are.) I'd love to get an AMD [6650XT-ish] for my teen's gaming PC. But they also do some programming with me and I slightly worry about "oh, your PC can't do CUDA, therefore can't do XYZ that's interesting to you". I probably will still just go AMD and then have us use rented GPUs if we want to do any CUDA demos together.

YetAnotherNick 4y ago

Almost the entirety of the post is not related to the title. The summary is of this is given in last: > OpenAI Triton only officially supports Nvidia GPUs today, but that is changing in the near future. Multiple other hardware vendors will be supported in the future, and this open-source project is gaining incredible steam. It's not even the first tool for allowing ML workloads in non NVidia cards. ROCm already exists. TF already supports M1/M2 GPU already. The entire point is that no cards come close to Nvidia on raw GPGPU performance, and this is not going to change in near future.

atemerev 4y ago
Have you actually tried to write something with ROCm compared to CUDA? For me, ROCm is very low-level, boilerplate-ridden and nearly unusable.
- JonChesterfield 4y ago
  HIP is very similar to cuda. There's a perl string substitution thing that will try to convert one to the other. There is lower level code below cuda and below hip. In cuda, it's closed source and hidden. In rocm, you can follow the control flow all the way into the amdkfd kernel module. I like having the option to program at a lower level than cuda. When I need to write in hip, I like having all the source from the stack trace.
  - pjmlp 4y ago
    Except the small detail that CUDA is polyglot and quite a few runtimes target PTX. Those Perl scripts won't do nothing for them.
    - JonChesterfield 4y ago
      In which sense is cuda polyglot? It's a dialect of C++. Maybe you mean cuda the toolchain (driver libs, ptxas etc) as opposed to cuda the language. Shame they have the same name. Anything targeting PTX via LLVM should be able to target GCN by changing flags, though that may not be totally seamless. Things writing ptx assembly directly would be more awkward to retarget, there's no direct equivalent there.

atemerev 4y ago

A great report, which concludes with “everything we described here only supports CUDA backend for now, but it will change at some indeterminate point in the future”. And there’s a reason for that. OpenCL and particularly ROCm are much lower level and much less immediately usable compared to CUDA. AMD now has HIP, but HIP is “something that is almost CUDA-compatible, with no real improvements”, instead of actually doing something new and better. CUDA is an old API, and certainly has its rough edges, but I wonder why nobody has managed to do something genuinely _better_ in all these years. ROCm is so low-level and pedantic it is nearly unusable. A simple implementation of FFT takes about 150 lines of C++/CUDA and about 2000 lines in ROCm — so no, no competition at all.

pjmlp 4y ago
One of the best CUDA decisions was to go polyglot, and in that context, first class support for Fortran. The competition is still like "Fortran what??!" every time I happen to watch a Khronos or AMD talk on compute.
washadjeffmad 4y ago
I remember people saying the same of the Mantle/Vulkan API when compared to DirectX. Now it powers gaming on Linux and the Steam Deck, and now even Intel uses DXVK in their new Arc drivers. ROCm is a framework, and its existence likely has a bigger impact on the quality of CUDA and the good behavior of Jensen than we realize.
- pjmlp 4y ago
  While being mostly ignored everywhere else. Enjoy your 2% market share.
Hizonner 4y ago
I know essentially nothing about any of these tools, but I get the impression that the report is arguing that the API for ML is really up at the Python layer, and ML programmers don't interact directly with the stuff you're talking about. If the new PyTorch funnels everything through abstractions that don't require you to implement as many primitives to port it onto a new backend, then it matters less than before if ROCm (or whatever...) is harder to use, because the person doing the port don't have to do as much with it, and the actual application programmer never sees it at all.

iopq 4y ago

> It's easier to debug Python, largely since it's interpreted. Uhh, it's easier to debug because it's not C++. Rust is fairly easy to debug and profile, despite being compiled

lostmsu 4y ago
C++ is easier to debug than Rust. Rust debuggers are very limited with expression evaluation. Specifically, trait calls are still impossible AFAIK.
- pjmlp 4y ago
  While I don't have an installation around me now, I am sure I could do that in CodeLLDB/VSCode.

photonbeam 4y ago

Im curious, what stops AMD writing an api/abi compatibility layer for cuda?

bno1 4y ago
https://rocmdocs.amd.com/en/latest/Programming_Guides/HIP-porting-guide.html https://rocmdocs.amd.com/en/latest/Programming_Guides/HIP-po...
JonChesterfield 4y ago
Lawsuit risk from nvidia seems likely
fxtentacle 4y ago
My impression: arrogance The whole ROCm stuff is built around the idea that you push out a half-working, buggy, and slow mess as Open Source and then the community will take over and maintain it into a usable product. But that hasn't happened so far. Instead of building their own FOSS libraries, people are happy to use NVIDIA's proprietary libraries.
mepian 4y ago
The fate of ZLUDA points to the existence of some kind of legal obstacle: https://thenewstack.io/zluda-a-cuda-for-intel-gpus-needs-a-new-maintainer/ https://thenewstack.io/zluda-a-cuda-for-intel-gpus-needs-a-n...

fxtentacle 4y ago

"skipping Nvidia’s closed-source CUDA libraries, such as cuBLAS, in favor of open-source libraries, such as cutlass" Eww, so it'll be 90% slower? cuBLAS is insanely optimized. It's the reason why CUDA dominates the field even though one could in theory use a ROCm build of TensorFlow or PyTorch. But performance is abysmal without cuBLAS. So the entire article appears to hinge on a huge assumption: What if Free Open Source was equivalent to NVIDIA's top-tier priorietary library? (It's not)

cjbgkagh 4y ago
Cutlass is from NVidia and AFAIK gets pretty close to cuBLAS performance. I think the use case for cutlass is when you only need a few kernels and don’t want to pull in a huge cuBLAS dependency and are ok paying a small perf penalty for that. Then there are times when you need custom kernels that not available in cuBLAS and for that cutlass is about as fast as it gets.
- dylan522p 4y ago
  cutlass is faster actually, in most cases where cutlass supports same stuff.
  - cjbgkagh 4y ago
    I presumed they’d fold cutlass improvements into cuBASE so cuBASE would be at least as fast. Maybe some compile time optimizations?

snird 4y ago

Shameless plug, I wrote a similar post a few months ago focusing more on the APIs side of things: https://www.the-investing-desk.com/nvidia-will-lose-its-grip-on-the-ai-industry/ https://www.the-investing-desk.com/nvidia-will-lose-its-grip...

singhrac 4y ago

In my opinion the only thing that would break the NVIDIA monopoly is if some other vendor (maybe Intel) sells a graphics card that breaks NVIDIA's consumer/datacenter divide (e.g. offers A100-size memory for consumers to purchase, ideally less than $1k), and starts working on 100% PyTorch compatibility with their software extension. No custom kernel patches, no Perl substitution, I want to (a) install a driver, (b) pip install a model, and (c) run it. The vast vast majority of ML researchers do not want to spend time fiddling with compiler flags. Give them something they can't do with NVIDIA, make the process not awful, and there's money to be found. At the very least, "winning" at AI will increase your revenue multiple...

7speter 4y ago
Nvidia already does sort of break that divide, don’t they? By offering cuda on consumer cards, they offer a gateway to curious users, and something to get by with at home for professional users. I think intel was late to the game with Arc because they insisted on making a card that could compete with 80-90% of nvidias features on their first mass market try, and had to build much of that infrastructure from scratch.