6 ms·

CUDA vs. ROCm: A case study

161 points by shihab 3y ago

ekelsen 3y ago
AMD attempted responses go all the way back to 2007 when CUDA first debuted with "Close to Metal" (https://en.wikipedia.org/wiki/Close_to_Metal https://en.wikipedia.org/wiki/Close_to_Metal). They've had nearly 20 years to fix the situation and have failed to do so. Maybe some third party player like Lamini AI will do what they couldn't and get acquired for it.
- shihab 3y ago
  The thing about modern AI, it's that operations involved here (e.g. Dense matmuls) are lot simpler and GPU friendly than what you'd find in a typical HPC applications. This means you can get pretty close to peak hardware performance using high-level languages like Python or OpenAI's Triton. I think it's unlikely that the push to improve ROCm's standard libraries will come from an AI-focused startup
  - cavisne 3y ago
    Even with AMD working on hard on specific benchmarks for marketing purposes they could not get close to peak hardware performance on their brand new chips. https://www.semianalysis.com/p/amd-mi300-performance-faster-than https://www.semianalysis.com/p/amd-mi300-performance-faster-...
cherryteastain 3y ago
I wish AMD would just drop ROCm at this stage, and focus on SYCL. The rocRAND/hipRAND woes in this article are if anything showing ROCm in a better light than it really is; here it at least worked and performed within the same ballpark as CUDA. Often it simply does not work at all, or if it works it's behind by a lot more. At work I simply gave up on our 4x Radeon Pro W6800 workstation because launching Tensorflow with more than 1 GPU would cause a kernel panic every time, and AMD engineers never offered a fix other than "reinstall Ubuntu". ROCm feels like such a half assed product that (to me at least) feels like it's been made to tick a box and look cool in corporate presentations. It's not made with the proper mindset to compete against CUDA. Lisa Su claims they're doubling down on ROCm but to me it feels like they're falling behind relative to Nvidia, not catching up. Banding together with Intel to support SYCL would in my opinion 1. Ensure there's a lot more momentum behind a single, cross-platform, industry-standard competitor 2. Entice other industry heavyweights like MSFT, Qualcomm, ARM etc to also take the cross-platform solutions more seriously 3. Encourage heavy investment into the developer experience and tooling for the cross-platform solution
- arcanus 3y ago
  SYCL is a terrible option. The performance of any application on SYCL is currently quite poor. Why drop ROCm (used on the world's largest supercomputer?) for an even more unproven product that AMD has no control over the roadmap? > Banding together with Intel to support SYCL would in my opinion Except that Intel has more control over SYCL and has repeatedly hurt AMD products with anticompetitive behavior in the past. Why would AMD permit their software to be controlled be a competitor? AMD is executing with ROCm NOW, with multiple Top500 supercomputer wins and deployment in major cloud vendors with MI300X. Yes, the software needs to improve, but the practice of throwing out software to start over is not a good strategy. AMD has far more momentum in the data center than INTC at present.
  - shihab 3y ago
    I agree about ROCm. When it comes to comparison to SYCL, HIP is much closer to the spirit of SYCL than ROCm is. Both aim to help writing a single codebase that'll run across multiple hardwares. For now though, the trajectory of SYCL appears much more promising to me than HIP. HIP is already split in two parts for CPU and GPU, which is baffling, and neither part seems to receive much love from AMD.
  - pjmlp 3y ago
    It hardly matters if most desktops and laptops will be running CUDA/SYCL instead.
  - hjabird 3y ago
    > The performance of any application on SYCL is currently quite poor. SYCL can get pretty much equivalent performance in Kernels to eg. CUDA. Try looking at SYCL performance papers on Arxiv. Eg. see [1]. That isn't to say that SYCL code is optimised on every platform without tweaking - you do still need to put effort into target specific optimizations to get the best performance, like you would in the CUDA or HIP. > Why drop ROCm (used on the world's largest supercomputer?) Some of the world's largest super-computers / HPC applications do use SYCL for AMD! The application I'm most aware of for this is GROMACS. As to why? - because having 3 version of the same code using different programming APIs is a big maintenance burden. [1] https://arxiv.org/pdf/2309.09609.pdf https://arxiv.org/pdf/2309.09609.pdf
dpflan 3y ago
Are there use-cases for LLMs assisting in developing this software? Basically, I'm wondering if LLMs for developing a GPU-API exist and how can they can accelerate development such that this "moat" becomes more of a river that other can join?
- seanhunter 3y ago
  I'm struggling to see how that helps given the current problem they have. What they have is several extremely half-assed implementations of somewhat different APIs that are basically the same and abysmal documentation. Pointing an LLM at it would make yet another half-assed implemetation of yet another different API, where what documentation it had would be convincingly wrong a bunch of the time. What they need is: 1) Make one API [1] that builds out of the box, works really well and gets the best out of their hardware right away. Bonus points if its not a total pig to use. 2) Take the thing that they built for 1 and make a straightforward python binding for it that works for tensorflow and pytorch.[2] 3) Have a beer on me. Seriously I would buy them one. Ideally there's an optional step 0 which is to stop making all kinds of press statements saying they are taking this seriously and paying sloppy journalists to say they are catching up in the race against nvidia if they can't actually get their act together to do steps 1 and 2. [1] In c, c++ or rust so it's easy to embed in python and other languages etc [2] By work here I mean the definition of done is a normal person can do `pip install tensorflow` and it will actually use an amd gpu with hardware accelleration right away. Not some bullshit where you have to jump through a bunch of hoops and apply a whole lot of custom patches and it kinda sorta sometimes works but actually mostly just either falls back to CPU or crashes half the time
  - dpflan 3y ago
    It does seem like they need a reset, and given that need for a reset, a new-starting point: how do they move quickly? I was thinking that LLM could assist in creating a coherent approach, helping in creating code, documentation, etc (i.e. dog-food'ing their 'product') -- what I would think could make this development more rapid.
anvuong 3y ago
“Please note the library is being actively developed, and is known to be incomplet; it might also be incorrekt and there could be a few bad bugs lurking.” That gives me a good laugh.
- thrtythreeforty 3y ago
  I use exactly the same verbiage in my draft documents. I think it's a reference to C++ standards draft documents, which have that on the cover. I don't know if that's the original usage though, it's just where I saw it first.
frognumber 3y ago
That's more-or-less my experience with AMD, only worse. Critical thing too are burned developers like myself. I'm looking forward to Intel v. NVidia. Arc A770 is a pretty serious competitor. It's the lowest-cost way to run OPT-175B. Given a 7-slot motherboard, $270 * 7 = $1890 for 112GB of VRAM in one computer. That's sweet. Compute speed would be on-par with top-of-the-line NVidia workstation GPU. Three of those are enough to run the largest open-source LLMs at around $9000. We're just drivers + libraries + documentation away, and Intel is not bad at drivers + libraries + documentation.
- vGPU 3y ago
  Sure but what’s the energy and overhead tradeoff?
- ParetoOptimal 3y ago
  > Arc A770 is a pretty serious competitor. It's the lowest-cost way to run OPT-175B. With what software?
- sangnoir 3y ago
  If AMD can significantly undercut Nvidia on perf/$ like they did to Intel with Zen 1; developers will fall over themselves to improve the software. When the performance price-point is low enough, the pain will be worth it.
- MezzoDelCammin 3y ago
  $9k for three of those? I'm affraid Your math aint mathing. 7-slot motherboards are IMO Threadripper teritory. The CPU alone will set You back at least $1k5, MB to go along with that is another $1k. Not including the stack of DDR5 that You'll likely throw into that and the PSU / case / coolers to feed it power and evacuate the heat from this monstrosity. That's at least $3000 for the base system "before" You add any GPUs. And at some point I'd like to read someone more enlightened to comment on what the memory bandwidth is likely to be and how exactly will You distribute the PCIe lanes...
  - frognumber 3y ago
quanto 3y ago
AMD cards + ROCm are used in top supercomputers for (non-deep learning) HPC. Why is this the case? I understand that AMD GPUs offer better cost efficiency for F32 & F64 FLOPs, RAM, and wattage. But however, if ROCm is such a half baked piece, shouldn't that advantage be gone? What drives AMD adoption in the HPC space then?
- throwawaymaths 3y ago
  That's not how sales works for those kinds of installations
- taw28 3y ago
  There is certainly a requirement that prevents payout on the contract for those supers if performance numbers aren’t met. Further, those performance numbers are from real apps, so the system WILL be useful. Not getting paid is not an option, so performance/usability will come.
  - quanto 3y ago
    how does that contractual obligation translate to technical implementation? Do those supercomputers get an optimized version of ROCm to fulfill said obligation?
    - p_l 3y ago
      Vendor iterates until client gets an usable environment, even if that means 50 forks of different libraries with custom patches that in the end work only on that one system.
    - izacus 3y ago
      Absolutely. You usually end up with software stack where NOTHING can be updated, most of the stuff is forked with custom patches and the learnings there aren't reusable elsewhere because the code is full of "// replace this with hardcoded constant 59843 because it prevents crash on HPC machine". It's a good marketing metric, but probably contraproductive the AMDs longterm success in the field. They're spending engineering time building something they'll unlikely to be able to translate into other fields.
outside1234 3y ago
How is there not an OpenGL at this point in this space? It seems like it is in all of the hyperscalers benefit to get this going - why haven’t they?
- CuriousCosmic 3y ago
  I mean there's OpenCL which has widespread support and is pretty much about as pleasant to write as OpenGL. It has existed for a very, very long time but people hate writing in it so they just defaulted to CUDA and ignored everyone but NVIDIA.
  - thrtythreeforty 3y ago
    Doesn't it take, like, 100 lines of code to call a kernel in OpenCL? Compared to Nvidia's 1? Developer friction is huge. Don't discount the amount of boilerplate it takes to get started with something.
    - throwaway0665 3y ago
      Just like OpenGL
    - dotnet00 3y ago
      Bigger than it taking a lot of boilerplate is the issue that there was never sufficiently widespread and bug free support for mixing CPU and GPU code in OpenCL in the way CUDA allows. So you end up having to do a lot more work to port stuff over. When porting stuff to CUDA, you can share a lot of code. Simplifies the task so much.
    - CuriousCosmic 3y ago
      Yep. That was the point I was making. Much like OpenGL, writing OpenCL is absolutely miserable so people just stuck with less portable alternatives that didn't write like pulling teeth.
    - pjmlp 3y ago
      Additionally OpenCL was stuck in C99, Khronos ignored C++ for a long time, ignored Fortran to this day, and never cared about other languages. CUDA is C, C++ and Fortran for at least a decade, and with PTX created a third party ecosystem that SPIR still doesn't have to this day.
latchkey 3y ago
Recent video of Lisa Su, good watch: https://twitter.com/TheSixFiveMedia/status/1737177221490450594 https://twitter.com/TheSixFiveMedia/status/17371772214904505...
Zetobal 3y ago
I hate ROCm so much I can't even describe how much I suffered because of this pos software. It wasn't good 3 years ago but manageable now it's just I don't even know how they got it worse. I really just wished my employer would give up on AMD for GPUs.
- mnd999 3y ago
  It doesn’t even work on half of the GPUs (Navi23 hello). It’s like they’re not even trying which is insane when there’s so much cash on the table.
  - fer 3y ago
    I had a look 3 years ago and the compatibility matrix was an absolute shitshow: older cards not supported at all, newer cards not supported at all, and support for those in the middle was full of holes depending on extension, version, etc. I couldn't even figure out what I needed to buy, so I didn't.
physicsguy 3y ago
This was exactly my experience with it too. It's moved on a bit since then but when I looked at rocFFT a couple of years ago, the documentation was really poor and it was missing features. When I switched from FFTW to cuFFT many years ago (~2015), the transition was very smooth, the documentation was great, and all features were supported. They even put a shim "FFTW" compatible header file in so that you didn't need to rewrite your code to make it work (leaving some performance on the table).
JonChesterfield 3y ago
Documentation. The python example is a bit on the nose. I've not had good times with ROCm's documentation either. Can anyone point to an example of good documentation for a big software system where they can also sketch how that was achieved? E.g. Cuda's docs are pretty good but I've no idea how they came to be, or how they stay up to date. LLVM's docs are a small amount of handwritten webpages which correlate with reality to some extent, the source for which lives in the same repo as the code. I have an idea that it needs to combine programmers writing some things, some testing infra to notice things like internal links will 404 and some non-developers writing things. I started trying to document one of my own systems as freeform notes under obsidian and while it kind of works at the time it diverges from reality pretty quickly, and that's without trying to have anyone else working on either the docs or the system. So what's the proper, established answer to this?
- not_the_fda 3y ago
  You have well thought out stable user facing APIs. You can change the underlying implementation but the user facing API remains stable. Then you hire good technical writers to write and maintain the documentation.
slavik81 3y ago
The rocrand library did not have any real documentation at all until 2023. It's still pretty barebones, but the updates in article regarding the Python API seem to suggest that this is a work-in-progress.
zamalek 3y ago
I'm convinced that Julia is how the moat will be crossed. There are some pretty incredible GPU packages for it (I'm looking at you KernelAbstractions.jl). The Python science community seems more than happy to carry on focusing on NVIDIA and are a lost cause. I somewhat don't blame them: the MI300X might be miles ahead and all, but AMD are not only oblivious to the desktop market (you know, where new ideas are prototyped) but are also seemingly actively hostile[1]. NVIDIA has people doing somewhat interesting things with a 3060 (which can eventually graduate to a 4090 or even a H100), while AMD don't what to hear about it unless you have a "pro" GPU. Definitely a case of dollar-wise and penny-foolish. [1] https://rocm.docs.amd.com/en/docs-5.5.1/release/gpu_os_support.html https://rocm.docs.amd.com/en/docs-5.5.1/release/gpu_os_suppo... * FWIW you can override this with an envar, but AMD aren't exactly forthcoming with that information.
HarHarVeryFunny 3y ago
At this point, rather than chasing CUDA/cuDNN compatability, it would seem more productive for AMD to be targetting high level language support. Forget CUDA compatibility, and instead support work such as Mojo and MLIR. It seems that in an ideal world PyTorch support for AMD wouldn't rely on ROCm, but rather be based on high level code compiled to MLIR with AMD target support, with this same MLIR representation supporting Mojo and any other languages such as Julia that want optimized AMD support.
up2isomorphism 3y ago
Any article that trying to make a big deal about 30% performance difference without comparing price and cost is just making a case for itself.

161 points by shihab 3y ago

AMD attempted responses go all the way back to 2007 when CUDA first debuted with "Close to Metal" (https://en.wikipedia.org/wiki/Close_to_Metal https://en.wikipedia.org/wiki/Close_to_Metal). They've had nearly 20 years to fix the situation and have failed to do so. Maybe some third party player like Lamini AI will do what they couldn't and get acquired for it.

shihab 3y ago
The thing about modern AI, it's that operations involved here (e.g. Dense matmuls) are lot simpler and GPU friendly than what you'd find in a typical HPC applications. This means you can get pretty close to peak hardware performance using high-level languages like Python or OpenAI's Triton. I think it's unlikely that the push to improve ROCm's standard libraries will come from an AI-focused startup
- cavisne 3y ago
  Even with AMD working on hard on specific benchmarks for marketing purposes they could not get close to peak hardware performance on their brand new chips. https://www.semianalysis.com/p/amd-mi300-performance-faster-than https://www.semianalysis.com/p/amd-mi300-performance-faster-...

cherryteastain 3y ago

I wish AMD would just drop ROCm at this stage, and focus on SYCL. The rocRAND/hipRAND woes in this article are if anything showing ROCm in a better light than it really is; here it at least worked and performed within the same ballpark as CUDA. Often it simply does not work at all, or if it works it's behind by a lot more. At work I simply gave up on our 4x Radeon Pro W6800 workstation because launching Tensorflow with more than 1 GPU would cause a kernel panic every time, and AMD engineers never offered a fix other than "reinstall Ubuntu". ROCm feels like such a half assed product that (to me at least) feels like it's been made to tick a box and look cool in corporate presentations. It's not made with the proper mindset to compete against CUDA. Lisa Su claims they're doubling down on ROCm but to me it feels like they're falling behind relative to Nvidia, not catching up. Banding together with Intel to support SYCL would in my opinion 1. Ensure there's a lot more momentum behind a single, cross-platform, industry-standard competitor 2. Entice other industry heavyweights like MSFT, Qualcomm, ARM etc to also take the cross-platform solutions more seriously 3. Encourage heavy investment into the developer experience and tooling for the cross-platform solution

arcanus 3y ago
SYCL is a terrible option. The performance of any application on SYCL is currently quite poor. Why drop ROCm (used on the world's largest supercomputer?) for an even more unproven product that AMD has no control over the roadmap? > Banding together with Intel to support SYCL would in my opinion Except that Intel has more control over SYCL and has repeatedly hurt AMD products with anticompetitive behavior in the past. Why would AMD permit their software to be controlled be a competitor? AMD is executing with ROCm NOW, with multiple Top500 supercomputer wins and deployment in major cloud vendors with MI300X. Yes, the software needs to improve, but the practice of throwing out software to start over is not a good strategy. AMD has far more momentum in the data center than INTC at present.
- shihab 3y ago
  I agree about ROCm. When it comes to comparison to SYCL, HIP is much closer to the spirit of SYCL than ROCm is. Both aim to help writing a single codebase that'll run across multiple hardwares. For now though, the trajectory of SYCL appears much more promising to me than HIP. HIP is already split in two parts for CPU and GPU, which is baffling, and neither part seems to receive much love from AMD.
- pjmlp 3y ago
  It hardly matters if most desktops and laptops will be running CUDA/SYCL instead.
- hjabird 3y ago
  > The performance of any application on SYCL is currently quite poor. SYCL can get pretty much equivalent performance in Kernels to eg. CUDA. Try looking at SYCL performance papers on Arxiv. Eg. see [1]. That isn't to say that SYCL code is optimised on every platform without tweaking - you do still need to put effort into target specific optimizations to get the best performance, like you would in the CUDA or HIP. > Why drop ROCm (used on the world's largest supercomputer?) Some of the world's largest super-computers / HPC applications do use SYCL for AMD! The application I'm most aware of for this is GROMACS. As to why? - because having 3 version of the same code using different programming APIs is a big maintenance burden. [1] https://arxiv.org/pdf/2309.09609.pdf https://arxiv.org/pdf/2309.09609.pdf

dpflan 3y ago

Are there use-cases for LLMs assisting in developing this software? Basically, I'm wondering if LLMs for developing a GPU-API exist and how can they can accelerate development such that this "moat" becomes more of a river that other can join?

seanhunter 3y ago
I'm struggling to see how that helps given the current problem they have. What they have is several extremely half-assed implementations of somewhat different APIs that are basically the same and abysmal documentation. Pointing an LLM at it would make yet another half-assed implemetation of yet another different API, where what documentation it had would be convincingly wrong a bunch of the time. What they need is: 1) Make one API [1] that builds out of the box, works really well and gets the best out of their hardware right away. Bonus points if its not a total pig to use. 2) Take the thing that they built for 1 and make a straightforward python binding for it that works for tensorflow and pytorch.[2] 3) Have a beer on me. Seriously I would buy them one. Ideally there's an optional step 0 which is to stop making all kinds of press statements saying they are taking this seriously and paying sloppy journalists to say they are catching up in the race against nvidia if they can't actually get their act together to do steps 1 and 2. [1] In c, c++ or rust so it's easy to embed in python and other languages etc [2] By work here I mean the definition of done is a normal person can do `pip install tensorflow` and it will actually use an amd gpu with hardware accelleration right away. Not some bullshit where you have to jump through a bunch of hoops and apply a whole lot of custom patches and it kinda sorta sometimes works but actually mostly just either falls back to CPU or crashes half the time
- dpflan 3y ago
  It does seem like they need a reset, and given that need for a reset, a new-starting point: how do they move quickly? I was thinking that LLM could assist in creating a coherent approach, helping in creating code, documentation, etc (i.e. dog-food'ing their 'product') -- what I would think could make this development more rapid.

anvuong 3y ago

“Please note the library is being actively developed, and is known to be incomplet; it might also be incorrekt and there could be a few bad bugs lurking.” That gives me a good laugh.

thrtythreeforty 3y ago
I use exactly the same verbiage in my draft documents. I think it's a reference to C++ standards draft documents, which have that on the cover. I don't know if that's the original usage though, it's just where I saw it first.

frognumber 3y ago

That's more-or-less my experience with AMD, only worse. Critical thing too are burned developers like myself. I'm looking forward to Intel v. NVidia. Arc A770 is a pretty serious competitor. It's the lowest-cost way to run OPT-175B. Given a 7-slot motherboard, $270 * 7 = $1890 for 112GB of VRAM in one computer. That's sweet. Compute speed would be on-par with top-of-the-line NVidia workstation GPU. Three of those are enough to run the largest open-source LLMs at around $9000. We're just drivers + libraries + documentation away, and Intel is not bad at drivers + libraries + documentation.

vGPU 3y ago
Sure but what’s the energy and overhead tradeoff?
ParetoOptimal 3y ago
> Arc A770 is a pretty serious competitor. It's the lowest-cost way to run OPT-175B. With what software?
sangnoir 3y ago
If AMD can significantly undercut Nvidia on perf/$ like they did to Intel with Zen 1; developers will fall over themselves to improve the software. When the performance price-point is low enough, the pain will be worth it.
MezzoDelCammin 3y ago
$9k for three of those? I'm affraid Your math aint mathing. 7-slot motherboards are IMO Threadripper teritory. The CPU alone will set You back at least $1k5, MB to go along with that is another $1k. Not including the stack of DDR5 that You'll likely throw into that and the PSU / case / coolers to feed it power and evacuate the heat from this monstrosity. That's at least $3000 for the base system "before" You add any GPUs. And at some point I'd like to read someone more enlightened to comment on what the memory bandwidth is likely to be and how exactly will You distribute the PCIe lanes...
- frognumber 3y ago

quanto 3y ago

AMD cards + ROCm are used in top supercomputers for (non-deep learning) HPC. Why is this the case? I understand that AMD GPUs offer better cost efficiency for F32 & F64 FLOPs, RAM, and wattage. But however, if ROCm is such a half baked piece, shouldn't that advantage be gone? What drives AMD adoption in the HPC space then?

throwawaymaths 3y ago
That's not how sales works for those kinds of installations
taw28 3y ago
There is certainly a requirement that prevents payout on the contract for those supers if performance numbers aren’t met. Further, those performance numbers are from real apps, so the system WILL be useful. Not getting paid is not an option, so performance/usability will come.
- quanto 3y ago
  how does that contractual obligation translate to technical implementation? Do those supercomputers get an optimized version of ROCm to fulfill said obligation?
  - p_l 3y ago
    Vendor iterates until client gets an usable environment, even if that means 50 forks of different libraries with custom patches that in the end work only on that one system.
  - izacus 3y ago
    Absolutely. You usually end up with software stack where NOTHING can be updated, most of the stuff is forked with custom patches and the learnings there aren't reusable elsewhere because the code is full of "// replace this with hardcoded constant 59843 because it prevents crash on HPC machine". It's a good marketing metric, but probably contraproductive the AMDs longterm success in the field. They're spending engineering time building something they'll unlikely to be able to translate into other fields.

outside1234 3y ago

How is there not an OpenGL at this point in this space? It seems like it is in all of the hyperscalers benefit to get this going - why haven’t they?

CuriousCosmic 3y ago
I mean there's OpenCL which has widespread support and is pretty much about as pleasant to write as OpenGL. It has existed for a very, very long time but people hate writing in it so they just defaulted to CUDA and ignored everyone but NVIDIA.
- thrtythreeforty 3y ago
  Doesn't it take, like, 100 lines of code to call a kernel in OpenCL? Compared to Nvidia's 1? Developer friction is huge. Don't discount the amount of boilerplate it takes to get started with something.
  - throwaway0665 3y ago
    Just like OpenGL
  - dotnet00 3y ago
    Bigger than it taking a lot of boilerplate is the issue that there was never sufficiently widespread and bug free support for mixing CPU and GPU code in OpenCL in the way CUDA allows. So you end up having to do a lot more work to port stuff over. When porting stuff to CUDA, you can share a lot of code. Simplifies the task so much.
  - CuriousCosmic 3y ago
    Yep. That was the point I was making. Much like OpenGL, writing OpenCL is absolutely miserable so people just stuck with less portable alternatives that didn't write like pulling teeth.
  - pjmlp 3y ago
    Additionally OpenCL was stuck in C99, Khronos ignored C++ for a long time, ignored Fortran to this day, and never cared about other languages. CUDA is C, C++ and Fortran for at least a decade, and with PTX created a third party ecosystem that SPIR still doesn't have to this day.

latchkey 3y ago

Recent video of Lisa Su, good watch: https://twitter.com/TheSixFiveMedia/status/1737177221490450594 https://twitter.com/TheSixFiveMedia/status/17371772214904505...

Zetobal 3y ago

I hate ROCm so much I can't even describe how much I suffered because of this pos software. It wasn't good 3 years ago but manageable now it's just I don't even know how they got it worse. I really just wished my employer would give up on AMD for GPUs.

mnd999 3y ago
It doesn’t even work on half of the GPUs (Navi23 hello). It’s like they’re not even trying which is insane when there’s so much cash on the table.
- fer 3y ago
  I had a look 3 years ago and the compatibility matrix was an absolute shitshow: older cards not supported at all, newer cards not supported at all, and support for those in the middle was full of holes depending on extension, version, etc. I couldn't even figure out what I needed to buy, so I didn't.

physicsguy 3y ago

This was exactly my experience with it too. It's moved on a bit since then but when I looked at rocFFT a couple of years ago, the documentation was really poor and it was missing features. When I switched from FFTW to cuFFT many years ago (~2015), the transition was very smooth, the documentation was great, and all features were supported. They even put a shim "FFTW" compatible header file in so that you didn't need to rewrite your code to make it work (leaving some performance on the table).

JonChesterfield 3y ago

Documentation. The python example is a bit on the nose. I've not had good times with ROCm's documentation either. Can anyone point to an example of good documentation for a big software system where they can also sketch how that was achieved? E.g. Cuda's docs are pretty good but I've no idea how they came to be, or how they stay up to date. LLVM's docs are a small amount of handwritten webpages which correlate with reality to some extent, the source for which lives in the same repo as the code. I have an idea that it needs to combine programmers writing some things, some testing infra to notice things like internal links will 404 and some non-developers writing things. I started trying to document one of my own systems as freeform notes under obsidian and while it kind of works at the time it diverges from reality pretty quickly, and that's without trying to have anyone else working on either the docs or the system. So what's the proper, established answer to this?

not_the_fda 3y ago
You have well thought out stable user facing APIs. You can change the underlying implementation but the user facing API remains stable. Then you hire good technical writers to write and maintain the documentation.

slavik81 3y ago

The rocrand library did not have any real documentation at all until 2023. It's still pretty barebones, but the updates in article regarding the Python API seem to suggest that this is a work-in-progress.

zamalek 3y ago

I'm convinced that Julia is how the moat will be crossed. There are some pretty incredible GPU packages for it (I'm looking at you KernelAbstractions.jl). The Python science community seems more than happy to carry on focusing on NVIDIA and are a lost cause. I somewhat don't blame them: the MI300X might be miles ahead and all, but AMD are not only oblivious to the desktop market (you know, where new ideas are prototyped) but are also seemingly actively hostile[1]. NVIDIA has people doing somewhat interesting things with a 3060 (which can eventually graduate to a 4090 or even a H100), while AMD don't what to hear about it unless you have a "pro" GPU. Definitely a case of dollar-wise and penny-foolish. [1] https://rocm.docs.amd.com/en/docs-5.5.1/release/gpu_os_support.html https://rocm.docs.amd.com/en/docs-5.5.1/release/gpu_os_suppo... * FWIW you can override this with an envar, but AMD aren't exactly forthcoming with that information.

HarHarVeryFunny 3y ago

At this point, rather than chasing CUDA/cuDNN compatability, it would seem more productive for AMD to be targetting high level language support. Forget CUDA compatibility, and instead support work such as Mojo and MLIR. It seems that in an ideal world PyTorch support for AMD wouldn't rely on ROCm, but rather be based on high level code compiled to MLIR with AMD target support, with this same MLIR representation supporting Mojo and any other languages such as Julia that want optimized AMD support.

up2isomorphism 3y ago

Any article that trying to make a big deal about 30% performance difference without comparing price and cost is just making a case for itself.