8 ms·

Can someone ELI5 why AMD is not in this game? Is it really so much harder to implement this in a non-platform-specific library?

bilekas 3y ago
I'm no expert but if I understand correctly the CUDA cores are the main pull and the API to them. They're supposed to be more optimized and more stable compared to AMD. That's how it was before anyway, not sure today.
- Aardwolf 3y ago
  Isn't the main component for AI matrix multiplication? What makes it so hard to create a good alternative API for matrix multiplication?
  - bilekas 3y ago
    Well I think there are 2 types right ? Tensor cores (which afaik AMD dont have) which are better for matrix ops, and CUDO which are better for general parallel ops. Maybe someone more clever than me can go into the specifics, I only understand the minimum of the low lvl GPU details. Nice high lvl document [0] https://www.acecloudhosting.com/blog/cuda-cores-vs-tensor-cores/#Difference_Between_CUDA_Cores_and_Tensor_Cores https://www.acecloudhosting.com/blog/cuda-cores-vs-tensor-co...
  - marcyb5st 3y ago
    I think API for matrix multiplication is just a part of the issue. CUDA tooling has better ergonomics, it's easier to set up and treated as first class citizen in tools like Tensorflow and Pytorch. So, while I can't talk about the hardware differences in detail, developer experience is greatly on nVidia side and now AMD has a moat to overcome to catch up.
  - dotnet00 3y ago
    It's a lot more complicated than just writing a matrix multiplication kernel because there are all sorts of operations you need to have on top of matrix multiplication (non linearities, various ways of manipulating the data) and this sort of effort is only really worthwhile if it's well optimized. On top of that, AMD's compute stack is fairly immature, their OpenCL support is buggy and ROCm compiles device specific code, so it has very limited hardware support and is kind of unrealistic to distribute compiled binaries for. Then, getting to the optimization aspect, NVIDIA has many tools which provide detailed information on the GPU's behavior, making it much easier to identify bottlenecks and optimize. AMD is still working on these. Finally, NVIDIA went out of its way to support ML applications. They provide a lot of their own tooling to make using them easier. AMD seems to have struggled on the "easier" part.
  - emmender 3y ago
    there is nccl, gpudirect, nvlink and so on and so forth.. It is not just matmul on gpus.
born-jre 3y ago
short answer is they have somewhat competent hardware but software sucks or you can watch george hotz rant about how amd driver sucks https://www.youtube.com/watch?v=Mr0rWJhv9jU https://www.youtube.com/watch?v=Mr0rWJhv9jU
- alecco 3y ago
  He got a tarball fix for the driver after his rant got viral. Still not looking good IMHO. https://geohot.github.io/blog/jekyll/update/2023/06/07/a-dive-into-amds-drivers.html https://geohot.github.io/blog/jekyll/update/2023/06/07/a-div...
  - xiphias2 3y ago
    It looks like some great engineers inside the company fighting the burocracy.
  - jandrese 3y ago
    > So it fixed the main reported issue! Sadly, they asked me not to distribute it, and gave no more details on what the issue is. I think they missed the thrust of the rant.
- HellDunkel 3y ago
  he goes from building a ml rack out of ati graphics cards to coding some python to recommending reading the unabomber manifesto, from marx to saying he owns a rolls royce... lord, please have mercy!
marcyb5st 3y ago
I think there are several reasons. Firstly, nVidia has been at it much longer. Just because of this tools on nVidia side feel easier to set up / are more polished (at least that was my feeling when fiddling with ROCm like a year ago). Second but still related to #1, from the beginning even consumers nVidia cards were able to run CUDA and this made so that hobbyist and prosumers/researchers on a budget bought nVidia cards compounding even further the time/tooling advantage nVidia had. I.e. a huge user base of not only gamers but people that use their cards to do other things than gaming and know that things work on these cards. These are, IMHO, the main reasons why everyone targets CUDA and explain why frameworks like Tensorflow or Pytorch targeted it as a first class citizen.
- rapsey 3y ago
  If AMD was software competent the frameworks would support their drivers just as well. No one wants a monopoly.
  - marcyb5st 3y ago
    Agreed. Sorry if I gave the impression of being pro-nVidia. I am not. But the reality is that when Tensorflow and Pytorch came to be there was no alternative. Now you need to jump through hoops to make it work with non CUDA hardware. Additionally, while drivers play a role, I think the main difference is in the computing libraries (CUDA vs ROCm)
WithinReason 3y ago
Short version: AMD's software incompetence. Very few hardware companies have the competence to properly support their HW. You see this problem again and again, HW companies designing HW that's great on paper but can't be used properly because it's not properly supported with SW. Nvidia understands this and has 10 times as many SW engineers than HW engineers. AMD doesn't. Intel might too.
- SilverBirch 3y ago
  I think you're totally right with this, just to add - it's often possible to do a lot of things that are neat in hardware but create difficult problems in software. Virtually always when this happens it turns out to be nearly impossible to actually create software that takes advantage of it. So it's massively important to have a closed loop feedback system between the software and hardware so that the hardware guys don't accidentally tie the software up in knots. This is common in companies that consider themselves hardware companies first.
  - davidgl 3y ago
    Examples being the PS3 cell architecture and HP's Itanium chips
- pjc50 3y ago
  Strongly agree. There's a surprising cultural difference between the two. As a software engineer in a different hardware company, I can see where the fault lines are, and it takes continual management effort to make it work properly. (I note that if we had an "open" GPU architecture in the same way that we have CPU architectures, things might be a lot better, but the openness of the IBM PC seems to be a historical accident that no company will allow again)
- SubjectToChange 3y ago
  Chalking it up to “software incompetence” is a bit simplistic, to say the least. AMD was on the brink of bankruptcy not too long ago and their GPU division was struggling to even trend water. They didn’t have an alternative to CUDA because they couldn’t afford one and no one would use it anyway, OpenCL stagnated because most vendors didn’t want to implement functionality that only the biggest players wanted, and their graphics division had to pivot from optimizing for gaming (where they could sell) to optimizing for compute as well. Now that AMD has the capital they are playing catch up to Nvidia. But it’s going to take time for their software to improve. Hiring at boat load of programmers all at once isn’t going to solve that.
captainbland 3y ago
CUDA is the best supported solution, tends to get you access to the best performance, has a great profiler (it will literally tell you things like "your memory accesses don't seem to be coalesced properly" or "your kernel is ALU limited" as well as a bunch of useful stats), even works in windows, all of that. OpenCL is (was?) the main open alternative to CUDA and was mainly backed by AMD and Apple. Apple got bored of it when they decided Metal was the future. AMD got bored of it when they developed rocm and HIP (basically a partially complete compatibility layer with CUDA). There's also stuff like DirectML which only works on windows and e.g. various (Vulkan, directx etc.) compute shaders which are really more oriented at games. There's also a bit of a performance aspect to it. Obviously GPGPU stuff is massively performance sensitive and CUDA gets you the best performance on the most widely supported platform. AMD are also the main competitor in the space but all but totally dropped support for GPGPU in desktop cards, trying to instead focus on gaming for Radeon and compute in MI/CDNA. They seem to have realised their mistake a bit and are now introducing some support for RDNA2+ cards.
- throwaway888abc 3y ago
  Thanks
- znpy 3y ago
  > partially complete compatibility layer So… partial compatibility layer?
  - captainbland 3y ago
    Is your point that "partially complete" is a redundant phrasing? In this case I still prefer my version. I feel that it puts greater emphasis on the fact that it can potentially be complete, given the massive value that could give to the project. Also "I would have written you a shorter letter but I did not have the time" sentiment springs to mind.
- sorenjan 3y ago
  If I remember correctly it's not just that AMD has poor support for their consumer cards, their Rocm code doesn't compile to a device agnostic intermediate, so you have to recompile for each chip. New and old Cuda compatible cards (like all Geforce cards) can run your already shipped Cuda code, as long as it doesn't use new and unsupported features. So even if AMD had supported more cards, the development and user experience would be much worse anyway where you have to find out if your specific card is supported.
lhl 3y ago
llama.cpp can be run with a speedup for AMD GPUs when compiled with `LLAMA_CLBLAST=1` and there is also a HIPified fork [1] being worked on by a community contributor. The other week I was poking on how hard it would be to get an AMD card running w/ acceleration on Linux and was pleasantly surprised, it wasn't too bad: https://mostlyobvious.org/?link=/Reference%2FSoftware%2FGenerative%20AI%2FLLM%20on%20AMD%20GPU%20on%20Arch%20Linux https://mostlyobvious.org/?link=/Reference%2FSoftware%2FGene... That being said, it's important to note that ROCm is Linux only. Not only that, but ROCm's GPU support has actually been decreasing over the past few years. The current list: https://rocm.docs.amd.com/en/latest/release/gpu_os_support.html#gpu-support-table https://rocm.docs.amd.com/en/latest/release/gpu_os_support.h... Previously (2022): https://docs.amd.com/bundle/Hardware_and_Software_Reference_Guide/page/Hardware_and_Software_Support.html https://docs.amd.com/bundle/Hardware_and_Software_Reference_... The ELI5 is that a few years back, AMD split their graphics (RDNA) and compute (CDNA) architectures, which Nvidia does too, but notably (what Nvidia definitely doesn't do, and a key to their success IMO) AMD also decided they would simply not support any CUDA-parity compute features on Windows or their non "compute" cards. In practice, this means that community/open-source developers will never have, tinker, port, or develop on AMD hardware, while on Nvidia you can start with a GTX/RTX card on your laptop, and use the same code up to an H100 or DGX. llama.cpp is a super-high profile project, has almost 200 contributiors now, but AFAIK, no contributors from AMD. If AMD doesn't have the manpower, IMO they should simply be sending nsa free hardware to top open source project/library developers (and on the software side, their #1 priority should be making sure every single current GPU they sell is at least "enabled" if not "supported" in ROCm, on Linux and Windows). [1] https://github.com/SlyEcho/llama.cpp/tree/hipblas https://github.com/SlyEcho/llama.cpp/tree/hipblas
- Kelteseth 3y ago
  I just tried this [1] and it still uses my CPU even though the prompt says otherwise. [1] https://github.com/ggerganov/llama.cpp/issues/1433#issuecomment-1589544870 https://github.com/ggerganov/llama.cpp/issues/1433#issuecomm...
  - lhl 3y ago
    I saw there was an answer already in your issue, although you plan on doing a lot of inferencing on your GPU, I'd highly recommend you consider dual-booting into Linux. It turns out exllama merged ROCm support last week and more than 2X faster than the CLBlast code. A 13b gptq model at full context clocks in at 15t/s on my old Radeon VII. (Rumor has it that ROCm 5.6 may add Windows support, although it remains to be seen what that exactly entails.)
lofaszvanitt 3y ago
Planned economy, planned who does what.
sorenjan 3y ago
I'm hopeful for SYCL [0] to become the cross platform alternative, but there doesn't seem to be a lot of uptake from projects like this, so maybe my hope is misplaced. It's an official Khronos standard, and Intel seems to like it [1], but neither of those things are enough to change things. Someone who knows about this space that can comment on the likelihood that SYCL will be a good option eventually? Cross platform and cross vendor compatibility would be really nice, and not supporting the proprietary de facto standard would also be a bonus as long as the alternative works well enough. [0] https://www.khronos.org/sycl/ https://www.khronos.org/sycl/ [1] https://spec.oneapi.io/versions/latest/elements/sycl/source/index.html https://spec.oneapi.io/versions/latest/elements/sycl/source/...
- mschuetz 3y ago
  > It's an official Khronos standard I think that's the problem. Khronos isn't known for good UX, and being from Khronos is exactly the reason why I'm not even bothering to check it out. I want an alternative to CUDA, but I also want it to be as easy to use as CUDA.
  - LoganDark 3y ago
    Being from Khronos is also a reason why it might actually be usable in a decade's time, like Vulkan. (Vulkan is from 2015 and is just recently starting to become usable.)
    - sicariusnoctis 3y ago
      From a non-expert's standpoint, Vulkan feels quite unusable and complex.
      - mschuetz 3y ago
        From an expert's standpoint, it's still quite unusable and complex.

bilekas 3y ago

I'm no expert but if I understand correctly the CUDA cores are the main pull and the API to them. They're supposed to be more optimized and more stable compared to AMD. That's how it was before anyway, not sure today.

Aardwolf 3y ago
Isn't the main component for AI matrix multiplication? What makes it so hard to create a good alternative API for matrix multiplication?
- bilekas 3y ago
  Well I think there are 2 types right ? Tensor cores (which afaik AMD dont have) which are better for matrix ops, and CUDO which are better for general parallel ops. Maybe someone more clever than me can go into the specifics, I only understand the minimum of the low lvl GPU details. Nice high lvl document [0] https://www.acecloudhosting.com/blog/cuda-cores-vs-tensor-cores/#Difference_Between_CUDA_Cores_and_Tensor_Cores https://www.acecloudhosting.com/blog/cuda-cores-vs-tensor-co...
- marcyb5st 3y ago
  I think API for matrix multiplication is just a part of the issue. CUDA tooling has better ergonomics, it's easier to set up and treated as first class citizen in tools like Tensorflow and Pytorch. So, while I can't talk about the hardware differences in detail, developer experience is greatly on nVidia side and now AMD has a moat to overcome to catch up.
- dotnet00 3y ago
  It's a lot more complicated than just writing a matrix multiplication kernel because there are all sorts of operations you need to have on top of matrix multiplication (non linearities, various ways of manipulating the data) and this sort of effort is only really worthwhile if it's well optimized. On top of that, AMD's compute stack is fairly immature, their OpenCL support is buggy and ROCm compiles device specific code, so it has very limited hardware support and is kind of unrealistic to distribute compiled binaries for. Then, getting to the optimization aspect, NVIDIA has many tools which provide detailed information on the GPU's behavior, making it much easier to identify bottlenecks and optimize. AMD is still working on these. Finally, NVIDIA went out of its way to support ML applications. They provide a lot of their own tooling to make using them easier. AMD seems to have struggled on the "easier" part.
- emmender 3y ago
  there is nccl, gpudirect, nvlink and so on and so forth.. It is not just matmul on gpus.

born-jre 3y ago

short answer is they have somewhat competent hardware but software sucks or you can watch george hotz rant about how amd driver sucks https://www.youtube.com/watch?v=Mr0rWJhv9jU https://www.youtube.com/watch?v=Mr0rWJhv9jU

alecco 3y ago
He got a tarball fix for the driver after his rant got viral. Still not looking good IMHO. https://geohot.github.io/blog/jekyll/update/2023/06/07/a-dive-into-amds-drivers.html https://geohot.github.io/blog/jekyll/update/2023/06/07/a-div...
- xiphias2 3y ago
  It looks like some great engineers inside the company fighting the burocracy.
- jandrese 3y ago
  > So it fixed the main reported issue! Sadly, they asked me not to distribute it, and gave no more details on what the issue is. I think they missed the thrust of the rant.
HellDunkel 3y ago
he goes from building a ml rack out of ati graphics cards to coding some python to recommending reading the unabomber manifesto, from marx to saying he owns a rolls royce... lord, please have mercy!

marcyb5st 3y ago

I think there are several reasons. Firstly, nVidia has been at it much longer. Just because of this tools on nVidia side feel easier to set up / are more polished (at least that was my feeling when fiddling with ROCm like a year ago). Second but still related to #1, from the beginning even consumers nVidia cards were able to run CUDA and this made so that hobbyist and prosumers/researchers on a budget bought nVidia cards compounding even further the time/tooling advantage nVidia had. I.e. a huge user base of not only gamers but people that use their cards to do other things than gaming and know that things work on these cards. These are, IMHO, the main reasons why everyone targets CUDA and explain why frameworks like Tensorflow or Pytorch targeted it as a first class citizen.

rapsey 3y ago
If AMD was software competent the frameworks would support their drivers just as well. No one wants a monopoly.
- marcyb5st 3y ago
  Agreed. Sorry if I gave the impression of being pro-nVidia. I am not. But the reality is that when Tensorflow and Pytorch came to be there was no alternative. Now you need to jump through hoops to make it work with non CUDA hardware. Additionally, while drivers play a role, I think the main difference is in the computing libraries (CUDA vs ROCm)

WithinReason 3y ago

Short version: AMD's software incompetence. Very few hardware companies have the competence to properly support their HW. You see this problem again and again, HW companies designing HW that's great on paper but can't be used properly because it's not properly supported with SW. Nvidia understands this and has 10 times as many SW engineers than HW engineers. AMD doesn't. Intel might too.

SilverBirch 3y ago
I think you're totally right with this, just to add - it's often possible to do a lot of things that are neat in hardware but create difficult problems in software. Virtually always when this happens it turns out to be nearly impossible to actually create software that takes advantage of it. So it's massively important to have a closed loop feedback system between the software and hardware so that the hardware guys don't accidentally tie the software up in knots. This is common in companies that consider themselves hardware companies first.
- davidgl 3y ago
  Examples being the PS3 cell architecture and HP's Itanium chips
pjc50 3y ago
Strongly agree. There's a surprising cultural difference between the two. As a software engineer in a different hardware company, I can see where the fault lines are, and it takes continual management effort to make it work properly. (I note that if we had an "open" GPU architecture in the same way that we have CPU architectures, things might be a lot better, but the openness of the IBM PC seems to be a historical accident that no company will allow again)
SubjectToChange 3y ago
Chalking it up to “software incompetence” is a bit simplistic, to say the least. AMD was on the brink of bankruptcy not too long ago and their GPU division was struggling to even trend water. They didn’t have an alternative to CUDA because they couldn’t afford one and no one would use it anyway, OpenCL stagnated because most vendors didn’t want to implement functionality that only the biggest players wanted, and their graphics division had to pivot from optimizing for gaming (where they could sell) to optimizing for compute as well. Now that AMD has the capital they are playing catch up to Nvidia. But it’s going to take time for their software to improve. Hiring at boat load of programmers all at once isn’t going to solve that.

captainbland 3y ago

CUDA is the best supported solution, tends to get you access to the best performance, has a great profiler (it will literally tell you things like "your memory accesses don't seem to be coalesced properly" or "your kernel is ALU limited" as well as a bunch of useful stats), even works in windows, all of that. OpenCL is (was?) the main open alternative to CUDA and was mainly backed by AMD and Apple. Apple got bored of it when they decided Metal was the future. AMD got bored of it when they developed rocm and HIP (basically a partially complete compatibility layer with CUDA). There's also stuff like DirectML which only works on windows and e.g. various (Vulkan, directx etc.) compute shaders which are really more oriented at games. There's also a bit of a performance aspect to it. Obviously GPGPU stuff is massively performance sensitive and CUDA gets you the best performance on the most widely supported platform. AMD are also the main competitor in the space but all but totally dropped support for GPGPU in desktop cards, trying to instead focus on gaming for Radeon and compute in MI/CDNA. They seem to have realised their mistake a bit and are now introducing some support for RDNA2+ cards.

throwaway888abc 3y ago
Thanks
znpy 3y ago
> partially complete compatibility layer So… partial compatibility layer?
- captainbland 3y ago
  Is your point that "partially complete" is a redundant phrasing? In this case I still prefer my version. I feel that it puts greater emphasis on the fact that it can potentially be complete, given the massive value that could give to the project. Also "I would have written you a shorter letter but I did not have the time" sentiment springs to mind.
sorenjan 3y ago
If I remember correctly it's not just that AMD has poor support for their consumer cards, their Rocm code doesn't compile to a device agnostic intermediate, so you have to recompile for each chip. New and old Cuda compatible cards (like all Geforce cards) can run your already shipped Cuda code, as long as it doesn't use new and unsupported features. So even if AMD had supported more cards, the development and user experience would be much worse anyway where you have to find out if your specific card is supported.

lhl 3y ago

llama.cpp can be run with a speedup for AMD GPUs when compiled with `LLAMA_CLBLAST=1` and there is also a HIPified fork [1] being worked on by a community contributor. The other week I was poking on how hard it would be to get an AMD card running w/ acceleration on Linux and was pleasantly surprised, it wasn't too bad: https://mostlyobvious.org/?link=/Reference%2FSoftware%2FGenerative%20AI%2FLLM%20on%20AMD%20GPU%20on%20Arch%20Linux https://mostlyobvious.org/?link=/Reference%2FSoftware%2FGene... That being said, it's important to note that ROCm is Linux only. Not only that, but ROCm's GPU support has actually been decreasing over the past few years. The current list: https://rocm.docs.amd.com/en/latest/release/gpu_os_support.html#gpu-support-table https://rocm.docs.amd.com/en/latest/release/gpu_os_support.h... Previously (2022): https://docs.amd.com/bundle/Hardware_and_Software_Reference_Guide/page/Hardware_and_Software_Support.html https://docs.amd.com/bundle/Hardware_and_Software_Reference_... The ELI5 is that a few years back, AMD split their graphics (RDNA) and compute (CDNA) architectures, which Nvidia does too, but notably (what Nvidia definitely doesn't do, and a key to their success IMO) AMD also decided they would simply not support any CUDA-parity compute features on Windows or their non "compute" cards. In practice, this means that community/open-source developers will never have, tinker, port, or develop on AMD hardware, while on Nvidia you can start with a GTX/RTX card on your laptop, and use the same code up to an H100 or DGX. llama.cpp is a super-high profile project, has almost 200 contributiors now, but AFAIK, no contributors from AMD. If AMD doesn't have the manpower, IMO they should simply be sending nsa free hardware to top open source project/library developers (and on the software side, their #1 priority should be making sure every single current GPU they sell is at least "enabled" if not "supported" in ROCm, on Linux and Windows). [1] https://github.com/SlyEcho/llama.cpp/tree/hipblas https://github.com/SlyEcho/llama.cpp/tree/hipblas

Kelteseth 3y ago
I just tried this [1] and it still uses my CPU even though the prompt says otherwise. [1] https://github.com/ggerganov/llama.cpp/issues/1433#issuecomment-1589544870 https://github.com/ggerganov/llama.cpp/issues/1433#issuecomm...
- lhl 3y ago
  I saw there was an answer already in your issue, although you plan on doing a lot of inferencing on your GPU, I'd highly recommend you consider dual-booting into Linux. It turns out exllama merged ROCm support last week and more than 2X faster than the CLBlast code. A 13b gptq model at full context clocks in at 15t/s on my old Radeon VII. (Rumor has it that ROCm 5.6 may add Windows support, although it remains to be seen what that exactly entails.)

lofaszvanitt 3y ago

Planned economy, planned who does what.

sorenjan 3y ago

I'm hopeful for SYCL [0] to become the cross platform alternative, but there doesn't seem to be a lot of uptake from projects like this, so maybe my hope is misplaced. It's an official Khronos standard, and Intel seems to like it [1], but neither of those things are enough to change things. Someone who knows about this space that can comment on the likelihood that SYCL will be a good option eventually? Cross platform and cross vendor compatibility would be really nice, and not supporting the proprietary de facto standard would also be a bonus as long as the alternative works well enough. [0] https://www.khronos.org/sycl/ https://www.khronos.org/sycl/ [1] https://spec.oneapi.io/versions/latest/elements/sycl/source/index.html https://spec.oneapi.io/versions/latest/elements/sycl/source/...

mschuetz 3y ago
> It's an official Khronos standard I think that's the problem. Khronos isn't known for good UX, and being from Khronos is exactly the reason why I'm not even bothering to check it out. I want an alternative to CUDA, but I also want it to be as easy to use as CUDA.
- LoganDark 3y ago
  Being from Khronos is also a reason why it might actually be usable in a decade's time, like Vulkan. (Vulkan is from 2015 and is just recently starting to become usable.)
  - sicariusnoctis 3y ago
    From a non-expert's standpoint, Vulkan feels quite unusable and complex.
    - mschuetz 3y ago
      From an expert's standpoint, it's still quite unusable and complex.