3 ms·

What is the root cause of AMD not being good for AI?

arcanemachiner 3y ago
“Nvidia's SDK, CUDA, is the industry standard and they have built a huge range of tools on top of that for training large models, inference-serving, optimization, transfer learning, infrastructure management and operations automation, and a lot of vertical-specific applications.” https://www.google.com/search?q=why%20is%20nvidia%20better%20for%20ai https://www.google.com/search?q=why%20is%20nvidia%20better%2...
- mjan22640 3y ago
  So its not the hw, but an API issue? Is it not possible to port the tools to the AMD API? Or does the AMD API not allow to properly utilize the compute power of the hw?
  - ekianjo 3y ago
    No CUDA is heavily integrated with how Nvidia hardware operates
  - dagw 3y ago
    AMDs APIs have historically been way behind CUDA, both in features and ease of use, making it much harder to reach feature and performance parity with CUDA. The second big problem is that AMD has a history of abandoning APIs. While Nvidia has been focusing on CUDA from the start, AMD is on its third or fourth API. A lot of developers got burned early on by putting in a lot of effort into supporting AMD cards, only to have their work become useless as AMD dropped old API. So if you wanted to support AMD today you have to choose between using their latest API which is currently only supported on a handful of their latest top end cards, or using their previous API, which is much worse and may or may not be supported in 2 years time.
    - slavik81 3y ago
      > their latest API [...] is currently only supported on a handful of their latest top end cards While there's very few cards that are officially supported, I've enabled most RDNA 1 and RDNA 2 GPUs in the OS-provided packages on Debian Unstable and Ubuntu 23.10. I haven't tested every architecture, but my experience has been that they generally work quite well. The version of HIP packaged for Debian is still a few versions behind, so we haven't been able to enable RDNA 3 yet, but it shouldn't be too long before that is resolved. As you have described, it has been a very long road to get to this point (with a lot of mistakes along the way), but I think things are getting better. That said, I work on ROCm, so my evaluation may be unintentionally biased.
      - dagw 3y ago
        I think things are getting better Hopefully. Although I haven't programmed with it yet, the latest ROCm looks nice (at least from reading the docs). Now all you have to do is actually get to a point where it's actually supported on hardware people actually own and convince people that it's worth investing their time in, and then AMD can start making progress.
        slavik81 3y ago
        The Debian packages support a much wider range of hardware than the AMD packages do. Just about any discrete AMD GPU hardware released in the past five years should already work or will work soon.
  - mananaysiempre 3y ago
    CUDA is proprietary to Nvidia, and it’s unlikely they would ever allow implementing it on other vendors’ hardware. DirectML mentioned here is proprietary to Microsoft but at least hardware-neutral. It is also much less popular. OpenCL is open but receives hardly any usage at all, unfortunately. (To be fair, Nvidia were at the forefront of this GPGPU thing since before there were dedicated APIs for it, so they certainly deserve some credit, but at this point they seem in desperate need for someone to finally knock them down.)
  - Karliss 3y ago
    Nvidia has provided the same API incrementally extending it since 2007. And supported it on most of their GPUS including consumer cards. Oldest cards supported by the latest version of their SDK were made in ~2013-2016. See table in https://en.wikipedia.org/wiki/CUDA#GPUs_supported https://en.wikipedia.org/wiki/CUDA#GPUs_supported AMD has been a lot less consistent with their support changing their mind almost every 3 years. ROCm was initially released in 2016 (that's 9 years of headstart for NVIDIA). If you look at https://docs.amd.com/en/latest/release/gpu_os_support.html https://docs.amd.com/en/latest/release/gpu_os_support.html on Linux only the enterprise server cards are supported. Situation slightly different on windows https://docs.amd.com/en/latest/release/windows_support.html https://docs.amd.com/en/latest/release/windows_support.html where they support last 2 generations of consumer cards (partially). I remember there were also a few years with even more bizarre situation where they supported GPUs that are couple of generations old but not their latest cards. Not hard to see how developers are hesitant to spend time adding support for AMD if large portion of AMD users won't be able to use it and there is a high chance that after few years AMD will change their mind again coming out with new API or dropping support for cards that are only couple of years old.
  - whywhywhywhy 3y ago
    CUDA has over a decade of tooling and talent built up while AMD/Apple fumbled with the far more awkward to develop with OpenCL that they eventually killed. This disparity is due to that fumble.
mmis1000 3y ago
The software support for AMD gpu is probably lags 10 years or so after Nvidia. They just release there sdk for windows this year (more specifically 2023-05-24, Rocm 5.5.1). While CUDA exists for both platform like forever. So there won't be a common user group besides some lab (which have the man power to invest in linux software and setup new tool chains) that use AMD card for AI related task. But since Rocm for windows finally released. The situation may finally change. We can probably just wait and see what will go on.
- selfhoster11 3y ago
  My impression was rather that CUDA is successful because it will work on any NVidia GPU that's not yet old enough to vote, which means a number of things: - people had lots of time to get used to it - hobbyists and smaller research outfits/labs could make use of it on the cheap and less powerful hardware (where even a cheap GPGPU beats computing in software), driving: - mind share (compare: Photoshop/Microsoft's tolerance to piracy) - sales of hardware when the users decided to upgrade, buying more and/or and/or more modern and/or better cards - the longer it has been available, the more stable of a programming target it appears to be, driving a virtuous cycle In contrast, the range of AMD hardware supporting ROCm was narrow and unstable (AFAIK one card was only supported for a couple of years). This would of course make ROCm unattractive for building with, especially for the hobbyists and smaller outfits that would otherwise drive mindshare but couldn't afford datacenter grade hardware that was supported.
  - mmis1000 3y ago
    Recent rocm on windows also supports more cards than it used to be. Consumer rdna2/3 cards are mostly on the support list. Although these only available on windows for some reason. And the only consumer card supported on linux is still 'AMD Radeon™ VII'. No idea if AMD will drop whole list like it previously done this time though.

“Nvidia's SDK, CUDA, is the industry standard and they have built a huge range of tools on top of that for training large models, inference-serving, optimization, transfer learning, infrastructure management and operations automation, and a lot of vertical-specific applications.” https://www.google.com/search?q=why%20is%20nvidia%20better%20for%20ai https://www.google.com/search?q=why%20is%20nvidia%20better%2...

mjan22640 3y ago
So its not the hw, but an API issue? Is it not possible to port the tools to the AMD API? Or does the AMD API not allow to properly utilize the compute power of the hw?
- ekianjo 3y ago
  No CUDA is heavily integrated with how Nvidia hardware operates
- dagw 3y ago
  AMDs APIs have historically been way behind CUDA, both in features and ease of use, making it much harder to reach feature and performance parity with CUDA. The second big problem is that AMD has a history of abandoning APIs. While Nvidia has been focusing on CUDA from the start, AMD is on its third or fourth API. A lot of developers got burned early on by putting in a lot of effort into supporting AMD cards, only to have their work become useless as AMD dropped old API. So if you wanted to support AMD today you have to choose between using their latest API which is currently only supported on a handful of their latest top end cards, or using their previous API, which is much worse and may or may not be supported in 2 years time.
  - slavik81 3y ago
    > their latest API [...] is currently only supported on a handful of their latest top end cards While there's very few cards that are officially supported, I've enabled most RDNA 1 and RDNA 2 GPUs in the OS-provided packages on Debian Unstable and Ubuntu 23.10. I haven't tested every architecture, but my experience has been that they generally work quite well. The version of HIP packaged for Debian is still a few versions behind, so we haven't been able to enable RDNA 3 yet, but it shouldn't be too long before that is resolved. As you have described, it has been a very long road to get to this point (with a lot of mistakes along the way), but I think things are getting better. That said, I work on ROCm, so my evaluation may be unintentionally biased.
    - dagw 3y ago
      I think things are getting better Hopefully. Although I haven't programmed with it yet, the latest ROCm looks nice (at least from reading the docs). Now all you have to do is actually get to a point where it's actually supported on hardware people actually own and convince people that it's worth investing their time in, and then AMD can start making progress.
      - slavik81 3y ago
        The Debian packages support a much wider range of hardware than the AMD packages do. Just about any discrete AMD GPU hardware released in the past five years should already work or will work soon.
- mananaysiempre 3y ago
  CUDA is proprietary to Nvidia, and it’s unlikely they would ever allow implementing it on other vendors’ hardware. DirectML mentioned here is proprietary to Microsoft but at least hardware-neutral. It is also much less popular. OpenCL is open but receives hardly any usage at all, unfortunately. (To be fair, Nvidia were at the forefront of this GPGPU thing since before there were dedicated APIs for it, so they certainly deserve some credit, but at this point they seem in desperate need for someone to finally knock them down.)
- Karliss 3y ago
  Nvidia has provided the same API incrementally extending it since 2007. And supported it on most of their GPUS including consumer cards. Oldest cards supported by the latest version of their SDK were made in ~2013-2016. See table in https://en.wikipedia.org/wiki/CUDA#GPUs_supported https://en.wikipedia.org/wiki/CUDA#GPUs_supported AMD has been a lot less consistent with their support changing their mind almost every 3 years. ROCm was initially released in 2016 (that's 9 years of headstart for NVIDIA). If you look at https://docs.amd.com/en/latest/release/gpu_os_support.html https://docs.amd.com/en/latest/release/gpu_os_support.html on Linux only the enterprise server cards are supported. Situation slightly different on windows https://docs.amd.com/en/latest/release/windows_support.html https://docs.amd.com/en/latest/release/windows_support.html where they support last 2 generations of consumer cards (partially). I remember there were also a few years with even more bizarre situation where they supported GPUs that are couple of generations old but not their latest cards. Not hard to see how developers are hesitant to spend time adding support for AMD if large portion of AMD users won't be able to use it and there is a high chance that after few years AMD will change their mind again coming out with new API or dropping support for cards that are only couple of years old.
- whywhywhywhy 3y ago
  CUDA has over a decade of tooling and talent built up while AMD/Apple fumbled with the far more awkward to develop with OpenCL that they eventually killed. This disparity is due to that fumble.

mmis1000 3y ago

The software support for AMD gpu is probably lags 10 years or so after Nvidia. They just release there sdk for windows this year (more specifically 2023-05-24, Rocm 5.5.1). While CUDA exists for both platform like forever. So there won't be a common user group besides some lab (which have the man power to invest in linux software and setup new tool chains) that use AMD card for AI related task. But since Rocm for windows finally released. The situation may finally change. We can probably just wait and see what will go on.

selfhoster11 3y ago
My impression was rather that CUDA is successful because it will work on any NVidia GPU that's not yet old enough to vote, which means a number of things: - people had lots of time to get used to it - hobbyists and smaller research outfits/labs could make use of it on the cheap and less powerful hardware (where even a cheap GPGPU beats computing in software), driving: - mind share (compare: Photoshop/Microsoft's tolerance to piracy) - sales of hardware when the users decided to upgrade, buying more and/or and/or more modern and/or better cards - the longer it has been available, the more stable of a programming target it appears to be, driving a virtuous cycle In contrast, the range of AMD hardware supporting ROCm was narrow and unstable (AFAIK one card was only supported for a couple of years). This would of course make ROCm unattractive for building with, especially for the hobbyists and smaller outfits that would otherwise drive mindshare but couldn't afford datacenter grade hardware that was supported.
- mmis1000 3y ago
  Recent rocm on windows also supports more cards than it used to be. Consumer rdna2/3 cards are mostly on the support list. Although these only available on windows for some reason. And the only consumer card supported on linux is still 'AMD Radeon™ VII'. No idea if AMD will drop whole list like it previously done this time though.