39 ms·

I'm really rooting for AMD to break the CUDA monopoly. To this end, I genuinely don't know whether a translation layer is a good thing or not. On the upside it

I'm really rooting for AMD to break the CUDA monopoly. To this end, I genuinely don't know whether a translation layer is a good thing or not. On the upside it makes the hardware much more viable instantly and will boost adoption, on the downside you run the risk that devs will never support ROCm, because you can just use the translation layer. I think this is essentially the same situation as Proton+DXVK for Linux gaming. I think that that is a net positive for Linux, but I'm less sure about this. Getting good performance out of GPU compute requires much more tuning to the concrete architecture, which I'm afraid devs just won't do for AMD GPUs through this layer, always leaving them behind their Nvidia counterparts. However, AMD desperately needs to do something. Story time: On the weekend I wanted to play around with Stable Diffusion. Why pay for cloud compute, when I have a powerful GPU at home, I thought. Said GPU is a 7900 XTX, i.e. the most powerful consumer card from AMD at this time. Only very few AMD GPUs are supported by ROCm at this time, but mine is, thankfully. So, how hard could it possibly to get Stable Diffusion running on my GPU? Hard. I don't think my problems were actually caused by AMD: I had ROCm installed and my card recognized by rocminfo in a matter of minutes. But the whole ML world is so focused on Nvidia that it took me ages to get a working installation of pytorch and friends. The InvokeAI installer, for example, asks if you want to use CUDA or ROCm, but then always installs the CUDA variant whatever you answer. Ultimately, I did get a model to load, but the software crashed my graphical session before generating a single image. The whole experience left me frustrated and wanting to buy an Nvidia GPU again...

Certhas 3y ago
They are focusing on HPC first. Which seems reasonable if your software stack is lacking. Look for sophisticated customers that can help build an ecosystem. As I mentioned elsewhere, 25% of GPU compute on the Top 500 Supercomputer list is AMD. This all on the back of a card that came out only three years ago. We are very rapidly moving towards a situation where there are many, many high-performance developers that will target ROCm.
- ametrau 3y ago
  Is a top 500 super computer list a good way of measuring relevancy in the future?
  - latchkey 3y ago
    No, it isn't. What is a better measure is to look at businesses like what I'm building (and others), where we take on the capex/opex risk around top end AMD products and bring them to the masses through bare metal rentals. Previously, these sorts of cards were only available to the Top 500.
  - llm_trw 3y ago
    Yes it is, it's how cuda got it's dominance 10 years ago. Businesses don't release their source code, super computers are attached to labs and universities and have much better licenses for software, or publish papers about it.
whywhywhywhy 3y ago
> I'm really rooting for AMD to break the CUDA monopoly Personally I want Nvidia to break the x86-64 monopoly, with how amazing properly spec'd Nvidia cards are to work with I can only dream of a world where Nvidia is my CPU too.
- kuschkufan 3y ago
  apt username
- smcleod 3y ago
  That’s already been done with ARM.
- weebull 3y ago
  > Personally I want Nvidia to break the x86-64 monopoly The one supplied by two companies?
  - Keyframe 3y ago
    Maybe he meant homogeneity which Nvidia did try and tries with Arm.. but, on the other hand how wild would it be for Nvidia to enter x86-64 as well? It's probably never going to happen due to licensing if nothing else, lest we remember nForce chipset ordeal with intel legal.
    - paulmd 3y ago
      https://en.wikipedia.org/wiki/Project_Denver#History https://en.wikipedia.org/wiki/Project_Denver#History
    - weebull 3y ago
      Indeed, but I think people forget that the reason AMD have a license in the first place was because Intel's customers in the early days required a second source for it's processors. Who owns the Cyrix x86 license these days?
  - paulmd 3y ago
bntyhntr 3y ago
I would love to be able to have a native stable diffusion experience, my rx 580 takes 30s to generate a single image. But it does work after following https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Install-and-Run-on-AMD-GPUs https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki... I got this up and running on my windows machine in short order and I don't even know what stable diffusion is. But again, it would be nice to have first class support to locally participate in the fun.
- Cu3PO42 3y ago
  I have heard that DirectML was a somewhat easier story, but allegedly has worse performance (and obviously it's Windows only...). But I'm not entirely suprised that setup is somewhat easier on Windows, where bundling everything is an accepted approach. With AMD's official 15GB(!) Docker image, I was now able to get the A1111 UI running. With SD 1.5 and 30 sample iterations, generating an image takes under 2s. I'm still struggling to get InvokeAI running.
  - washadjeffmad 3y ago
    That has to include the model(s), no? Also, nothing is easier on Windows. It's a wonder that anything works there, except for the power of recalcitrance. Not dogging Windows users, but once your brain heals, it just can't go back.
    - Cu3PO42 3y ago
      It actually doesn't include the models! The image is Ubuntu with ROCm and a number of ML libraries, such as Torch, preinstalled. > Also, nothing is easier on Windows. As much as I, too, dislike Windows, I still have to disagree. I have encountered (proprietary) software which was much easier to get working on Windows. For example, Cisco AnyConnect with SmartCard authentication has been a nightmare for me on Linux.
westurner 3y ago
> Proton+DXVK for Linux gaming "Building the DirectX shader compiler better than Microsoft?" (2024) https://news.ycombinator.com/item?id=39324800 https://news.ycombinator.com/item?id=39324800 E.g. llama.cpp already supports hipBLAS; is there an advantage to this ROCm CUDA-compatibility layer - ZLUDA on Radeon (and not yet Intel OneAPI) - instead or in addition? https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#hipblas https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#hi... https://news.ycombinator.com/item?id=38588573 https://news.ycombinator.com/item?id=38588573 What can't WebGPU abstract away from CUDA unportability? https://news.ycombinator.com/item?id=38527552 https://news.ycombinator.com/item?id=38527552
- HarHarVeryFunny 3y ago
  BLAS will only get you so far. About the highest level operation it has is matmul, which you can use to build convolution (im2col, matmul, col2im), but that won't be as performant as a hand optimized cuDNN convolution kernel. Same goes for any other high level neural net building blocks - trying to build them on top of BLAS will not get you remotely close to performance of a custom kernel. What's nice about BLAS is that there are optimized implementations for CPUs (Intel MKL) as well as NVIDIA (cuBLAS) and AMD (hipBLAS), so while it's very much limited in what it can do, you can at least write portable code around it.
  - westurner 3y ago
    "CUDNN API supported by HIP" has a coverage table: https://rocm.docs.amd.com/projects/HIPIFY/en/amd-staging/tables/CUDNN_API_supported_by_HIP.html https://rocm.docs.amd.com/projects/HIPIFY/en/amd-staging/tab... ROCm/hipDNN wraps CuDNN on Nvidia and MiOpen on AMD; but hasn't been updated in awhile: https://github.com/ROCm/hipDNN https://github.com/ROCm/hipDNN https://news.ycombinator.com/item?id=37808036 https://news.ycombinator.com/item?id=37808036 : conda-forge has various BLAS implementations, including MKL-optimized BLAS, and compatible NumPy and SciPy builds. BLAS: Basic Linear Algebra Sub programs: https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprogra... "Using CuPy on AMD GPU (experimental)" https://docs.cupy.dev/en/v13.0.0/install.html#using-cupy-on-amd-gpu-experimental https://docs.cupy.dev/en/v13.0.0/install.html#using-cupy-on-... : $ sudo apt install hipblas hipsparse rocsparse rocrand rocthrust rocsolver rocfft hipcub rocprim rccl
nocombination 3y ago
As other folks have commented, CUDA not being an open standard is a large part of the problem. That and the developers who target CUDA directly when writing Stable Diffusion algorithms—they are forcing the monopoly. Even at the cost of not being able to squeeze every ounce out of the GPU, portability greatly improves software access when people target Vulkan et al.
formerly_proven 3y ago
> I'm really rooting for AMD to break the CUDA monopoly. To this end, I genuinely don't know whether a translation layer is a good thing or not. On the upside it makes the hardware much more viable instantly and will boost adoption, on the downside you run the risk that devs will never support ROCm, because you can just use the translation layer. On the other hand: > The next major ROCm release (ROCm 6.0) will not be backward [source] compatible with the ROCm 5 series. Even worse, not even the driver is backwards-compatible: > There are some known limitations though like currently only targeting the ROCm 5.x API and not the newly-released ROCm 6.x releases.. In turn having to stick to ROCm 5.7 series as the latest means that using the ROCm DKMS modules don't build against the Linux 6.5 kernel now shipped by Ubuntu 22.04 LTS HWE stacks, for example. Hopefully there will be enough community support to see ZLUDA ported to ROCM 6 so at least it can be maintained with current software releases.
nialv7 3y ago
I am surprised that everybody seem to have forgotten the (in)famous Embrace, Extend and Extinguish strategy. It's time for Open Source to be on the extinguishing side for once.
sophrocyne 3y ago
Hey there - I'm a maintainer (and CEO) of Invoke. It's something we're monitoring as well. ROCm has been challenging to work with - we're actively talking to AMD to keep apprised of ways we can mitigate some of the more troublesome experiences that users have with getting Invoke running on AMD (and hoping to expand official support to Windows AMD) The problem is that a lot of the solutions proposed involve significant/unsustainable dev effort (i.e., supporting an entirely different inference paradigm), rather than "drop in" for the existing Torch/diffusers pipelines. While I don't know enough about your set up to offer immediate solutions, if you join the discord, am sure folks would be happy to try walking through some manual troubleshooting/experimentation to get you up and running - discord.gg/invoke-ai
- latchkey 3y ago
  Invoke is awesome. Let me know if you guys want some MI300x to develop/test on. =) We've also got some good contacts at AMD if you need help there as well.
- Cu3PO42 3y ago
  Hi! I really appreciate you taking the time to reply. I have since gotten Invoke to run and was already able to get some results I'm really quite happy with, so thank you for your time and commitment working on Invoke! I understand that ROCm is still challenging, but it seems my problems were less related to ROCm or Invoke itself and more to Python dependency management. It really boiled down to getting the correct (ROCm) versions of packages installed. Installing Invoke from PyPi always removed my Torch and installed CUDA-enabled Torch (as well as cuBLAS, cuDNN, ...). Once I had the correct versions of packages, everything just worked. To me, your pyproject.toml looks perfectly sane, so I wasn't sure how to go about fixing the problem. What ended up working for me was to use one of AMD's ROCm OCI base images, manually installing all dependencies, foregoing a virtual environment, cloning your repo (, building the frontend), and then installing from there. The majority of my struggle would have been solved by a recent working Docker image containing a working setup. (The one on Docker Hub is 9 months old.) Trying to build the Dockerfile from your repo, I also ended up with a CUDA-enabled Torch. It did install the correct one first, but in a later step removed the ROCm-enabled Torch to switch it for the CUDA-enabled one. I hope you'll consider investing some resources into publishing newer, working builds of your Docker image.
bavell 3y ago
Try with ComfyUI... works great and easy setup on my 6750XT. I've had it working for about a year now with SD, LlamaCpp and WhisperCpp.
mtrower 3y ago
This is the exact reason* I bought a 4090 for my recent rebuild instead of the rDNA card I actually wanted. I really wanted to go with AMD for the driver integration with the Linux graphics stack —- I’m so, so tired of shenanigans when it comes to decades old features of X not working or working poorly due to some nvidia bug/non-integration. But being able to leverage my graphics card for GPGPU was a top priority for me, and like you, I was appalled with the ROCm situation. Not necessarily the tech itself (though I did not enjoy the docker approach), but more the developer situation surrounding it. * well, that and some vague notions about RTX

Certhas 3y ago

They are focusing on HPC first. Which seems reasonable if your software stack is lacking. Look for sophisticated customers that can help build an ecosystem. As I mentioned elsewhere, 25% of GPU compute on the Top 500 Supercomputer list is AMD. This all on the back of a card that came out only three years ago. We are very rapidly moving towards a situation where there are many, many high-performance developers that will target ROCm.

ametrau 3y ago
Is a top 500 super computer list a good way of measuring relevancy in the future?
- latchkey 3y ago
  No, it isn't. What is a better measure is to look at businesses like what I'm building (and others), where we take on the capex/opex risk around top end AMD products and bring them to the masses through bare metal rentals. Previously, these sorts of cards were only available to the Top 500.
- llm_trw 3y ago
  Yes it is, it's how cuda got it's dominance 10 years ago. Businesses don't release their source code, super computers are attached to labs and universities and have much better licenses for software, or publish papers about it.

whywhywhywhy 3y ago

> I'm really rooting for AMD to break the CUDA monopoly Personally I want Nvidia to break the x86-64 monopoly, with how amazing properly spec'd Nvidia cards are to work with I can only dream of a world where Nvidia is my CPU too.

kuschkufan 3y ago
apt username
smcleod 3y ago
That’s already been done with ARM.
weebull 3y ago
> Personally I want Nvidia to break the x86-64 monopoly The one supplied by two companies?
- Keyframe 3y ago
  Maybe he meant homogeneity which Nvidia did try and tries with Arm.. but, on the other hand how wild would it be for Nvidia to enter x86-64 as well? It's probably never going to happen due to licensing if nothing else, lest we remember nForce chipset ordeal with intel legal.
  - paulmd 3y ago
    https://en.wikipedia.org/wiki/Project_Denver#History https://en.wikipedia.org/wiki/Project_Denver#History
  - weebull 3y ago
    Indeed, but I think people forget that the reason AMD have a license in the first place was because Intel's customers in the early days required a second source for it's processors. Who owns the Cyrix x86 license these days?
- paulmd 3y ago

bntyhntr 3y ago

I would love to be able to have a native stable diffusion experience, my rx 580 takes 30s to generate a single image. But it does work after following https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Install-and-Run-on-AMD-GPUs https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki... I got this up and running on my windows machine in short order and I don't even know what stable diffusion is. But again, it would be nice to have first class support to locally participate in the fun.

Cu3PO42 3y ago
I have heard that DirectML was a somewhat easier story, but allegedly has worse performance (and obviously it's Windows only...). But I'm not entirely suprised that setup is somewhat easier on Windows, where bundling everything is an accepted approach. With AMD's official 15GB(!) Docker image, I was now able to get the A1111 UI running. With SD 1.5 and 30 sample iterations, generating an image takes under 2s. I'm still struggling to get InvokeAI running.
- washadjeffmad 3y ago
  That has to include the model(s), no? Also, nothing is easier on Windows. It's a wonder that anything works there, except for the power of recalcitrance. Not dogging Windows users, but once your brain heals, it just can't go back.
  - Cu3PO42 3y ago
    It actually doesn't include the models! The image is Ubuntu with ROCm and a number of ML libraries, such as Torch, preinstalled. > Also, nothing is easier on Windows. As much as I, too, dislike Windows, I still have to disagree. I have encountered (proprietary) software which was much easier to get working on Windows. For example, Cisco AnyConnect with SmartCard authentication has been a nightmare for me on Linux.

westurner 3y ago

> Proton+DXVK for Linux gaming "Building the DirectX shader compiler better than Microsoft?" (2024) https://news.ycombinator.com/item?id=39324800 https://news.ycombinator.com/item?id=39324800 E.g. llama.cpp already supports hipBLAS; is there an advantage to this ROCm CUDA-compatibility layer - ZLUDA on Radeon (and not yet Intel OneAPI) - instead or in addition? https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#hipblas https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#hi... https://news.ycombinator.com/item?id=38588573 https://news.ycombinator.com/item?id=38588573 What can't WebGPU abstract away from CUDA unportability? https://news.ycombinator.com/item?id=38527552 https://news.ycombinator.com/item?id=38527552

HarHarVeryFunny 3y ago
BLAS will only get you so far. About the highest level operation it has is matmul, which you can use to build convolution (im2col, matmul, col2im), but that won't be as performant as a hand optimized cuDNN convolution kernel. Same goes for any other high level neural net building blocks - trying to build them on top of BLAS will not get you remotely close to performance of a custom kernel. What's nice about BLAS is that there are optimized implementations for CPUs (Intel MKL) as well as NVIDIA (cuBLAS) and AMD (hipBLAS), so while it's very much limited in what it can do, you can at least write portable code around it.
- westurner 3y ago
  "CUDNN API supported by HIP" has a coverage table: https://rocm.docs.amd.com/projects/HIPIFY/en/amd-staging/tables/CUDNN_API_supported_by_HIP.html https://rocm.docs.amd.com/projects/HIPIFY/en/amd-staging/tab... ROCm/hipDNN wraps CuDNN on Nvidia and MiOpen on AMD; but hasn't been updated in awhile: https://github.com/ROCm/hipDNN https://github.com/ROCm/hipDNN https://news.ycombinator.com/item?id=37808036 https://news.ycombinator.com/item?id=37808036 : conda-forge has various BLAS implementations, including MKL-optimized BLAS, and compatible NumPy and SciPy builds. BLAS: Basic Linear Algebra Sub programs: https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprogra... "Using CuPy on AMD GPU (experimental)" https://docs.cupy.dev/en/v13.0.0/install.html#using-cupy-on-amd-gpu-experimental https://docs.cupy.dev/en/v13.0.0/install.html#using-cupy-on-... : $ sudo apt install hipblas hipsparse rocsparse rocrand rocthrust rocsolver rocfft hipcub rocprim rccl

nocombination 3y ago

As other folks have commented, CUDA not being an open standard is a large part of the problem. That and the developers who target CUDA directly when writing Stable Diffusion algorithms—they are forcing the monopoly. Even at the cost of not being able to squeeze every ounce out of the GPU, portability greatly improves software access when people target Vulkan et al.

formerly_proven 3y ago

> I'm really rooting for AMD to break the CUDA monopoly. To this end, I genuinely don't know whether a translation layer is a good thing or not. On the upside it makes the hardware much more viable instantly and will boost adoption, on the downside you run the risk that devs will never support ROCm, because you can just use the translation layer. On the other hand: > The next major ROCm release (ROCm 6.0) will not be backward [source] compatible with the ROCm 5 series. Even worse, not even the driver is backwards-compatible: > There are some known limitations though like currently only targeting the ROCm 5.x API and not the newly-released ROCm 6.x releases.. In turn having to stick to ROCm 5.7 series as the latest means that using the ROCm DKMS modules don't build against the Linux 6.5 kernel now shipped by Ubuntu 22.04 LTS HWE stacks, for example. Hopefully there will be enough community support to see ZLUDA ported to ROCM 6 so at least it can be maintained with current software releases.

nialv7 3y ago

I am surprised that everybody seem to have forgotten the (in)famous Embrace, Extend and Extinguish strategy. It's time for Open Source to be on the extinguishing side for once.

sophrocyne 3y ago

Hey there - I'm a maintainer (and CEO) of Invoke. It's something we're monitoring as well. ROCm has been challenging to work with - we're actively talking to AMD to keep apprised of ways we can mitigate some of the more troublesome experiences that users have with getting Invoke running on AMD (and hoping to expand official support to Windows AMD) The problem is that a lot of the solutions proposed involve significant/unsustainable dev effort (i.e., supporting an entirely different inference paradigm), rather than "drop in" for the existing Torch/diffusers pipelines. While I don't know enough about your set up to offer immediate solutions, if you join the discord, am sure folks would be happy to try walking through some manual troubleshooting/experimentation to get you up and running - discord.gg/invoke-ai

latchkey 3y ago
Invoke is awesome. Let me know if you guys want some MI300x to develop/test on. =) We've also got some good contacts at AMD if you need help there as well.
Cu3PO42 3y ago
Hi! I really appreciate you taking the time to reply. I have since gotten Invoke to run and was already able to get some results I'm really quite happy with, so thank you for your time and commitment working on Invoke! I understand that ROCm is still challenging, but it seems my problems were less related to ROCm or Invoke itself and more to Python dependency management. It really boiled down to getting the correct (ROCm) versions of packages installed. Installing Invoke from PyPi always removed my Torch and installed CUDA-enabled Torch (as well as cuBLAS, cuDNN, ...). Once I had the correct versions of packages, everything just worked. To me, your pyproject.toml looks perfectly sane, so I wasn't sure how to go about fixing the problem. What ended up working for me was to use one of AMD's ROCm OCI base images, manually installing all dependencies, foregoing a virtual environment, cloning your repo (, building the frontend), and then installing from there. The majority of my struggle would have been solved by a recent working Docker image containing a working setup. (The one on Docker Hub is 9 months old.) Trying to build the Dockerfile from your repo, I also ended up with a CUDA-enabled Torch. It did install the correct one first, but in a later step removed the ROCm-enabled Torch to switch it for the CUDA-enabled one. I hope you'll consider investing some resources into publishing newer, working builds of your Docker image.

bavell 3y ago

Try with ComfyUI... works great and easy setup on my 6750XT. I've had it working for about a year now with SD, LlamaCpp and WhisperCpp.

mtrower 3y ago

This is the exact reason* I bought a 4090 for my recent rebuild instead of the rDNA card I actually wanted. I really wanted to go with AMD for the driver integration with the Linux graphics stack —- I’m so, so tired of shenanigans when it comes to decades old features of X not working or working poorly due to some nvidia bug/non-integration. But being able to leverage my graphics card for GPGPU was a top priority for me, and like you, I was appalled with the ROCm situation. Not necessarily the tech itself (though I did not enjoy the docker approach), but more the developer situation surrounding it. * well, that and some vague notions about RTX