5 ms·

How is AMD GPU compatibility with leading generative AI workflows? I'm under the impression everything is CUDA.

by ashleyn 1y ago

trenchpilgrim 1y ago
Certain chips can work with useful local models, but compatibility is far behind CUDA.
pja 1y ago
llama.cpp combined with Mesa’s Vulkan support for AMD GPUs has worked pretty well with everything I’ve thrown it at.
- throwdbaaway 1y ago
  https://llm-tracker.info/_TOORG/Strix-Halo https://llm-tracker.info/_TOORG/Strix-Halo has very comprehensive test results for running llama.cpp with Strix Halo. This one is particularly interesting: > But when we switch to longer context, we see something interesting happen. WMMA + FA basically loses no performance at this longer context length! > Vulkan + FA still has better pp but tg is significantly lower. More data points would be better, but seems like Vulkan performance may continue to decrease as context extends while the HIP+rocWMMA backend should perform better. lhl has also been sharing these test results in https://forum.level1techs.com/t/strix-halo-ryzen-ai-max-395-llm-benchmark-results/ https://forum.level1techs.com/t/strix-halo-ryzen-ai-max-395-..., and his latest comment provides a great summary of the current state: > (What is bad is that basically every single model has a different optimal backend, and most of them have different optimal backends for pp (handling context) vs tg (new text)). Anyway, for me, the greatest thing about the Strix Halo + llama.cpp combo is that we can throw one or more egpu into the mix, as echoed by level1tech video (https://youtu.be/ziZDzrDI7AM?t=485 https://youtu.be/ziZDzrDI7AM?t=485), which should help a lot with PP performance.
wolfgangK 1y ago
Indeed, recent Flash Attention is a pain point for non CUDA.
DiabloD3 1y ago
CUDA isn't really used for new code. Its used for legacy codebases. In the LLM world, you really only see CUDA being used with Triton and/or PyTorch consumers that haven't moved onto better pastures (mainly because they only know Python and aren't actually programmers). That said, AMD can run most CUDA code through ROCm, and AMD officially supports Triton and PyTorch, so even the academics have a way out of Nvidia hell.
- TiredOfLife 1y ago
  ROCm doesn't work on this device
  - geerlingguy 1y ago
    You mean the AI Max chips? ROCm works fine there, as long as you're running 6.4.1 or later, no hacks required. I tested on Fedora Rawhide and it was just dnf install rocm.
  - DiabloD3 1y ago
    Yes it does. ROCm support for new chips, due to being available for paid support contracts, comes like 1-2 months after the chip comes out (ie, when they're 100% sure it works with the current, also new, driver). I'd rather it works and ships late than doesn't work and ships early and then get gaslit about the bugs (lol Nvidia, why are you like this?)
- sexeriy237 1y ago
  If you're not doing machine code by hand, you're not a programmer
  - phanimahesh 1y ago
    If you are not winding copper around magnets by hand, you are not a real programmer
    - DiabloD3 1y ago
nh43215rgb 1y ago
In practical generative AI workflows (LLMs), I think AMD Max+395 chips with unified memory is as good as Mac Studio or MacBook Pro configurations in handling big models locally and support fast inference speeds (However Top-end Apple silicon (M4 Max, Studio Ultra) can reach 546GB/s memory bandwidth, while the AMD unified memory system is around 256GB/s). I think for inference use either will work fine. For everything else I think CUDA ecosystem is a better bet (correct me if I'm wrong).
sbinnee 1y ago
My impression is the same. To train anything you just need to have CUDA gpus. For inference I think AMD and Apple M chips are getting better and better.
- jychang 1y ago
  For inference, Nvidia/AMD/Intel/Apple are all generally on the same tier now. There's a post on github of a madman who got llama.cpp generating tokens for an AI model that's running on an Intel Arc, Nvidia 3090, and AMD gpu at the same time. https://github.com/ggml-org/llama.cpp/pull/5321 https://github.com/ggml-org/llama.cpp/pull/5321
Aeolun 1y ago
All of Ollama and Stable Diffusion based stuff now works on my AMD cards. Maybe it’s different if you want to actually train things, but I have no issues running anything that fits in memory any more.
ftvkyo 1y ago
There is a project called SCALE that allows building CUDA code natively for AMD GPUs. It is designed as a drop-in replacement for Nvidia CUDA, and it is free for personal and educational use. You can find out more here: https://docs.scale-lang.com/stable/ https://docs.scale-lang.com/stable/ There are still many things that need implementing, most important ones being cuDNN and CUDA Graph API, but in my opinion, the list of things that are supported now is already quite impressive (and keeps improving): https://github.com/spectral-compute/scale-validation/tree/master?tab=readme-ov-file#current-status https://github.com/spectral-compute/scale-validation/tree/ma... Disclaimer: I am one of the developers of SCALE.
dismalaf 1y ago
> I'm under the impression everything is CUDA A very quick Google search would show that pretty much everything also runs on ROCm. Torch runs on CUDA and ROCm. Llama.cpp runs on CUDA, ROCm, SYCL, Vulkan and others...

How is AMD GPU compatibility with leading generative AI workflows? I'm under the impression everything is CUDA.

trenchpilgrim 1y ago

Certain chips can work with useful local models, but compatibility is far behind CUDA.

pja 1y ago

llama.cpp combined with Mesa’s Vulkan support for AMD GPUs has worked pretty well with everything I’ve thrown it at.

throwdbaaway 1y ago
https://llm-tracker.info/_TOORG/Strix-Halo https://llm-tracker.info/_TOORG/Strix-Halo has very comprehensive test results for running llama.cpp with Strix Halo. This one is particularly interesting: > But when we switch to longer context, we see something interesting happen. WMMA + FA basically loses no performance at this longer context length! > Vulkan + FA still has better pp but tg is significantly lower. More data points would be better, but seems like Vulkan performance may continue to decrease as context extends while the HIP+rocWMMA backend should perform better. lhl has also been sharing these test results in https://forum.level1techs.com/t/strix-halo-ryzen-ai-max-395-llm-benchmark-results/ https://forum.level1techs.com/t/strix-halo-ryzen-ai-max-395-..., and his latest comment provides a great summary of the current state: > (What is bad is that basically every single model has a different optimal backend, and most of them have different optimal backends for pp (handling context) vs tg (new text)). Anyway, for me, the greatest thing about the Strix Halo + llama.cpp combo is that we can throw one or more egpu into the mix, as echoed by level1tech video (https://youtu.be/ziZDzrDI7AM?t=485 https://youtu.be/ziZDzrDI7AM?t=485), which should help a lot with PP performance.

wolfgangK 1y ago

Indeed, recent Flash Attention is a pain point for non CUDA.

DiabloD3 1y ago

CUDA isn't really used for new code. Its used for legacy codebases. In the LLM world, you really only see CUDA being used with Triton and/or PyTorch consumers that haven't moved onto better pastures (mainly because they only know Python and aren't actually programmers). That said, AMD can run most CUDA code through ROCm, and AMD officially supports Triton and PyTorch, so even the academics have a way out of Nvidia hell.

TiredOfLife 1y ago
ROCm doesn't work on this device
- geerlingguy 1y ago
  You mean the AI Max chips? ROCm works fine there, as long as you're running 6.4.1 or later, no hacks required. I tested on Fedora Rawhide and it was just dnf install rocm.
- DiabloD3 1y ago
  Yes it does. ROCm support for new chips, due to being available for paid support contracts, comes like 1-2 months after the chip comes out (ie, when they're 100% sure it works with the current, also new, driver). I'd rather it works and ships late than doesn't work and ships early and then get gaslit about the bugs (lol Nvidia, why are you like this?)
sexeriy237 1y ago
If you're not doing machine code by hand, you're not a programmer
- phanimahesh 1y ago
  If you are not winding copper around magnets by hand, you are not a real programmer
  - DiabloD3 1y ago

nh43215rgb 1y ago

In practical generative AI workflows (LLMs), I think AMD Max+395 chips with unified memory is as good as Mac Studio or MacBook Pro configurations in handling big models locally and support fast inference speeds (However Top-end Apple silicon (M4 Max, Studio Ultra) can reach 546GB/s memory bandwidth, while the AMD unified memory system is around 256GB/s). I think for inference use either will work fine. For everything else I think CUDA ecosystem is a better bet (correct me if I'm wrong).

sbinnee 1y ago

My impression is the same. To train anything you just need to have CUDA gpus. For inference I think AMD and Apple M chips are getting better and better.

jychang 1y ago
For inference, Nvidia/AMD/Intel/Apple are all generally on the same tier now. There's a post on github of a madman who got llama.cpp generating tokens for an AI model that's running on an Intel Arc, Nvidia 3090, and AMD gpu at the same time. https://github.com/ggml-org/llama.cpp/pull/5321 https://github.com/ggml-org/llama.cpp/pull/5321

Aeolun 1y ago

All of Ollama and Stable Diffusion based stuff now works on my AMD cards. Maybe it’s different if you want to actually train things, but I have no issues running anything that fits in memory any more.

ftvkyo 1y ago

There is a project called SCALE that allows building CUDA code natively for AMD GPUs. It is designed as a drop-in replacement for Nvidia CUDA, and it is free for personal and educational use. You can find out more here: https://docs.scale-lang.com/stable/ https://docs.scale-lang.com/stable/ There are still many things that need implementing, most important ones being cuDNN and CUDA Graph API, but in my opinion, the list of things that are supported now is already quite impressive (and keeps improving): https://github.com/spectral-compute/scale-validation/tree/master?tab=readme-ov-file#current-status https://github.com/spectral-compute/scale-validation/tree/ma... Disclaimer: I am one of the developers of SCALE.

dismalaf 1y ago

> I'm under the impression everything is CUDA A very quick Google search would show that pretty much everything also runs on ROCm. Torch runs on CUDA and ROCm. Llama.cpp runs on CUDA, ROCm, SYCL, Vulkan and others...