5 ms·

I cannot get AMD ROCm running on my debian 12 system which is what I think is causing Ollama to use CPU instead of GPU. So I guess there is still a long way to

I cannot get AMD ROCm running on my debian 12 system which is what I think is causing Ollama to use CPU instead of GPU. So I guess there is still a long way to go.

ants_everywhere 2y ago
I've had more luck with the ROCm docker container. I run it via k8s. It was pretty painless to set up and has been mostly painless since. Prior to that it was nearly impossible to get Jax running reliably on ROCm. Even with the container, you have to be careful installing Python libraries because they can still break things.
- lenova 2y ago
  I just recently went down the AMD GPU + ROCm rabbit hole as well. ROCm 6.2 was just released in August of this year and introduces a lot better support, though as the above poster mentioned, isn't merged into most recent OSes. This Github repo is good for tracking the latest Ubuntu + ROCm install process: https://github.com/nktice/AMD-AI https://github.com/nktice/AMD-AI
  - latchkey 2y ago
    That's a nice repo of random installation notes. Very helpful, thanks!
superkuh 2y ago
You'd probably have a lot better luck using Vulkan acceleration (not ROCm) of llama.cpp as backend to ollama. It is incomparibly easier to set up and maintain compared to ROCm. You can actually do it on your computer's normal OS instead of inside a bunch of container/vms where the system libs are entirely customized to running just that one application. AMD's support of consumer cards is very, very short. By the time it's stable enough for a new card to run the card is no longer supported. In 2021 I bought an AMD GPU that came out 3 years before and 1 year after I bought it (4 years since release) they dropped ROCm support.
- 6y56h56 2y ago
  Thank you, I did not know there is an alternate route to achieve GPU usage. Will look into this.
- coppsilgold 2y ago
  ROCm is not even worth the effort for inference workloads. Vulkan is much more convenient and performs fine. llama.cpp and stable-diffusion.cpp offer Vulkan backends but generally you can run most models on Vulkan if you use IREE[1]. [1] <https://iree.dev/guides/ml-frameworks/ https://iree.dev/guides/ml-frameworks/>
  - lhl 2y ago
    While Vulkan can be a good fallback, for LLM inference at least, the performance difference is not as insignificant as you believe. I just ran a test on the latest pull just to make sure this is still the case on llama.cpp HEAD, but text generation is +44% faster and prompt processing is +202% (~3X) faster with ROCm vs Vulkan. Note: if you're building llama.cpp, all you have to do is swap GGML_HIPBLAS=1 and GGML_VULKAN=1 so the extra effort is just installing ROCm? (vs the Vulkan devtools) ROCm: CUDA_VISIBLE_DEVICES=1 ./llama-bench -m /models/gguf/llama-2-7b.Q4_0.gguf ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 ROCm devices: Device 0: Radeon RX 7900 XTX, compute capability 11.0, VMM: no | model | size | params | backend | ngl | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: | | llama 7B Q4_0 | 3.56 GiB | 6.74 B | ROCm | 99 | pp512 | 3258.67 ± 29.23 | | llama 7B Q4_0 | 3.56 GiB | 6.74 B | ROCm | 99 | tg128 | 103.31 ± 0.03 | build: 31ac5834 (3818) Vulkan: GGML_VK_VISIBLE_DEVICES=1 ./llama-bench -m /models/gguf/llama-2-7b.Q4_0.gguf | model | size | params | backend | ngl | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: | ggml_vulkan: Found 1 Vulkan devices: Vulkan0: Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | llama 7B Q4_0 | 3.56 GiB | 6.74 B | Vulkan | 99 | pp512 | 1077.49 ± 2.00 | | llama 7B Q4_0 | 3.56 GiB | 6.74 B | Vulkan | 99 | tg128 | 71.83 ± 0.06 | build: 31ac5834 (3818) EDIT: HN should really support markdown...
jchw 2y ago
At the risk of pissing people off, I think you may be better served by a distribution that provides a more up-to-date kernel. Debian 12 will give you Linux 6.1 LTS, which is probably OK if you're using an older Radeon card, but I've heard support for the 7900 XT/X series is a bit dicey and beyond that (e.g. Radeon 890M) non-existent. If there were improvements on the AMDGPU DRM driver side, you would not see them in Debian any time soon, as the 6.1 LTS kernel will be stuck with roughly whatever shipped January of last year. This is just a shortcoming in the Linux kernel, due to its lack of any kind of stable ABI for drivers. Of course it is possible this would help nothing or even hurt. My experience running stable (or even newer) kernels has been quite good, though. I run stable or newer across a few devices and run into hiccups not more than once every few years, which is definitely worth it to be able to get new driver improvements years in advance. (FWIW Debian is not even supported by ROCm[1]... although distros with even older kernels are. But, even if ROCm works, I can't imagine you will get ideal hardware support when running older kernels. I am not sure if ROCm has some workaround for enterprise Linux distributions specifically, but it feels like they must, given how many of their customers in the datacenter are likely to want to use them.) [1]: https://rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html https://rocm.docs.amd.com/en/latest/compatibility/compatibil...
- slavik81 2y ago
  > I've heard support for the 7900 XT/X series is a bit dicey The firmware-amd-graphics package in stable is too old to properly support RDNA 3. It kind of works, but it is quite buggy. All RDNA 3 users on Debian 12 should be sure to install the kernel and firmware from bookworm-backports. There is full support for RDNA 3 hardware enabled on Debian Testing (both in the drivers and runtime libraries). The Debian ROCm Team intended to backport all the ROCm packages from Testing into Bookworm, but have been held up as LLVM 17 is not available in bookworm-backports (yet?). > FWIW Debian is not even supported by ROCm ROCm does not support Debian, but Debian supports ROCm. Most of the libraries that comprise ROCm have been directly packaged by the distribution.
llm_trw 2y ago
Like everything in machine learning it only really runs on Ubuntu 22.04. Anything else is unsupported and you need to spend weeks tinkering to get it to work, then never upgrade.
JonChesterfield 2y ago
ROCm versions are fairly closely tied to kernel versions. Debian 12 should run fine with the ROCm that was released around the time of whatever kernel you're running, but it's going to be a bad experience mixing the latest ROCm with an elderly kernel or vice versa. Old kernel + old kernel driver + new rocm => the driver doesn't really know what the userspace is doing and you get the bugs which have been fixed since Old kernel + new kernel driver => very ymmv, the internel kernel api is not stable New kernel + matching driver, old rocm is probably OK, unless you're using upstream clang in which case it's all bad once more ROCm was designed and implemented in the HPC environment, where you know the exact kernel in use and the whole stack is deployed as one self consistent lump. Driver, compiler, libraries and so forth. It's not having such a good time in the Linux world of mix and match because the aggressive internal testing structure assumes you're using a consistent system. Backwards/forwards ABI and API compatibility is difficult, expensive and slow so it's not where the money is being spent. Rightly so, probably.

I cannot get AMD ROCm running on my debian 12 system which is what I think is causing Ollama to use CPU instead of GPU. So I guess there is still a long way to go.

ants_everywhere 2y ago

I've had more luck with the ROCm docker container. I run it via k8s. It was pretty painless to set up and has been mostly painless since. Prior to that it was nearly impossible to get Jax running reliably on ROCm. Even with the container, you have to be careful installing Python libraries because they can still break things.

lenova 2y ago
I just recently went down the AMD GPU + ROCm rabbit hole as well. ROCm 6.2 was just released in August of this year and introduces a lot better support, though as the above poster mentioned, isn't merged into most recent OSes. This Github repo is good for tracking the latest Ubuntu + ROCm install process: https://github.com/nktice/AMD-AI https://github.com/nktice/AMD-AI
- latchkey 2y ago
  That's a nice repo of random installation notes. Very helpful, thanks!

superkuh 2y ago

You'd probably have a lot better luck using Vulkan acceleration (not ROCm) of llama.cpp as backend to ollama. It is incomparibly easier to set up and maintain compared to ROCm. You can actually do it on your computer's normal OS instead of inside a bunch of container/vms where the system libs are entirely customized to running just that one application. AMD's support of consumer cards is very, very short. By the time it's stable enough for a new card to run the card is no longer supported. In 2021 I bought an AMD GPU that came out 3 years before and 1 year after I bought it (4 years since release) they dropped ROCm support.

6y56h56 2y ago
Thank you, I did not know there is an alternate route to achieve GPU usage. Will look into this.
coppsilgold 2y ago
ROCm is not even worth the effort for inference workloads. Vulkan is much more convenient and performs fine. llama.cpp and stable-diffusion.cpp offer Vulkan backends but generally you can run most models on Vulkan if you use IREE[1]. [1] <https://iree.dev/guides/ml-frameworks/ https://iree.dev/guides/ml-frameworks/>
- lhl 2y ago
  While Vulkan can be a good fallback, for LLM inference at least, the performance difference is not as insignificant as you believe. I just ran a test on the latest pull just to make sure this is still the case on llama.cpp HEAD, but text generation is +44% faster and prompt processing is +202% (~3X) faster with ROCm vs Vulkan. Note: if you're building llama.cpp, all you have to do is swap GGML_HIPBLAS=1 and GGML_VULKAN=1 so the extra effort is just installing ROCm? (vs the Vulkan devtools) ROCm: CUDA_VISIBLE_DEVICES=1 ./llama-bench -m /models/gguf/llama-2-7b.Q4_0.gguf ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 ROCm devices: Device 0: Radeon RX 7900 XTX, compute capability 11.0, VMM: no | model | size | params | backend | ngl | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: | | llama 7B Q4_0 | 3.56 GiB | 6.74 B | ROCm | 99 | pp512 | 3258.67 ± 29.23 | | llama 7B Q4_0 | 3.56 GiB | 6.74 B | ROCm | 99 | tg128 | 103.31 ± 0.03 | build: 31ac5834 (3818) Vulkan: GGML_VK_VISIBLE_DEVICES=1 ./llama-bench -m /models/gguf/llama-2-7b.Q4_0.gguf | model | size | params | backend | ngl | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: | ggml_vulkan: Found 1 Vulkan devices: Vulkan0: Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | llama 7B Q4_0 | 3.56 GiB | 6.74 B | Vulkan | 99 | pp512 | 1077.49 ± 2.00 | | llama 7B Q4_0 | 3.56 GiB | 6.74 B | Vulkan | 99 | tg128 | 71.83 ± 0.06 | build: 31ac5834 (3818) EDIT: HN should really support markdown...

jchw 2y ago

At the risk of pissing people off, I think you may be better served by a distribution that provides a more up-to-date kernel. Debian 12 will give you Linux 6.1 LTS, which is probably OK if you're using an older Radeon card, but I've heard support for the 7900 XT/X series is a bit dicey and beyond that (e.g. Radeon 890M) non-existent. If there were improvements on the AMDGPU DRM driver side, you would not see them in Debian any time soon, as the 6.1 LTS kernel will be stuck with roughly whatever shipped January of last year. This is just a shortcoming in the Linux kernel, due to its lack of any kind of stable ABI for drivers. Of course it is possible this would help nothing or even hurt. My experience running stable (or even newer) kernels has been quite good, though. I run stable or newer across a few devices and run into hiccups not more than once every few years, which is definitely worth it to be able to get new driver improvements years in advance. (FWIW Debian is not even supported by ROCm[1]... although distros with even older kernels are. But, even if ROCm works, I can't imagine you will get ideal hardware support when running older kernels. I am not sure if ROCm has some workaround for enterprise Linux distributions specifically, but it feels like they must, given how many of their customers in the datacenter are likely to want to use them.) [1]: https://rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html https://rocm.docs.amd.com/en/latest/compatibility/compatibil...

slavik81 2y ago
> I've heard support for the 7900 XT/X series is a bit dicey The firmware-amd-graphics package in stable is too old to properly support RDNA 3. It kind of works, but it is quite buggy. All RDNA 3 users on Debian 12 should be sure to install the kernel and firmware from bookworm-backports. There is full support for RDNA 3 hardware enabled on Debian Testing (both in the drivers and runtime libraries). The Debian ROCm Team intended to backport all the ROCm packages from Testing into Bookworm, but have been held up as LLVM 17 is not available in bookworm-backports (yet?). > FWIW Debian is not even supported by ROCm ROCm does not support Debian, but Debian supports ROCm. Most of the libraries that comprise ROCm have been directly packaged by the distribution.

llm_trw 2y ago

Like everything in machine learning it only really runs on Ubuntu 22.04. Anything else is unsupported and you need to spend weeks tinkering to get it to work, then never upgrade.

JonChesterfield 2y ago

ROCm versions are fairly closely tied to kernel versions. Debian 12 should run fine with the ROCm that was released around the time of whatever kernel you're running, but it's going to be a bad experience mixing the latest ROCm with an elderly kernel or vice versa. Old kernel + old kernel driver + new rocm => the driver doesn't really know what the userspace is doing and you get the bugs which have been fixed since Old kernel + new kernel driver => very ymmv, the internel kernel api is not stable New kernel + matching driver, old rocm is probably OK, unless you're using upstream clang in which case it's all bad once more ROCm was designed and implemented in the HPC environment, where you know the exact kernel in use and the whole stack is deployed as one self consistent lump. Driver, compiler, libraries and so forth. It's not having such a good time in the Linux world of mix and match because the aggressive internal testing structure assumes you're using a consistent system. Backwards/forwards ABI and API compatibility is difficult, expensive and slow so it's not where the money is being spent. Rightly so, probably.