4 ms·

Or a Linux machine with a Ryzen using the internal GPU and the unified RAM (scroll down at llama.cpp and look for ROCm).

jacooper 3y ago
Wait ROCm support Ryzen APUs and still doesn't support dedicatedly GPUs like the 6700XT?!
- jacooper 3y ago
  Supports* dedicated*
  - trissi1996 3y ago
    While not being officially supported, rocm runs just fine on my 6700XT, i just have to set an env var(export HSA_OVERRIDE_GFX_VERSION=10.3.0)
    - jacooper 3y ago
      Really? Does everything run? Even AI stuff? Do you have any links where I can read more about that?
      - trissi1996 3y ago
        Everything I've tried to get running, worked quite smoothly. Although I only tried LLMs via llama-cpp and stable diffusion via ComfyUI. I don't see any reason why other AI stuff wouldn't work as long as it supports rocm. Also I only tried it on linux, AFAIK windows is a lot more difficult to get running, if it works at all... With llama-cpp, I successfully tried various LLMs(e.g. LLAMA 13B, Mixtral etc) with very solid performance. Even for models that don't fit in VRAM completely, performance can be surprisingly solid, as long as you compile with AVX extensions. (and your CPU supports those) Stable Diffusion via ComfyUI also works very well. However, be aware of VRAM limitations with the larger SDXL variants, especially when running a heavy desktop environment. Regarding setup guides/links, there isn't a good centralized resource sadly, so some tinkering is needed. Unlike some of those CUDA 1-click solutions, ROCm requires more manual setup, especially for the models only unofficially supported. Here are a couple of links that might be helpful: https://old.reddit.com/r/LocalLLaMA/comments/18ourt4/my_setup_for_using_rocm_with_rx_6700xt_gpu_on/ https://old.reddit.com/r/LocalLLaMA/comments/18ourt4/my_setu... https://old.reddit.com/r/StableDiffusion/comments/ww436j/howto_stable_diffusion_on_an_amd_gpu/ https://old.reddit.com/r/StableDiffusion/comments/ww436j/how... https://rentry.org/eq3hg https://rentry.org/eq3hg In general the r/localllama & r/StableDiffusion subreddits are good places to search for info.
stavros 3y ago
What? I can do this? Runs to the PC EDIT: I cannot, I need to install ROCm to compile with it, and then install something called hipBLAS, and who knows what else.
- assbuttbuttass 3y ago
  OpenCL should also work on AMD cards, and is way easier to install
  - stavros 3y ago
    Does that let me use unified memory on the GPU, though? Or is it just so I can use my CPU memory? EDIT: Oh, no, I have an nVidia GPU, AMD CPU.
    - mkesper 3y ago
      I bet your AMD CPU has an internal GPU, too. That's what you can use with the unified memory.
  - brucethemoose2 3y ago
    It is dead slow on integrated graphics, unfortunately.
- mkesper 3y ago
  Well, yes, you need to install ROCm and depdendencies. Have a look at https://rocm.docs.amd.com/projects/install-on-linux/en/latest/tutorial/quick-start.html#rocm-install-quick https://rocm.docs.amd.com/projects/install-on-linux/en/lates... Debian trixie (not yet released) has most dependencies as packages. Or you can try a docker container https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/docker.html https://rocm.docs.amd.com/projects/install-on-linux/en/lates...
  - stavros 3y ago
    I'll try that, thanks!
KeplerBoy 3y ago
Or a jetson orin agx (~2k$). Probably the cheapest way to get an Nvidia GPU with 64 GB of RAM.
- robterrell 3y ago
  Have you tried this yourself? Curious to know how well this works for an LLM home lab.
  - kkielhofner 3y ago
    I’ve worked with Jetson going back to the TK1 and I highly recommend you do not do this. Nvidia has significant dominance in the AI space because of their work on software and the overall platform. With the Jetson line being the sole exception. Use it for what it’s for - a targeted build for an embedded/specific application requiring small size and low power. The software is a mess. Support for Jetson (generally) is a far afterthought or not considered at all around projects at Nvidia and the broader ecosystem. When it is supported at all it lags behind significantly, using ancient distros (Jetpack), etc. To make matters worse the user base is so (relatively) tiny there are bugs and strange behavior everywhere. Just don’t do it.
    - dchichkov 3y ago
      This is a bit surprising to hear. Current Jetpack 6 is Ubuntu 22.04 - this is the current Ubuntu LTS release. There's nothing ancient about it, no? I'm pretty sure, if I go and check versions of CUDA, PyTorch, Tensorflow - it'd be also relatively recent. I'd suggest checking what examples are available, see what community is doing, see if what you need had already been tried - https://www.jetson-ai-lab.com https://www.jetson-ai-lab.com From what I've seen, mainstream LLM libraries like VLLM, llamacpp that use CUDA under the hood tend to work out-of-the-box. And there are tutorials available: https://www.jetson-ai-lab.com/tutorial_text-generation.html https://www.jetson-ai-lab.com/tutorial_text-generation.html. I think that TensorFlow/Pytorch are also well maintained, although I've not checked recently.
      - 3y ago
dimask 3y ago
How much RAM are you able to set aside for a ryzen igpu?
- zaat 3y ago
  I think my motherboard allow me to dedicate 12. I didn't see any improvement using CPU + ROCm compared to CPU alone. Using CPU alone I can get 4.2 - 5 Tokens/s, with ROCm I can get 4.5 - 5.2 T/s. With CPU + RTX 2070 8GB I get 6.2-7 T/s.
ode 3y ago
How fast is it with a setup like this?

jacooper 3y ago

Wait ROCm support Ryzen APUs and still doesn't support dedicatedly GPUs like the 6700XT?!

jacooper 3y ago
Supports* dedicated*
- trissi1996 3y ago
  While not being officially supported, rocm runs just fine on my 6700XT, i just have to set an env var(export HSA_OVERRIDE_GFX_VERSION=10.3.0)
  - jacooper 3y ago
    Really? Does everything run? Even AI stuff? Do you have any links where I can read more about that?
    - trissi1996 3y ago
      Everything I've tried to get running, worked quite smoothly. Although I only tried LLMs via llama-cpp and stable diffusion via ComfyUI. I don't see any reason why other AI stuff wouldn't work as long as it supports rocm. Also I only tried it on linux, AFAIK windows is a lot more difficult to get running, if it works at all... With llama-cpp, I successfully tried various LLMs(e.g. LLAMA 13B, Mixtral etc) with very solid performance. Even for models that don't fit in VRAM completely, performance can be surprisingly solid, as long as you compile with AVX extensions. (and your CPU supports those) Stable Diffusion via ComfyUI also works very well. However, be aware of VRAM limitations with the larger SDXL variants, especially when running a heavy desktop environment. Regarding setup guides/links, there isn't a good centralized resource sadly, so some tinkering is needed. Unlike some of those CUDA 1-click solutions, ROCm requires more manual setup, especially for the models only unofficially supported. Here are a couple of links that might be helpful: https://old.reddit.com/r/LocalLLaMA/comments/18ourt4/my_setup_for_using_rocm_with_rx_6700xt_gpu_on/ https://old.reddit.com/r/LocalLLaMA/comments/18ourt4/my_setu... https://old.reddit.com/r/StableDiffusion/comments/ww436j/howto_stable_diffusion_on_an_amd_gpu/ https://old.reddit.com/r/StableDiffusion/comments/ww436j/how... https://rentry.org/eq3hg https://rentry.org/eq3hg In general the r/localllama & r/StableDiffusion subreddits are good places to search for info.

stavros 3y ago

What? I can do this? Runs to the PC EDIT: I cannot, I need to install ROCm to compile with it, and then install something called hipBLAS, and who knows what else.

assbuttbuttass 3y ago
OpenCL should also work on AMD cards, and is way easier to install
- stavros 3y ago
  Does that let me use unified memory on the GPU, though? Or is it just so I can use my CPU memory? EDIT: Oh, no, I have an nVidia GPU, AMD CPU.
  - mkesper 3y ago
    I bet your AMD CPU has an internal GPU, too. That's what you can use with the unified memory.
- brucethemoose2 3y ago
  It is dead slow on integrated graphics, unfortunately.
mkesper 3y ago
Well, yes, you need to install ROCm and depdendencies. Have a look at https://rocm.docs.amd.com/projects/install-on-linux/en/latest/tutorial/quick-start.html#rocm-install-quick https://rocm.docs.amd.com/projects/install-on-linux/en/lates... Debian trixie (not yet released) has most dependencies as packages. Or you can try a docker container https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/docker.html https://rocm.docs.amd.com/projects/install-on-linux/en/lates...
- stavros 3y ago
  I'll try that, thanks!

KeplerBoy 3y ago

Or a jetson orin agx (~2k$). Probably the cheapest way to get an Nvidia GPU with 64 GB of RAM.

robterrell 3y ago
Have you tried this yourself? Curious to know how well this works for an LLM home lab.
- kkielhofner 3y ago
  I’ve worked with Jetson going back to the TK1 and I highly recommend you do not do this. Nvidia has significant dominance in the AI space because of their work on software and the overall platform. With the Jetson line being the sole exception. Use it for what it’s for - a targeted build for an embedded/specific application requiring small size and low power. The software is a mess. Support for Jetson (generally) is a far afterthought or not considered at all around projects at Nvidia and the broader ecosystem. When it is supported at all it lags behind significantly, using ancient distros (Jetpack), etc. To make matters worse the user base is so (relatively) tiny there are bugs and strange behavior everywhere. Just don’t do it.
  - dchichkov 3y ago
    This is a bit surprising to hear. Current Jetpack 6 is Ubuntu 22.04 - this is the current Ubuntu LTS release. There's nothing ancient about it, no? I'm pretty sure, if I go and check versions of CUDA, PyTorch, Tensorflow - it'd be also relatively recent. I'd suggest checking what examples are available, see what community is doing, see if what you need had already been tried - https://www.jetson-ai-lab.com https://www.jetson-ai-lab.com From what I've seen, mainstream LLM libraries like VLLM, llamacpp that use CUDA under the hood tend to work out-of-the-box. And there are tutorials available: https://www.jetson-ai-lab.com/tutorial_text-generation.html https://www.jetson-ai-lab.com/tutorial_text-generation.html. I think that TensorFlow/Pytorch are also well maintained, although I've not checked recently.
    - 3y ago

dimask 3y ago

How much RAM are you able to set aside for a ryzen igpu?

zaat 3y ago
I think my motherboard allow me to dedicate 12. I didn't see any improvement using CPU + ROCm compared to CPU alone. Using CPU alone I can get 4.2 - 5 Tokens/s, with ROCm I can get 4.5 - 5.2 T/s. With CPU + RTX 2070 8GB I get 6.2-7 T/s.

ode 3y ago

How fast is it with a setup like this?