3 ms·

> Been running local LLMs on my 7900 XTX for months and the ROCm experience has been... rough. Just out of curiosity... how so? I only ask because I've been r

> Been running local LLMs on my 7900 XTX for months and the ROCm experience has been... rough. Just out of curiosity... how so? I only ask because I've been running local models (using Ollama) on my RX 7900 XTX for the last year and a half or so and haven't had a single problem that was ROCm specific that I can think of. Actually, I've barely had any problems at all, other than the card being limited to 24GB of VRAM. :-( I'm halfway tempted to splurge on a Radeon Pro board to get more VRAM, but ... haven't bitten the bullet yet.

skirmish 7mo ago
Did you have complete hardware lockups when VRAM is exceeded? I had quite a few on my 7900XTX with llama.cpp (Arch Linux, various driver versions). Once I dial in the quant and context size that never exceed VRAM, it is stable; before that I swear a lot and keep pressing the hardware reset button.
- mindcrime 7mo ago
  Nope. I've exceeded available VRAM a few times, and never had to do anything other than maybe restart Ollama. To be fair though, that's "exceed available VRAM" in terms of the initial model load (eg, using a model that would never load in 24GB). I don't know that I've ever started working with a successfully loaded model and then pushed past available VRAM by pushing stuff into the context. I've had a few of those "model psychosis" incidents where the context gets so big that the model just loses all coherence and starts spewing gibberish though. Those are always fun.
- ozgrakkurt 7mo ago
  Yes, it completely crashes the machine. I didn't even think it was unexpected until I read your comment. I guess this is what I come to expect when using anything except firefox or neovim
- hypercube33 7mo ago
  This happens on windows as well for the same reasons so it's not isolated to Rocm and Linux
naasking 7mo ago
> I only ask because I've been running local models (using Ollama) on my RX 7900 XTX for the last year and a half or so and haven't had a single problem that was ROCm specific that I can think of. It's probably using the Vulkan backend, that is pretty stable and performance is good.

skirmish 7mo ago

Did you have complete hardware lockups when VRAM is exceeded? I had quite a few on my 7900XTX with llama.cpp (Arch Linux, various driver versions). Once I dial in the quant and context size that never exceed VRAM, it is stable; before that I swear a lot and keep pressing the hardware reset button.

mindcrime 7mo ago
Nope. I've exceeded available VRAM a few times, and never had to do anything other than maybe restart Ollama. To be fair though, that's "exceed available VRAM" in terms of the initial model load (eg, using a model that would never load in 24GB). I don't know that I've ever started working with a successfully loaded model and then pushed past available VRAM by pushing stuff into the context. I've had a few of those "model psychosis" incidents where the context gets so big that the model just loses all coherence and starts spewing gibberish though. Those are always fun.
ozgrakkurt 7mo ago
Yes, it completely crashes the machine. I didn't even think it was unexpected until I read your comment. I guess this is what I come to expect when using anything except firefox or neovim
hypercube33 7mo ago
This happens on windows as well for the same reasons so it's not isolated to Rocm and Linux

naasking 7mo ago

> I only ask because I've been running local models (using Ollama) on my RX 7900 XTX for the last year and a half or so and haven't had a single problem that was ROCm specific that I can think of. It's probably using the Vulkan backend, that is pretty stable and performance is good.