3 ms·

Running Q3 on my AMD RX 9070XT. 32k context and 32/TPS. Apart from the context window preventing it from doing any large tasks, this thing is seriously powerful

by civvv 25d ago

Running Q3 on my AMD RX 9070XT. 32k context and 32/TPS. Apart from the context window preventing it from doing any large tasks, this thing is seriously powerful. I could probably push it to 64k context. Local open models are the future, and I am definitely getting a more powerful card. Very fun!

slim 25d ago
Running Q3 on 5060ti with 64k context. It runs great
Forgeties79 25d ago
What are you offloading to ram (or even CPU)? I’m using a 9080 (not XT) and having trouble with context/token rates
- civvv 25d ago
  I’m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~12.3 GiB Q3 quant, fully offloaded to the 16 GB 9070 XT. I disabled the vision projector to save VRAM and use one inference slot, Flash Attention, Q4 KV cache, --fit off, and --ctx-checkpoints 0. I’m running it with a 64K context window. The AMD driver also needs to be recent enough for ROCm 7.14; I targeted Adrenalin 26.6.4 or newer.
  - 7speter 25d ago
    You can offload the vision projector to CPU/sysRAM
    - slim 25d ago
      Awesome. How would you setup that ?
      - skirmish 25d ago
        --no-mmproj-offload See the documentation: https://github.com/ggml-org/llama.cpp/blob/master/docs/multimodal.md https://github.com/ggml-org/llama.cpp/blob/master/docs/multi...
brynx97 25d ago
Could you comment more on how you set this up? I have a mostly idle 9070XT I use for gaming, and I was considering using it with the newer local open models. Many thanks.
- civvv 25d ago
  I got Qwen3.8-27B running on my RX 9070 XT by setting up AMD’s ROCm 7.14/TheRock runtime for gfx1201 and adding the required HIP/LLVM paths to Windows. The official llama.cpp ROCm build detected the card correctly, but kept crashing with 0xC0000005 while loading the model, so I switched to the Windows Vulkan build of llama.cpp instead, which worked reliably. I’m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~12.3 GiB Q3 quant, fully offloaded to the 16 GB 9070 XT. I disabled the vision projector to save VRAM and use one inference slot, Flash Attention, Q4 KV cache, --fit off, and --ctx-checkpoints 0. I’m running it with a 64K context window. The AMD driver also needs to be recent enough for ROCm 7.14; I targeted Adrenalin 26.6.4 or newer.
zenoprax 25d ago
9070XT operator here: I'm using llama.cpp with the same model and quant and I'm getting 87,000 for my context limit. I tried the Unsloth models but they lowered it to around 30-40K so I went back to upstream. I'm on Linux and using some sort of unholy mess of ROCM libraries that I don't understand.

by civvv 25d ago

slim 25d ago

Running Q3 on 5060ti with 64k context. It runs great

Forgeties79 25d ago

What are you offloading to ram (or even CPU)? I’m using a 9080 (not XT) and having trouble with context/token rates

civvv 25d ago
I’m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~12.3 GiB Q3 quant, fully offloaded to the 16 GB 9070 XT. I disabled the vision projector to save VRAM and use one inference slot, Flash Attention, Q4 KV cache, --fit off, and --ctx-checkpoints 0. I’m running it with a 64K context window. The AMD driver also needs to be recent enough for ROCm 7.14; I targeted Adrenalin 26.6.4 or newer.
- 7speter 25d ago
  You can offload the vision projector to CPU/sysRAM
  - slim 25d ago
    Awesome. How would you setup that ?
    - skirmish 25d ago
      --no-mmproj-offload See the documentation: https://github.com/ggml-org/llama.cpp/blob/master/docs/multimodal.md https://github.com/ggml-org/llama.cpp/blob/master/docs/multi...

brynx97 25d ago

Could you comment more on how you set this up? I have a mostly idle 9070XT I use for gaming, and I was considering using it with the newer local open models. Many thanks.

civvv 25d ago
I got Qwen3.8-27B running on my RX 9070 XT by setting up AMD’s ROCm 7.14/TheRock runtime for gfx1201 and adding the required HIP/LLVM paths to Windows. The official llama.cpp ROCm build detected the card correctly, but kept crashing with 0xC0000005 while loading the model, so I switched to the Windows Vulkan build of llama.cpp instead, which worked reliably. I’m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~12.3 GiB Q3 quant, fully offloaded to the 16 GB 9070 XT. I disabled the vision projector to save VRAM and use one inference slot, Flash Attention, Q4 KV cache, --fit off, and --ctx-checkpoints 0. I’m running it with a 64K context window. The AMD driver also needs to be recent enough for ROCm 7.14; I targeted Adrenalin 26.6.4 or newer.

zenoprax 25d ago

9070XT operator here: I'm using llama.cpp with the same model and quant and I'm getting 87,000 for my context limit. I tried the Unsloth models but they lowered it to around 30-40K so I went back to upstream. I'm on Linux and using some sort of unholy mess of ROCM libraries that I don't understand.