3 ms·
There's a real hole here at Q3. A critical breakpoint here is sub 16-GB cards, which covers the 5080, 5070 Ti, 5060ti, and several other cards from this generat
by purpleflame1257 18d ago
There's a real hole here at Q3. A critical breakpoint here is sub 16-GB cards, which covers the 5080, 5070 Ti, 5060ti, and several other cards from this generation and the last. It would be instructive to see where the quality knee is.
- jadbox 18d agoQ3 XL and Q3 XS are the two I'm trying to decide on
- dofm 18d agoYou might want to test this new dynamic GGUF: https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF (I don’t know much about it, just saw a YouTube video about it last night)
- kennywinker 18d agoAnother one to try: https://huggingface.co/Jackrong/Qwopus3.8-27B-Flash-GGUF https://huggingface.co/Jackrong/Qwopus3.8-27B-Flash-GGUF Runs the 3bit model faster than the 2bit one runs on my old-ass card. Can’t vouch for its intelligence yet, but i suspect whatever loss in smarts it takes is made up for by the extra resolution.
- civvv 18d agoRunning Q3 on my AMD RX 9070XT. 32k context and 32/TPS. Apart from the context window preventing it from doing any large tasks, this thing is seriously powerful. I could probably push it to 64k context. Local open models are the future, and I am definitely getting a more powerful card. Very fun!
- slim 18d agoRunning Q3 on 5060ti with 64k context. It runs great
- Forgeties79 18d agoWhat are you offloading to ram (or even CPU)? I’m using a 9080 (not XT) and having trouble with context/token rates
- civvv 18d agoI’m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~12.3 GiB Q3 quant, fully offloaded to the 16 GB 9070 XT. I disabled the vision projector to save VRAM and use one inference slot, Flash Attention, Q4 KV cache, --fit off, and --ctx-checkpoints 0. I’m running it with a 64K context window. The AMD driver also needs to be recent enough for ROCm 7.14; I targeted Adrenalin 26.6.4 or newer.
- 7speter 18d agoYou can offload the vision projector to CPU/sysRAM
- brynx97 18d agoCould you comment more on how you set this up? I have a mostly idle 9070XT I use for gaming, and I was considering using it with the newer local open models. Many thanks.
- civvv 18d agoI got Qwen3.8-27B running on my RX 9070 XT by setting up AMD’s ROCm 7.14/TheRock runtime for gfx1201 and adding the required HIP/LLVM paths to Windows. The official llama.cpp ROCm build detected the card correctly, but kept crashing with 0xC0000005 while loading the model, so I switched to the Windows Vulkan build of llama.cpp instead, which worked reliably. I’m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~12.3 GiB Q3 quant, fully offloaded to the 16 GB 9070 XT. I disabled the vision projector to save VRAM and use one inference slot, Flash Attention, Q4 KV cache, --fit off, and --ctx-checkpoints 0. I’m running it with a 64K context window. The AMD driver also needs to be recent enough for ROCm 7.14; I targeted Adrenalin 26.6.4 or newer.
- zenoprax 17d ago9070XT operator here: I'm using llama.cpp with the same model and quant and I'm getting 87,000 for my context limit. I tried the Unsloth models but they lowered it to around 30-40K so I went back to upstream. I'm on Linux and using some sort of unholy mess of ROCM libraries that I don't understand.
- dofm 18d agoThere is an interesting new dynamic 3 bit quantisation I have been meaning to test: https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF Luke of Luke’s Dev Lab on YouTube had a look at it. It seems to outperform the typical 3-bit quantisation but whether it outperforms the new Unsloth dynamic I don’t know.
- selectodude 18d agoI have a 5080, three OpenAI Pro token resets, and I’m on paternity leave. Astra seems pretty clever. Maybe I’ll give it a task.