4 ms·
For Qwen3.5-27b I'm getting in the 20 to 25 tok/sec range on a 128GB Strix Halo box (Framework Desktop). That's with the 8-bit quant. It's definitely usable, b
by UncleOxidant 6mo ago
For Qwen3.5-27b I'm getting in the 20 to 25 tok/sec range on a 128GB Strix Halo box (Framework Desktop). That's with the 8-bit quant. It's definitely usable, but sometimes you're waiting a bit, though I'm not finding it problematic for the most part. I can run the Qwen3-coder-next (80b MoE) at 36tok/sec - hoping they release a Qwen3.6-coder soon.
- bityard 6mo agoI have a Framework Desktop too and 20-25 t/s is a lot better than I was expecting for such a large dense model. I'll have to try it out tonight. Are you using llama.cpp?
- UncleOxidant 6mo agoLMStudio, but it uses llama.cpp to run inference, so yeah. This is with the vulkan backend, not ROCm.
- petu 6mo ago> Qwen3.5-27b 8-bit quant 20 to 25 tok/sec It that with some kind of speculative decoding? Or total throughput for parallel requests?
- lambda 6mo agoThat sounds high for a Strix Halo with a dense 27b model. Are you talking about decode (prompt eval, which can happend in parallel) or generation when you quote tokens per second? Usually if people quote only one number they're quoting generation speed, and I would be surprised if you got that for generation speed on a Strix Halo.