4 ms·
On my RTX 5090, Qwen3.6-35B-A3B is my go to for coding, but Gemma-4-26b-a4b (Q4_0) is blazing fast and super high quality. I sometimes use the dense cousins of
by jszymborski 3mo ago
On my RTX 5090, Qwen3.6-35B-A3B is my go to for coding, but Gemma-4-26b-a4b (Q4_0) is blazing fast and super high quality. I sometimes use the dense cousins of these models (Qwen3.6-27B QB_0 and Gemma4-31B-QAT Q4_0) but they can be very slow. Integrating web search and code calling via Open-WebUI makes it so that I usually don't need to reach for Claude.
Here are the tok/s I get:
- Gemma-4-26B-A4B (Q4_0) = 214 tok/s
- Gemma4-31B-QAT (Q4_0) = 58 tok/s
- Qwen3.6-35B-A3B (QB_0) = 30 tok/s
- Qwen3.6-27B (QB_0) = 9 tok/s
EDIT: Updated tok/s after updating llama.cpp
- Ilaurens 3mo agoThese token numbers look off for a 5090. For comparison, an rtx 5000 Blackwell SFF with just 470gb/s bandwidth gets me 30 tok/s on gemma4-31B-QAT. Almost 60 tok/s with MTP enabled. A 5090 should get you much more than that!
- deleted 3mo ago[deleted]
- mft_ 3mo agoAgree - I got 9.7 tok/s on an M1 Max with unsloth's gemma-4-31B-it-qat-UD-Q4_K_XL.
- jszymborski 3mo agoThanks for flagging, this is on my local rig and it's driving my display too. I'm curious now, will take a closer look. These are the tok/s as reported by LMStudio. EDIT: Updating llama.ccp gets me 58 tok/s on Gemma 31b
- fhars 3mo agoThe Qwen-35B-A3B numbers are even weirder, did you drop a digit? I get half of that speed on a AMD Radeon RX 5500 XT (RADV NAVI14) (8192 MiB) (unsloth/Qwen3.6-35B-A3B-MTP-GGUF:Q6_K with q8_0 context).
- jszymborski 3mo agoNo, it's what I get repeatedly. Recall that it's QB_0 though. I'll get back to you when the Q4_K_M quant finishes downloading. EDIT: I get 259 tok/s with the Q4_K_M quant
- deleted 3mo ago[deleted]
- pixelesque 3mo agoI assume you mean Q8_0, not QB_0? (was confused, as I've not obviously seen a QB_0 before and can't find one now).
- jszymborski 3mo agobahaha, LMStudio's badge looks a lot like a B but the tooltip says it is indeed an 8 (I just assumed it was B for "byte", but that's a poor initialism considering "bit"). https://dl.jszym.com/share/boards/pictures/Screenshot_20260729.png https://dl.jszym.com/share/boards/pictures/Screenshot_202607...
- colordrops 3mo agoI see a lot of conflicting info about whether Qwen3.6-35B-A3B or Qwen3.6-27B is more capable. Is it one of those things where "it depends"?
- jszymborski 3mo agoI think sometimes the quantization is the source of the conflicting reports since those are often not included in discussions. Honestly, 27B is too slow for me to use very often, so I just do 35B-A3B and if it is underperforming I'll sometimes switch to 27B, and sometimes it does better, and other times it does not. That's when I reach for Kimi K2.6 or Claude (or just, y'know... use my noggin)
- flockonus 3mo agoTo the models you say for coding, can you give an example of what you code? I find them insufficient for my projects (mid sized), but curious what people see working.
- serf 3mo agoI use local models like qwen to do refactoring; the code and concept already exist, they need refining. I use sota models for concept, implementation plans, and initial deployment. I find that local models can perform at a very high level if given a very strict environment and the already bounded rules of an existing code base. aye aye aye.
- StopTencent 3mo ago[dead]
- NorwegianDude 3mo agoYou can get easily over 100 tok/s on the Gemma 4 31B QAT if you enable MTP. Same goes for the 27B. I'm getting 880 tok/s of throughout on a single RTX 5090 for batch tasks.
- jszymborski 3mo agoI've switched over to unsloth studio and am hitting those speeds now, thanks for the tip.
- walrus01 3mo agoUnsloth studio is just llama.cpp/llama-server under the hood, so you should see the same performance with the latest daily compiled llama-server (and the right CLI options to load a GGUF file) and any of your own choice of tooling on top of it.