3 ms·
You can get easily over 100 tok/s on the Gemma 4 31B QAT if you enable MTP. Same goes for the 27B. I'm getting 880 tok/s of throughout on a single RTX 5090 for
by NorwegianDude 2mo ago
You can get easily over 100 tok/s on the Gemma 4 31B QAT if you enable MTP. Same goes for the 27B. I'm getting 880 tok/s of throughout on a single RTX 5090 for batch tasks.
- jszymborski 2mo agoI've switched over to unsloth studio and am hitting those speeds now, thanks for the tip.
- walrus01 2mo agoUnsloth studio is just llama.cpp/llama-server under the hood, so you should see the same performance with the latest daily compiled llama-server (and the right CLI options to load a GGUF file) and any of your own choice of tooling on top of it.