3 ms·
They mention 5090 with regards to speed, Q6 will not have that speed? And speed matters a lot for many use cases
by sisve 15d ago
They mention 5090 with regards to speed, Q6 will not have that speed?
And speed matters a lot for many use cases
- selectodude 15d ago150 tokens per second on a ternary model implies that it’s GPU bound, I’d bet a Q6 model is even faster because it’s existed longer and seen more optimization. You’d have to be insane to not run an NVFP4 quant over a ternary quant on Blackwell if they both fit.
- wincy 14d agoWith Ninfer and Qwen 3.8 27b it uses a groupwise int mixed quant, and it gets 160 tokens/sec. The mixed quant is between 4 and 6 bits.
- Foobar8568 14d agoNinfer is compatible with a nvfp4 model for the 27b. Also nowadays I prefer to use the byteshape one, I get less loops, and I am not sure if I really see a difference in speed or quality. Pure vibe agentic coding on a C++ codebase or ocaml one, ocaml one has codex as reviewer as I am more interested by that project, the other is more for fun.