4 ms·
Yes the higher the quant, the better! The other approach is dynamically choosing to upcast some layers!
by danielhanchen 1y ago
Yes the higher the quant, the better! The other approach is dynamically choosing to upcast some layers!
- segmondy 1y agoI can say that this really works great, I'm a heavy user of the unsloth dyanmic quants. I run DeepSeek v3/r1 in Q3, and ernie-300b and KimiK2 in Q3 too. Amazing performance. I run Qwen3-235b in both Q4 and Q8 and can barely tell the difference so much so that I just keep Q4 since it's twice as fast.
- someone13 1y agoWhat hardware do you use, out of curiosity?
- jychang 1y agoIn the current era of MoE models, the system RAM memory bandwidth determines your speed more than the GPU does.
- danielhanchen 1y agoThanks for using them! :)