3 ms·
Nice. I flip flop between Qwen 3.5 9B Q6_M and Gemma4 12B Q4_K_M on a 4080 Super. They run at about the same speed and I can have them review each other's plan
by accrual 3mo ago
Nice. I flip flop between Qwen 3.5 9B Q6_M and Gemma4 12B Q4_K_M on a 4080 Super. They run at about the same speed and I can have them review each other's plan or diffs. For smaller projects I find them very capable, and I can step up to a better quant for slightly more challenging work.
- nok22kon 3mo agoyou can probably run Gemma4 26B on your card also at 4 bit. World of a difference compared with 12B.
- zingar 3mo agoWhere does “big model highly quantized” start getting worse than “smaller model less quantized”? Is there a general formula or is it just trial and error?
- nok22kon 3mo agopaper is a bit old, but matches current empirical recommandation: a good starting point is the biggest model you can fit at 4 bit https://arxiv.org/abs/2212.09720 https://arxiv.org/abs/2212.09720
- boppo1 3mo agoHave you tried qwen 27b q4_K_XL? It's a little bigger than the 4080 but not too much