3 ms·
I think the 8B+ question was about parameter count (8 billion+ parameters), not quantization level (8 bits per weight).
by dragonwriter 1y ago
I think the 8B+ question was about parameter count (8 billion+ parameters), not quantization level (8 bits per weight).
- vunderba 1y agoYeah I should have been more specific - Qwen 8b at a 5_K_M quant worked very well.