3 ms·
No it isn't, quantization is not free. You lose a significant amount of performance that you are not measuring properly in automated benchmarks when you quantiz
by Der_Einzige 3y ago
No it isn't, quantization is not free. You lose a significant amount of performance that you are not measuring properly in automated benchmarks when you quantize to that level.
You can see it in real time when you take most LLMs and compare them at different quantization levels. I can see the degradation even in the largest llama quite badly even at 8 bits.
- astrange 3y agoIf you take a model and quantize it it's obviously going to get worse, but what if you train it again after that?
- MacsHeadroom 3y agoQuantization is not free, but VRAM is even less free. If you have X amount of VRAM and can fit a 16bit model of size 2X in 8bit or a model of size 4X in 4bit then the 4X model in 4bit is ALWAYS superior with lower perplexity and better performance. You LOSE performance by using a smaller model in 8bit vs a larger model in 4bit.