Quantization is not free, but VRAM is even less free. If you have X amount of VRAM and can fit a 16bit model of size 2X in 8bit or a model of size 4X in 4bit t

Quantization is not free, but VRAM is even less free. If you have X amount of VRAM and can fit a 16bit model of size 2X in 8bit or a model of size 4X in 4bit then the 4X model in 4bit is ALWAYS superior with lower perplexity and better performance. You LOSE performance by using a smaller model in 8bit vs a larger model in 4bit.