It makes the model dumber. That seems simplistic, but its really simple as that. Naive 3 bit quantization will turn llama 7B into blubbering nonsense. But lla

It makes the model dumber. That seems simplistic, but its really simple as that. Naive 3 bit quantization will turn llama 7B into blubbering nonsense. But llama.cpp quantization is good! I recommend checking out the graphs ikawrakow made for their K-quants implementation: https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684 Basically, the more you quantize with K-quant, the dumber the model gets. 2 bit llama 13B quant, for instance, is about as dumb as 7B F16, but the dropoff is not nearly as severe from 3-6 bits.