3 ms·

The impact on accuracy is somewhere in the single-digit percentages at 4-bit quantization, from what I’ve been able to gather. Very small impact. To draw the an

The impact on accuracy is somewhere in the single-digit percentages at 4-bit quantization, from what I’ve been able to gather. Very small impact. To draw the analogy out further, if the model was able to get an A on a test before quantization, it would likely still get a B at worst afterwards, given a drop in the score of less than 10%. Depending on the task, the measured impact could even be negligible. It’s far more similar to the model being perpetually tired than it is to a TBI. You may nitpick the analogy, but analogies are never exact. You also ignore the other piece that I pointed out, which is how we treat other software that comes in multiple slightly different forms.

berniedurfee 3y ago
Reminds me of the never ending MP3 vs FLAC argument. The difference can be measured empirically, but is it noticeable in real world usage.
jameshart 3y ago
But we’re talking about a coding LLM here. A single digit percentage reduction in accuracy means, what, one or two times in a hundred, it writes == instead of !=?
- coder543 3y ago
  I think that’s too simplified. The best LLMs will still frequently make mistakes. Meta is advertising a HumanEval score of 67.8%. In a third of cases, the code generated still doesn’t satisfactorily solve the problem in that automated benchmark. The additional errors that quantization would introduce would only be a very small percentage of the overall errors, making the quantized and unquantized models practically indistinguishable to a human observer. Beyond that, lower accuracy can manifest in many ways, and “do the opposite” seems unlikely to be the most common way. There might be a dozen correct ways to solve a problem. The quantized model might choose a different path that still turns out to work, it’s just not exactly the same path. As someone else pointed out, FLAC is objectively more accurate than mp3, but how many people can really tell? Is it worth 3x the data to store/stream music in FLAC? The quantized model would run at probably 4x the speed of the unquantized model, assuming you had enough memory to choose between them. Is speed worth nothing? If I have to wait all day for the LLM to respond, I can probably do the work faster myself without its help. Is being able to fit the model onto the hardware you have worth nothing? In essence, quantization here is a 95% “accurate” implementation of a 67% accurate model, which yields a 300% increase in speed while using just 25% of the RAM. All numbers are approximate, even the HumanEval benchmark should be taken with a large grain of salt. If you have a very opulent computational experience, you can enjoy the luxury of the full 67.8% accurate model, but that just feels both wasteful and like a bad user experience.

berniedurfee 3y ago

Reminds me of the never ending MP3 vs FLAC argument. The difference can be measured empirically, but is it noticeable in real world usage.

jameshart 3y ago

But we’re talking about a coding LLM here. A single digit percentage reduction in accuracy means, what, one or two times in a hundred, it writes == instead of !=?

coder543 3y ago
I think that’s too simplified. The best LLMs will still frequently make mistakes. Meta is advertising a HumanEval score of 67.8%. In a third of cases, the code generated still doesn’t satisfactorily solve the problem in that automated benchmark. The additional errors that quantization would introduce would only be a very small percentage of the overall errors, making the quantized and unquantized models practically indistinguishable to a human observer. Beyond that, lower accuracy can manifest in many ways, and “do the opposite” seems unlikely to be the most common way. There might be a dozen correct ways to solve a problem. The quantized model might choose a different path that still turns out to work, it’s just not exactly the same path. As someone else pointed out, FLAC is objectively more accurate than mp3, but how many people can really tell? Is it worth 3x the data to store/stream music in FLAC? The quantized model would run at probably 4x the speed of the unquantized model, assuming you had enough memory to choose between them. Is speed worth nothing? If I have to wait all day for the LLM to respond, I can probably do the work faster myself without its help. Is being able to fit the model onto the hardware you have worth nothing? In essence, quantization here is a 95% “accurate” implementation of a 67% accurate model, which yields a 300% increase in speed while using just 25% of the RAM. All numbers are approximate, even the HumanEval benchmark should be taken with a large grain of salt. If you have a very opulent computational experience, you can enjoy the luxury of the full 67.8% accurate model, but that just feels both wasteful and like a bad user experience.