2 ms·
I would say 9 times out of 10, you will get better results from a Q4 model that’s a size class larger than a smaller model at Q8. But it’s best not to go below
by anon373839 2y ago
I would say 9 times out of 10, you will get better results from a Q4 model that’s a size class larger than a smaller model at Q8. But it’s best not to go below Q4.
- nenaoki 2y agoMy understanding is that models are currently undertrained and not very "dense", so Q4 doesn't hurt very much now but it may in future denser models.
- anon373839 2y agoThat may well be true. I know that earlier models like Llama 1 65B could tolerate more aggressive quantization, which supports that idea.