4 ms·
My understanding is that models are currently undertrained and not very "dense", so Q4 doesn't hurt very much now but it may in future denser models.
by nenaoki 2y ago
My understanding is that models are currently undertrained and not very "dense", so Q4 doesn't hurt very much now but it may in future denser models.
- anon373839 2y agoThat may well be true. I know that earlier models like Llama 1 65B could tolerate more aggressive quantization, which supports that idea.