3 ms·
Quantizing is a lossy process, you can't really claim to be running the 65B model llama at that point (though the 65b qgpt-llama does look like it might be very
by gpm 4y ago
Quantizing is a lossy process, you can't really claim to be running the 65B model llama at that point (though the 65b qgpt-llama does look like it might be very useful)
- Tepix 4y agoThe GPTQ paper https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323 claims "negligible accuracy degradation relative to the uncompressed baseline".