3 ms·
The effect is lesser than you think. 5 bit quantization has negligible performance loss compared to 16 bits: https://github.com/ggerganov/llama.cpp/pull/1684 ht
by YetAnotherNick 3y ago
The effect is lesser than you think. 5 bit quantization has negligible performance loss compared to 16 bits: https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684
- astrange 3y agoThis paper from last month has a method for acceptable 3-bit quantization and a start at 2-bit. https://arxiv.org/abs/2307.13304 https://arxiv.org/abs/2307.13304