3 ms·
How much does 3 bit quantization affect performance?
by _c8fm 3y ago
How much does 3 bit quantization affect performance?
- YetAnotherNick 3y agoAccording to few papers and https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684, ~2.5GB 7B parameters model size has the same performance as baseline 7B model(with 14GB size).