6 ms·
How much ELO on chatbot arena does 4 bit lose vs full fp16/bf16?
by lostmsu 2y ago
How much ELO on chatbot arena does 4 bit lose vs full fp16/bf16?
- moffkalast 2y agoI think this is the only known benchmark that actually compares different quants: https://oobabooga.github.io/benchmark.html https://oobabooga.github.io/benchmark.html It's not really all that consistent, but larger models can be compressed more without as much loss.