3 ms·
Fantastic article - I would love to see an analysis comparing larger quantized models to smaller unquantized models. e.g. is a 14b quantized model better than a
by SleekEagle 3y ago
Fantastic article - I would love to see an analysis comparing larger quantized models to smaller unquantized models. e.g. is a 14b quantized model better than a 7b unquantized model?
- lhl 3y agoBasically, the most aggressive quants for a larger sized model is always better than the unquantized model of a smaller model: https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684 Tim Dettmers did a bunch of research on this last year as well: https://arxiv.org/abs/2212.09720 https://arxiv.org/abs/2212.09720