4 ms·
I find it hard to trust post training quantizations. Why don't they run benchmarks to see the degradation in performance? It sketches me out because it should b
by WanderPanda 9mo ago
I find it hard to trust post training quantizations. Why don't they run benchmarks to see the degradation in performance? It sketches me out because it should be the easiest thing to automatically run a suite of benchmarks
- Miraste 9mo agoUnsloth doesn't seem to do this for every new model, but they did publish a report on their quant methods and the performance loss it causes. https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs It isn't much until you get down to very small quants.