3 ms·
My experience with heavily quantized models is they do better than a similar sized unquantized model but don't really perform anywhere near the pre-quantized mo
by tgtweak 1y ago
My experience with heavily quantized models is they do better than a similar sized unquantized model but don't really perform anywhere near the pre-quantized model.
- incomingpain 1y ago>My experience with heavily quantized models is they do better than a similar sized unquantized model but don't really perform anywhere near the pre-quantized model. People have tested it. Q8 has essentially no drop in quality, Q4 is measurable but still not realistically a problem. If this impacts you, just pay for the commercial saas option.
- haolez 1y agoThis assumes that the benchmarks are representative of real usage scenarios. I'm not saying that there is bad faith, but that benchmarking is really hard in the context of LLMs.
- incomingpain 1y ago>This assumes that the benchmarks are representative of real usage scenarios. I'm not saying that there is bad faith, but that benchmarking is really hard in the context of LLMs. It's a fair point, but the conclusion is 'i dont know' I could assume that it gets better because it'll keep to simpler code.