3 ms·
Are there comparisons between int4 QAT versions of these models and the more common GGUF Q4_K_M quantizations generated post-training? The QAT models appear to
by philipkglass 2y ago
Are there comparisons between int4 QAT versions of these models and the more common GGUF Q4_K_M quantizations generated post-training? The QAT models appear to be slightly larger:
https://ollama.com/library/gemma3/tags https://ollama.com/library/gemma3/tags
I presume QAT are better but I don't see how much better.
- mdp2021 2y ago> I presume QAT are better but I don't see how much better Not the data for Google's Gemma, but some numbers are here: https://aclanthology.org/2024.findings-acl.26/ https://aclanthology.org/2024.findings-acl.26/ ( https://aclanthology.org/2024.findings-acl.26.pdf https://aclanthology.org/2024.findings-acl.26.pdf )