3 ms·
The “original” and by far most common format for quantization is GPTQ. AWQ support is spreading more, which is nice.
by kkielhofner 3y ago
The “original” and by far most common format for quantization is GPTQ.
AWQ support is spreading more, which is nice.
- refulgentis 3y agoAgain, for a subset of the local LLM community. Quantization was not invented on Github, by llama.cpp, for LLMs in 2023.
- kkielhofner 3y agoIf a tree falls in a forrest and no one is around, does it make a sound? Of course quantization was invented well before LLMs. However, LLMs have dramatically accelerated development on quantization and resulted in an explosion in use.