3 ms·
Activation-Aware Weight Quantization for LLM Compression Outperforms GPTQ
- valine 3y agoBetter quantization will have a direct and meaningful impact for everyone running local LLMs. The technique has already been applied to both Vicuna and the multimodal LLaMA variant LLaVA. https://github.com/mit-han-lab/llm-awq https://github.com/mit-han-lab/llm-awq
- convexstrictly 3y agoThey claim average 1.45x speedup and maximum 1.7x speedup over GPTQ. High-level idea: About 1% of the weights contribute greatly to quantization error. So skip the quantization of these weights.