3 ms·
SpQR: Near-Lossless LLM Weight Compression
- homarp 3y agoSpQR allows lossless LLM inference at 4.75 bits with a 15% speedup. You can run a 33B LLM on a single 24GB GPU fully lossless. SpQR works by isolating sensitive weights with higher precision and roughly doubles improvements from GPTQ