3 ms·
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
- homarp 3y agocode here: https://github.com/Vahe1994/SpQR https://github.com/Vahe1994/SpQR
- convexstrictly 3y ago"We present SpQR, which allows lossless LLM inference at 4.75 bits with a 15% speedup. You can run a 33B LLM on a single 24GB GPU fully lossless. SpQR works by isolating sensitive weights with higher precision and roughly doubles improvements from GPTQ" -- Tim Dettmers https://twitter.com/Tim_Dettmers/status/1666076553665744896?s=20 https://twitter.com/Tim_Dettmers/status/1666076553665744896?...