4 ms·
> a 7B parameter model using float32s will almost certainly outperform a 7B model using float4s Q5 quantization performs almost on par with base models. Obviou
by dvt 3y ago
> a 7B parameter model using float32s will almost certainly outperform a 7B model using float4s
Q5 quantization performs almost on par with base models. Obviously there's some loss there, but this indicates that there's still a lot of compression that we're missing.
- jonnycomputer 3y agoI'm still amazed that quantization works at all, coming out as a mild degradation in quality rather than radical dysfunction. Not that I've thought it through that much. Does quantization work with most neural networks?
- rfoo 3y ago> Does quantization work with most neural networks? Yes. It works pretty well for CNN-based vision models. Or rather, I'd claim it works even better: with post-training quantization you can make most models work with minimal precision loss entirely in int8 (fixed point), that is, computation is over int8/int32, no floating point at all, instead of weight-only approach discussed here. If you do QAT something down to 2-bit weight and 4-bit activation would work. People aren't interested in a weight-only quantization back then because CNNs are in general "denser", i.e. bottleneck was on compute, not memory.
- jonnycomputer 3y agothanks!
- qeternity 3y agoIntuitively the output space is much smaller than the latent space. So during training, you need the higher precision so that the latent space converges. But during inference, you just need to be precise enough that your much smaller output space does.