5 ms·
Well I guess it's the “blowing up parameter count to make up for it” that confuses me, but maybe it's just ignorance. Like what would be the expected factor of
by ncruces 3y ago
Well I guess it's the “blowing up parameter count to make up for it” that confuses me, but maybe it's just ignorance.
Like what would be the expected factor of this blow up to make up the difference between ternary and whatever 16 bits encoding they were using?
I mean intuitively I'd expect to need ~10× the symbols to encode the same information? Are they using an order of magnitude more parameters, or is that not how it works?
- int_19h 3y agoWith existing common quantization techniques, a 70b model quantized to 3-bit still drastically outperforms an unquantized 35b model.
- p1esk 3y agoAre you sure? I was under impression that 3b quantization still results in a significant degradation. Which quantization method are you talking about?
- int_19h 3y agoIt does result in a significant degradation relative to unquantized model of the same size, but even with simple llama.cpp K-quantization, it's still worth it all the way down to 2-bit. The chart in this llama.cpp PR speaks for itself: https://github.com/ggerganov/llama.cpp/pull/1684#issue-1739619305 https://github.com/ggerganov/llama.cpp/pull/1684#issue-17396...
- p1esk 3y agoOh wow, you’re right. Though it seems that they are using very small weight group sizes: either 16 or 32 (fp16 scaling factor per group). In this paper it seems there’s no weights grouping, so it’s a bit apples to oranges.