3 ms·

I'm surprised the comments here are so negative / cynical. Of course "quantization doesn't come for free": it's a trade off. High precision models are very exp

by bambax 2y ago

I'm surprised the comments here are so negative / cynical. Of course "quantization doesn't come for free": it's a trade off. High precision models are very expensive to run: they require expensive hardware and lots of energy. Isn't it possible, or even likely, that low-precision models would be good enough for many tasks?

az09mugen 2y ago
I totally agree with your statement. I consider LLMs imprecise in any case, it's not a perfect/exact science, just statistics. I only use LLMs for tasks where an error margin can be allowed. In that perspective I'm totally fine with using a 5 MB LLM like this one : https://neuml.hashnode.dev/train-a-language-model-from-scratch https://neuml.hashnode.dev/train-a-language-model-from-scrat...
kromem 2y ago
Post-training quantization doesn't come for free, but the pretraining on constrained precision weights actually counterintuitively results in a performance increase per parameter as the number of parameters grows in the ternary BitNet paper. Even if there was zero efficiency gain in ternary weights, large models should probably be trained on a network of precision limited weights from here on out given the research so far. I suspect it relates to each weight relating to multiple 'features' in the network. The greater the precision, the more room it gives for competing features to compromise on node values that aren't best for either feature instead of reorganizing the feature mapping to avoid conflicts.
- trashtester 2y ago
  The number of bits used per weight during training, could be included in the regularization, perhaps? For instance, one could extend dropout regularization to several levels, where each weight could have random chances to include the most significant 2-16 bits part of the time (and still 0 part of the time), and where the impact on the gradient of having fewer bits could be used to tune the ideal number of bits for each weight. Then one could add L1 regularization for the total number of bits used to squeeze the total down down to whatever size one aims for.
k__ 2y ago
lol, just like with humans.
demosthanos 2y ago
> I'm surprised the comments here are so negative / cynical. And by "the comments" I'm assuming you mean the top comment [0] (and maybe its replies)? The rest don't really come off as negative at all, and you quoted directly from that top comment. FWIW, I don't find that comment either negative or cynical. It starts out with that sentence you quoted, but it goes on to make a very interesting point about quantization most likely working best for models which are undertrained—models which store less information than their architecture size would suggest. That's a very valid point that I found insightful and interesting, not cynical. [0] https://news.ycombinator.com/item?id=40531638 https://news.ycombinator.com/item?id=40531638
jononor 2y ago
Several years of ML research for CNNs indicate at least that one can do very well with 8 bit integers. Such quantization is basically standard now for any deployment outside of GPU (where 8bit isn't any faster anyway due to the hardware).

by bambax 2y ago

az09mugen 2y ago

I totally agree with your statement. I consider LLMs imprecise in any case, it's not a perfect/exact science, just statistics. I only use LLMs for tasks where an error margin can be allowed. In that perspective I'm totally fine with using a 5 MB LLM like this one : https://neuml.hashnode.dev/train-a-language-model-from-scratch https://neuml.hashnode.dev/train-a-language-model-from-scrat...

kromem 2y ago

Post-training quantization doesn't come for free, but the pretraining on constrained precision weights actually counterintuitively results in a performance increase per parameter as the number of parameters grows in the ternary BitNet paper. Even if there was zero efficiency gain in ternary weights, large models should probably be trained on a network of precision limited weights from here on out given the research so far. I suspect it relates to each weight relating to multiple 'features' in the network. The greater the precision, the more room it gives for competing features to compromise on node values that aren't best for either feature instead of reorganizing the feature mapping to avoid conflicts.

trashtester 2y ago
The number of bits used per weight during training, could be included in the regularization, perhaps? For instance, one could extend dropout regularization to several levels, where each weight could have random chances to include the most significant 2-16 bits part of the time (and still 0 part of the time), and where the impact on the gradient of having fewer bits could be used to tune the ideal number of bits for each weight. Then one could add L1 regularization for the total number of bits used to squeeze the total down down to whatever size one aims for.

k__ 2y ago

lol, just like with humans.

demosthanos 2y ago

> I'm surprised the comments here are so negative / cynical. And by "the comments" I'm assuming you mean the top comment [0] (and maybe its replies)? The rest don't really come off as negative at all, and you quoted directly from that top comment. FWIW, I don't find that comment either negative or cynical. It starts out with that sentence you quoted, but it goes on to make a very interesting point about quantization most likely working best for models which are undertrained—models which store less information than their architecture size would suggest. That's a very valid point that I found insightful and interesting, not cynical. [0] https://news.ycombinator.com/item?id=40531638 https://news.ycombinator.com/item?id=40531638

jononor 2y ago

Several years of ML research for CNNs indicate at least that one can do very well with 8 bit integers. Such quantization is basically standard now for any deployment outside of GPU (where 8bit isn't any faster anyway due to the hardware).