4 ms·
This is not generally true, sometimes quantisation can improve accuracy. I haven't seen that with LLMs yet though.
by WithinReason 3y ago
This is not generally true, sometimes quantisation can improve accuracy. I haven't seen that with LLMs yet though.
- arijun 3y agoInteresting, how would that work? Are there any well-known examples? Is it: the weights all happen to be where float is sparse, so quantization ends up increasing fidelity? Or is it more of a “worse is better” dropout-type situation?
- WithinReason 3y agoI suspect it works as regularisation of the network. It usually happens when you train with quantisation instead of post-training quantisation, an I haven't seen that done with LLMs yet.
- matsemann 3y agoFor image recognition it can sometimes be like that. My gut feeling is that lowering from fp32 to fp16 can get rid of some kind of overfitting or so.