3 ms·
Some parameters would be more sensitive than others I suppose? So could you use 4 bits for most, and 8 bits, or even 16, for the remaining?
by magicalhippo 4y ago
Some parameters would be more sensitive than others I suppose? So could you use 4 bits for most, and 8 bits, or even 16, for the remaining?
- sour-taste 4y agoI know nothing about this so my opinion means little, but I imagine it's hard to know which parameters are important enough to use more bits for. I do wonder if it would be possible to have the model determine during training how important each parameter is, while maybe rewarding it for having more small parameters?
- dongping 4y agoThat's exactly why bitsandbytes has a threshold parameter to control the quantization.
- magicalhippo 4y agoNice, good to know, thanks!