3 ms·
There are research papers where even 1 bit (not floating point) was enough, with some quality loss. 4 bits is effectively 16 different float point numbers - 8
by yalok 3y ago
There are research papers where even 1 bit (not floating point) was enough, with some quality loss.
4 bits is effectively 16 different float point numbers - 8 positive, 8 negative, no zero and no NaN/inf. 1 bit for sign and 3 bits for exponent, 0 bits for mantissa, mantissa is implied to be 4. It’s logarithmic - representing numbers in the range from -4^3 to 4^3, smallest numbers are 4^-3.
- s_m_t 3y agoThanks, I was thinking that zero, negative zero, inf, negative inf, and the NaN's were included like in IEEE 754
- carlmr 3y ago>1 bit (not floating point) I like how you specified that it's not floating point.
- phh 3y agoThanks. First source i see for what fp4 is. Gotta say I'm surprised: I would have chosen to lose one value, but have a zero. (though I have no doubt those people are much more clever and knowledgeable than I am)
- omikun 3y agoIf the weight is zero it doesn’t need to exist