4 ms·
7 bit mantissa doesn't sound like a lot, 2^-7 is 0.0078125... shouldn't there be at least 9 bits for the mantissa?
by jokoon 8y ago
7 bit mantissa doesn't sound like a lot, 2^-7 is 0.0078125... shouldn't there be at least 9 bits for the mantissa?
- yyww 8y agoThe format optimizes for dynamic range over precision.
- kps 8y agoThere's an existing standardized 16-bit float with 10 bits mantissa¹, that graphics people are fond of. This one is for machine learning [drink]. ¹ https://en.wikipedia.org/wiki/Half-precision_floating-point_format https://en.wikipedia.org/wiki/Half-precision_floating-point_...
- wolfgke 8y agoHave a look at section 1.1 and Figure 1-1 of the Intel Whitepaper. There this topic is discussed.
- masklinn 8y agoNote that 7 bits stored means 8 bit mantissa. And bfloat16 is new but not that new: Tensorflow had it 3 years back[0]. Apparently range is more useful than precision for machine learning, which would be why they went 8/8 instead of IEEE's 11/5 FP16. [0] https://github.com/tensorflow/tensorflow/blob/f41959ccb2d9d4c722fe8fc3351401d53bcf4900/tensorflow/core/framework/bfloat16.h https://github.com/tensorflow/tensorflow/blob/f41959ccb2d9d4...
- espadrine 8y agoMy understanding is that the optimization steps of neural networks (typically gradient descent + backpropagation) act a bit like Lloyd's algorithm. Neuron weights will push each other into place. Often, what matters is how they compare to each other, such that the error is minimized; reaching the theoretically-perfect weight is less important.
- Tuna-Fish 8y agoWith the bonus bit, that's 2.4 significant digits of precision. Plenty enough for ML, especially if you use a F32 as the accumulator.