3 ms·
Note that 7 bits stored means 8 bit mantissa. And bfloat16 is new but not that new: Tensorflow had it 3 years back[0]. Apparently range is more useful than pr
by masklinn 8y ago
Note that 7 bits stored means 8 bit mantissa.
And bfloat16 is new but not that new: Tensorflow had it 3 years back[0].
Apparently range is more useful than precision for machine learning, which would be why they went 8/8 instead of IEEE's 11/5 FP16.
[0] https://github.com/tensorflow/tensorflow/blob/f41959ccb2d9d4c722fe8fc3351401d53bcf4900/tensorflow/core/framework/bfloat16.h https://github.com/tensorflow/tensorflow/blob/f41959ccb2d9d4...
- espadrine 8y agoMy understanding is that the optimization steps of neural networks (typically gradient descent + backpropagation) act a bit like Lloyd's algorithm. Neuron weights will push each other into place. Often, what matters is how they compare to each other, such that the error is minimized; reaching the theoretically-perfect weight is less important.