3 ms·
float16 or bfloat16?
by danielscrubs 7y ago
float16 or bfloat16?
- bonoboTP 7y agoI wasn't aware of the difference, but now I looked. I've always used float16, not bfloat16 (tf.float16 is the usual float16), and it causes a large speed boost on Turing NVIDIA cards with tensor cores as opposed to float32. But certainly bfloat16 sounds attractive with its larger dynamic range, as float16 requires workarounds like loss scaling to avoid rounding down to zero when the exponent gets too small.
- bonoboTP 7y agoSeems like bfloat16 is only available for TPUs in TensorFlow[0-1]. [0] https://github.com/tensorflow/tensorflow/issues/21612 https://github.com/tensorflow/tensorflow/issues/21612 [1] https://github.com/tensorflow/tensorflow/issues/21317 https://github.com/tensorflow/tensorflow/issues/21317