For casual readers, the trend in neural nets has changed a bit, and for training TensorFloat-32 is gaining popularity: https://blogs.nvidia.com/blog/2020/05/14/

For casual readers, the trend in neural nets has changed a bit, and for training TensorFloat-32 is gaining popularity: https://blogs.nvidia.com/blog/2020/05/14/tensorfloat-32-precision-format/ https://blogs.nvidia.com/blog/2020/05/14/tensorfloat-32-prec... Pure f16 overflows/underflows in many scenarios