> We relate the degradation results to the numerical stability in training. fp16 format uses a smaller range for the exponent than bf16. While bf16 uses 8 bits

> We relate the degradation results to the numerical stability in training. fp16 format uses a smaller range for the exponent than bf16. While bf16 uses 8 bits for the exponent, fp16 only uses 5. Most floating point formats also have denormalized numbers which allow for a soft underflow. This can get exponentially closer to 0.0f for each additional bit in the mantissa. This makes underflow more of a concern for floating point formats.