You'd be surprised how many of the old, painstakingly tested fp algorithms, have been reimplemented in DL frameworks. I ported many algorithms from cephes to E

You'd be surprised how many of the old, painstakingly tested fp algorithms, have been reimplemented in DL frameworks. I ported many algorithms from cephes to Eigen for use in TensorFlow. Some I was able to make simd. Then the XLA folks reimplemented some in HLO using other approximations that are easier to parallelize. Then consider that some linear algebra and simple functions like softmax and softplus have to be implemented for fast execution in float16 and bfloat16 supporting hardware. You can iterate over the entire fp16 space quite quickly but the initial version isn't usually so thoroughly tested.. performance is king anyway in NNs. Then a couple months later someone ports an old model to V100 and it turns out your softplus underflows. Guess how many weeks it takes to figure out why your model quality went to s**?