I think a good answer to this is here talking about the tensor cores on the Nvidia Titan V: https://github.com/tensorflow/tensorflow/issues/15897#issuecomment-3

I think a good answer to this is here talking about the tensor cores on the Nvidia Titan V: https://github.com/tensorflow/tensorflow/issues/15897#issuecomment-356503342 https://github.com/tensorflow/tensorflow/issues/15897#issuec... If you can write a better optimized network, go ahead. But like in SSE2 vs AVX2 vs AVX512 benchmarks, FP performance on paper doesn't always translate into better real world FP performance. Now if Nvidia had switched to HBM2 like Google TPU2, it might be different.