> For machine learning Tensor Cores in the upcoming Volta GPUs seem much better idea, delivering 120 TFLOPS. Yes, but the real-world benchmark numbers nvidia p

> For machine learning Tensor Cores in the upcoming Volta GPUs seem much better idea, delivering 120 TFLOPS. Yes, but the real-world benchmark numbers nvidia published show a much smaller speedup than one would expect if one looked only at the 120TFLOPS number. This is because (a) "tensor core" is just a fancy name for "fast 4x4 matmul instruction" -- it's not applicable to everything you might do on the GPU, and (b) memory bandwidth did not increase commensurate with compute speed. (I work on the XLA GPU compiler in TensorFlow.)