4 ms·

Not everything in the compute pipeline is going to be converted to fp16 operations. Anytime you are doing accumulation or exponentials, you would have to have i

Not everything in the compute pipeline is going to be converted to fp16 operations. Anytime you are doing accumulation or exponentials, you would have to have it in fp32. There was a good talk from NVIDIA at last years GTC: http://on-demand.gputechconf.com/gtc/2018/presentation/s8923-training-neural-networks-with-mixed-precision-theory-and-practice.pdf http://on-demand.gputechconf.com/gtc/2018/presentation/s8923... Here is another relevant blog post: https://devblogs.nvidia.com/mixed-precision-training-deep-neural-networks/ https://devblogs.nvidia.com/mixed-precision-training-deep-ne... EDIT: Also not everything in the training loop is a matrix multiplication where tensor cores are useful.

shaklee3 8y ago
Right, but the tensor cores should be about 10x faster on the compute side, and about 2x the memory bandwidth. GEMM is usually constrained by compute, which is why the tensor cores exist. These benchmarks are for training, so the expectation is that they are running them in fp16 all the way through. Also, tensor cores can accumulate in fp32 registers with a slight hit to performance.
- KaoruAoiShiho 8y ago
  Turing tensor cores are not as fast as Volta ones. You won't see the 10x number repeated on any turing marketing material.
  - shaklee3 8y ago
    What's your source for that? They should be the same speed, but more capable. You absolutely see the 8x in their marketing: https://devblogs.nvidia.com/nvidia-turing-architecture-in-depth/ https://devblogs.nvidia.com/nvidia-turing-architecture-in-de...
- sabalaba 8y ago
  Is there a benchmark you've seen that matches the claimed 10x increase in performance for the Tensor Cores? NVIDIA hype train can sometimes make it difficult to find hard numbers.
  - shaklee3 8y ago
    I've done my own benchmarks where I've hit over 100 TFLOPS on the V100, and that's about 85% of the peak theoretical throughout of them. Granted the matrix size needs to be large enough, but it's definitely doable. Anandtech also showed similar results in their V100 review. I haven't yet seen a comparable SGEMM done on the 2080Ti, so I don't know how it'll compare. I have some coming in at the end of the month though, so I should know soon.
  - dimitry12 8y ago
    Titan V in my testing only approached 100+TFlops in purely synthetic large matmuls. With carefully tuned Transformer (matmul-heavy!) I could only make twice as fast as 1080ti (at 4 times the price). The only undisputable benefit was using double batch size.

shaklee3 8y ago

Right, but the tensor cores should be about 10x faster on the compute side, and about 2x the memory bandwidth. GEMM is usually constrained by compute, which is why the tensor cores exist. These benchmarks are for training, so the expectation is that they are running them in fp16 all the way through. Also, tensor cores can accumulate in fp32 registers with a slight hit to performance.

KaoruAoiShiho 8y ago
Turing tensor cores are not as fast as Volta ones. You won't see the 10x number repeated on any turing marketing material.
- shaklee3 8y ago
  What's your source for that? They should be the same speed, but more capable. You absolutely see the 8x in their marketing: https://devblogs.nvidia.com/nvidia-turing-architecture-in-depth/ https://devblogs.nvidia.com/nvidia-turing-architecture-in-de...
sabalaba 8y ago
Is there a benchmark you've seen that matches the claimed 10x increase in performance for the Tensor Cores? NVIDIA hype train can sometimes make it difficult to find hard numbers.
- shaklee3 8y ago
  I've done my own benchmarks where I've hit over 100 TFLOPS on the V100, and that's about 85% of the peak theoretical throughout of them. Granted the matrix size needs to be large enough, but it's definitely doable. Anandtech also showed similar results in their V100 review. I haven't yet seen a comparable SGEMM done on the 2080Ti, so I don't know how it'll compare. I have some coming in at the end of the month though, so I should know soon.
- dimitry12 8y ago
  Titan V in my testing only approached 100+TFlops in purely synthetic large matmuls. With carefully tuned Transformer (matmul-heavy!) I could only make twice as fast as 1080ti (at 4 times the price). The only undisputable benefit was using double batch size.