3 ms·

Yeah, pretty much. They are claiming the future is specialization in compute. Tensorcores are an attempt at this - they should more accurately be called "4x4 ma

Yeah, pretty much. They are claiming the future is specialization in compute. Tensorcores are an attempt at this - they should more accurately be called "4x4 matrix multiply in half precision with a full-precision add". That is the instruction they support. Unless you are training a very deep CNN with a lot of 4x4 kernels, they won't give you much of a speedup. You're left then with the memory-GPU b/w. For the 1080Ti, that's 484GB/s and it's about 950GB/s for the V100. So about twice the training performance for 10 times the price. Not a good deal, IMO. When you compare that with the top Vega cards, it's a complete rip-off - but Vega cards are currently useless for deep learning. I'm not convinced AMD have put the money in to build the libraries, so we're stuck with AMD for a while yet (unless Intel by some miracle pull one out with its new platform).

nightski 9y ago
What about NVLink? I'm pretty sure there is no equivalent for AMD right? That would be a pretty big deal.
- jamesblonde 9y ago
  NVLink is good if you are doing distributed training and sharing large amounts of gradients (very large models). If you're doing parallel experiments or your models are not huge, then PCI-e (16 GB/s) is generally good enough. AMD don't have anything close that i am aware of, but their new Ryzen board will have double the number of PCI lanes, which makes PCI-e competitive again - https://news.ycombinator.com/item?id=14450924 https://news.ycombinator.com/item?id=14450924
MachineBurning 9y ago
> Unless you are training a very deep CNN with a lot of 4x4 kernels, they won't give you much of a speedup That's not true. 4x4 kernels are not a requirement and Tensor Cores work on GEMMs as well: https://devblogs.nvidia.com/programming-tensor-cores-cuda-9/ https://devblogs.nvidia.com/programming-tensor-cores-cuda-9/ They can also be used in more HPC-like applications, such as iterative refinement, as shown by Jack Dongarra's group ( http://icl.cs.utk.edu/projectsfiles/magma/pubs/68-main.pdf http://icl.cs.utk.edu/projectsfiles/magma/pubs/68-main.pdf - ~3x speedup vs pure fp64 for the same accuracy)

nightski 9y ago

What about NVLink? I'm pretty sure there is no equivalent for AMD right? That would be a pretty big deal.

jamesblonde 9y ago
NVLink is good if you are doing distributed training and sharing large amounts of gradients (very large models). If you're doing parallel experiments or your models are not huge, then PCI-e (16 GB/s) is generally good enough. AMD don't have anything close that i am aware of, but their new Ryzen board will have double the number of PCI lanes, which makes PCI-e competitive again - https://news.ycombinator.com/item?id=14450924 https://news.ycombinator.com/item?id=14450924

MachineBurning 9y ago

> Unless you are training a very deep CNN with a lot of 4x4 kernels, they won't give you much of a speedup That's not true. 4x4 kernels are not a requirement and Tensor Cores work on GEMMs as well: https://devblogs.nvidia.com/programming-tensor-cores-cuda-9/ https://devblogs.nvidia.com/programming-tensor-cores-cuda-9/ They can also be used in more HPC-like applications, such as iterative refinement, as shown by Jack Dongarra's group ( http://icl.cs.utk.edu/projectsfiles/magma/pubs/68-main.pdf http://icl.cs.utk.edu/projectsfiles/magma/pubs/68-main.pdf - ~3x speedup vs pure fp64 for the same accuracy)