4 ms·

To repeat the other comments, right now CuDNN is the advantage - that is manifested as TensorFlow/Keras/PyTorch. AMD have RocM and in these benchmarks it was li

To repeat the other comments, right now CuDNN is the advantage - that is manifested as TensorFlow/Keras/PyTorch. AMD have RocM and in these benchmarks it was like 10X or more slower training for CNNs that P100s - https://www.pcper.com/reviews/Graphics-Cards/NVIDIA-TITAN-V-Review-Part-3-Deep-Learning-Performance https://www.pcper.com/reviews/Graphics-Cards/NVIDIA-TITAN-V-... How sustainable is the advantage? Not that big - you don't need cuda compatability like hiptensorflow tried in a classic shortcuts-dont-work way. Just an alternative CuDNN for Vega that is integrated in TensorFlow distributed binaries.

nabla9 9y ago
In other words Nvidia has no meaningful edge in hardware microarchitecture, patents and innovation. Is this correct? If you remove the software and compare the raw cost/flops/bandwidth/memory/energy to others the difference is small?
- jamesblonde 9y ago
  Yeah, pretty much. They are claiming the future is specialization in compute. Tensorcores are an attempt at this - they should more accurately be called "4x4 matrix multiply in half precision with a full-precision add". That is the instruction they support. Unless you are training a very deep CNN with a lot of 4x4 kernels, they won't give you much of a speedup. You're left then with the memory-GPU b/w. For the 1080Ti, that's 484GB/s and it's about 950GB/s for the V100. So about twice the training performance for 10 times the price. Not a good deal, IMO. When you compare that with the top Vega cards, it's a complete rip-off - but Vega cards are currently useless for deep learning. I'm not convinced AMD have put the money in to build the libraries, so we're stuck with AMD for a while yet (unless Intel by some miracle pull one out with its new platform).
  - nightski 9y ago
    What about NVLink? I'm pretty sure there is no equivalent for AMD right? That would be a pretty big deal.
    - jamesblonde 9y ago
      NVLink is good if you are doing distributed training and sharing large amounts of gradients (very large models). If you're doing parallel experiments or your models are not huge, then PCI-e (16 GB/s) is generally good enough. AMD don't have anything close that i am aware of, but their new Ryzen board will have double the number of PCI lanes, which makes PCI-e competitive again - https://news.ycombinator.com/item?id=14450924 https://news.ycombinator.com/item?id=14450924
  - MachineBurning 9y ago
    > Unless you are training a very deep CNN with a lot of 4x4 kernels, they won't give you much of a speedup That's not true. 4x4 kernels are not a requirement and Tensor Cores work on GEMMs as well: https://devblogs.nvidia.com/programming-tensor-cores-cuda-9/ https://devblogs.nvidia.com/programming-tensor-cores-cuda-9/ They can also be used in more HPC-like applications, such as iterative refinement, as shown by Jack Dongarra's group ( http://icl.cs.utk.edu/projectsfiles/magma/pubs/68-main.pdf http://icl.cs.utk.edu/projectsfiles/magma/pubs/68-main.pdf - ~3x speedup vs pure fp64 for the same accuracy)
- wmf 9y ago
  NVidia definitely does have an edge in GPU hardware. AMD Vega is roughly tied with NVidia Pascal in performance, but Vega consumes more power and came out two years later.
  - sanxiyn 9y ago
    NVIDIA may have an edge in GPU hardware, but probably not in GPU hardware architecture. AMD is contractually required to use GF process which is particularly bad in the current generation, but will be competitive in the next generation. My understanding is that process difference accounts for most of edge NVIDIA has.

nabla9 9y ago

In other words Nvidia has no meaningful edge in hardware microarchitecture, patents and innovation. Is this correct? If you remove the software and compare the raw cost/flops/bandwidth/memory/energy to others the difference is small?

jamesblonde 9y ago
Yeah, pretty much. They are claiming the future is specialization in compute. Tensorcores are an attempt at this - they should more accurately be called "4x4 matrix multiply in half precision with a full-precision add". That is the instruction they support. Unless you are training a very deep CNN with a lot of 4x4 kernels, they won't give you much of a speedup. You're left then with the memory-GPU b/w. For the 1080Ti, that's 484GB/s and it's about 950GB/s for the V100. So about twice the training performance for 10 times the price. Not a good deal, IMO. When you compare that with the top Vega cards, it's a complete rip-off - but Vega cards are currently useless for deep learning. I'm not convinced AMD have put the money in to build the libraries, so we're stuck with AMD for a while yet (unless Intel by some miracle pull one out with its new platform).
- nightski 9y ago
  What about NVLink? I'm pretty sure there is no equivalent for AMD right? That would be a pretty big deal.
  - jamesblonde 9y ago
    NVLink is good if you are doing distributed training and sharing large amounts of gradients (very large models). If you're doing parallel experiments or your models are not huge, then PCI-e (16 GB/s) is generally good enough. AMD don't have anything close that i am aware of, but their new Ryzen board will have double the number of PCI lanes, which makes PCI-e competitive again - https://news.ycombinator.com/item?id=14450924 https://news.ycombinator.com/item?id=14450924
- MachineBurning 9y ago
  > Unless you are training a very deep CNN with a lot of 4x4 kernels, they won't give you much of a speedup That's not true. 4x4 kernels are not a requirement and Tensor Cores work on GEMMs as well: https://devblogs.nvidia.com/programming-tensor-cores-cuda-9/ https://devblogs.nvidia.com/programming-tensor-cores-cuda-9/ They can also be used in more HPC-like applications, such as iterative refinement, as shown by Jack Dongarra's group ( http://icl.cs.utk.edu/projectsfiles/magma/pubs/68-main.pdf http://icl.cs.utk.edu/projectsfiles/magma/pubs/68-main.pdf - ~3x speedup vs pure fp64 for the same accuracy)
wmf 9y ago
NVidia definitely does have an edge in GPU hardware. AMD Vega is roughly tied with NVidia Pascal in performance, but Vega consumes more power and came out two years later.
- sanxiyn 9y ago
  NVIDIA may have an edge in GPU hardware, but probably not in GPU hardware architecture. AMD is contractually required to use GF process which is particularly bad in the current generation, but will be competitive in the next generation. My understanding is that process difference accounts for most of edge NVIDIA has.