5 ms·

Reports like this are great for embarrassing NVIDIA. But the problem is that this paper is already out of date, for embarrassing NVIDIA is the best way to get

by varelse 11y ago

Reports like this are great for embarrassing NVIDIA. But the problem is that this paper is already out of date, for embarrassing NVIDIA is the best way to get them to address deficiencies like this. This is mostly already fixed: https://github.com/soumith/convnet-benchmarks https://github.com/soumith/convnet-benchmarks Check out the huge speedup between cuDNN(R2) and cuDNN(R3). Also, while there is some fantastic ML and DNN expertise at Facebook, there are only a few GPU experts there. In contrast, NVIDIA is swarming with them. I would place my bets going forward on cuDNN. NVIDIA has bet the farm on deep learning at the expense of other problem domains. This is great for ML experts who want to add GPU support to their algorithms, and not so great for general HPC* (but then HPC just isn't as sexy as deep learning, now is it?). *For example, cuBLAS perf has serious issues and cliffs on Maxwell GPUs. Sigh...

smhx 11y ago
I run convnet-benchmarks, and also work at Facebook. Unlike what the public perception is, NVIDIA, Facebook and other companies collaborate very very closely. NVIDIA's latest R3 release has FFT-based convolutions inspired by Facebook's work, and Facebook integrated some of the optimization suggestions for fbfft that NVIDIA gave. Further optimizations on the FFT approach (like tiled-FFTs) have been suggested and prototyped at FB and carried over by NVIDIA. The relationship is more of a deep collaborative effort. At the end of the day, we want cuDNN to win, as it is a baseline system for all software. However, if you want to do research, you want to have open-source code, because you often want to do funky changes to the base algorithm. NVIDIA falls short on this, and we at Facebook pledged to always keep our code open source (bitten by the fact that we had to start with closed-source blackboxes for our initial FFT work).
- varelse 11y ago
  Sure, but most of the DNN people are just running glorified AlexNets and GoogleNets and not really looking deeply under the hood (I speak from 1st hand experience). For Caffe is open source. And cuDNN makes it fast. And if it doesn't, just jump and down like the above report did, and it will be fast in the next release. IMO if you're doing stuff for which cuDNN isn't good enough, you're probably capable of writing your own kernels, and that's awesome. Welcome to the Deep Learning equivalent of the 1%. Meanwhile, incorporate cuDNN into your production workflow to maximize comparative advantage. Or don't, whatever...
ris 11y ago
"Reports like this are great for embarrassing NVIDIA" I don't see how NVidia can really be displeased with this - this improvement continues to use CUDA and therefore lock users to NVidia hardware, failing to even acknowledge other solutions. People in this field are walking into a serious trap.
- varelse 11y ago
  You're taking things way too seriously. Besides, there really aren't any other practical deep learning solutions right now besides NVIDIA. And that's because AMD and Intel dropped the ball NVIDIA carried for the last 10 years until they finally knocked one out of the park here. No sympathy whatsoever to the pack of narrow-minded neck beards that didn't see the locomotive chugging down the tunnel until it ran them over. Even NVIDIA almost missed this one. Just ask Geoffrey Hinton about this some time. The OpenCL port Caffe has abysmal performance on AMD GPUs (see Caffe-CLGreenTea on Soumith's chart). https://github.com/BVLC/caffe/pull/2195 https://github.com/BVLC/caffe/pull/2195 Intel isn't even on the map yet, and there's no good training code for FPGAs. And ASICs? Maybe when Moore's Law slows down. And it's not a trap at all. Now that DNNs are seen as a multibillion dollar industry, the status quo will change in a few years. But until then, it's NVIDIA's gravy train, and they earned it by taking a huge risk on GPGPU. Now if you think you can build the competitor to NVIDIA and save us all from the looming trap, go for it! Meanwhile, all other things equal, use Caffe with cuDNN until someone earns your business with something better. As long as you're running at the API level, it really doesn't matter what's going on under the hood.

by varelse 11y ago

smhx 11y ago

I run convnet-benchmarks, and also work at Facebook. Unlike what the public perception is, NVIDIA, Facebook and other companies collaborate very very closely. NVIDIA's latest R3 release has FFT-based convolutions inspired by Facebook's work, and Facebook integrated some of the optimization suggestions for fbfft that NVIDIA gave. Further optimizations on the FFT approach (like tiled-FFTs) have been suggested and prototyped at FB and carried over by NVIDIA. The relationship is more of a deep collaborative effort. At the end of the day, we want cuDNN to win, as it is a baseline system for all software. However, if you want to do research, you want to have open-source code, because you often want to do funky changes to the base algorithm. NVIDIA falls short on this, and we at Facebook pledged to always keep our code open source (bitten by the fact that we had to start with closed-source blackboxes for our initial FFT work).

varelse 11y ago
Sure, but most of the DNN people are just running glorified AlexNets and GoogleNets and not really looking deeply under the hood (I speak from 1st hand experience). For Caffe is open source. And cuDNN makes it fast. And if it doesn't, just jump and down like the above report did, and it will be fast in the next release. IMO if you're doing stuff for which cuDNN isn't good enough, you're probably capable of writing your own kernels, and that's awesome. Welcome to the Deep Learning equivalent of the 1%. Meanwhile, incorporate cuDNN into your production workflow to maximize comparative advantage. Or don't, whatever...

ris 11y ago

"Reports like this are great for embarrassing NVIDIA" I don't see how NVidia can really be displeased with this - this improvement continues to use CUDA and therefore lock users to NVidia hardware, failing to even acknowledge other solutions. People in this field are walking into a serious trap.

varelse 11y ago
You're taking things way too seriously. Besides, there really aren't any other practical deep learning solutions right now besides NVIDIA. And that's because AMD and Intel dropped the ball NVIDIA carried for the last 10 years until they finally knocked one out of the park here. No sympathy whatsoever to the pack of narrow-minded neck beards that didn't see the locomotive chugging down the tunnel until it ran them over. Even NVIDIA almost missed this one. Just ask Geoffrey Hinton about this some time. The OpenCL port Caffe has abysmal performance on AMD GPUs (see Caffe-CLGreenTea on Soumith's chart). https://github.com/BVLC/caffe/pull/2195 https://github.com/BVLC/caffe/pull/2195 Intel isn't even on the map yet, and there's no good training code for FPGAs. And ASICs? Maybe when Moore's Law slows down. And it's not a trap at all. Now that DNNs are seen as a multibillion dollar industry, the status quo will change in a few years. But until then, it's NVIDIA's gravy train, and they earned it by taking a huge risk on GPGPU. Now if you think you can build the competitor to NVIDIA and save us all from the looming trap, go for it! Meanwhile, all other things equal, use Caffe with cuDNN until someone earns your business with something better. As long as you're running at the API level, it really doesn't matter what's going on under the hood.