Sure, but most of the DNN people are just running glorified AlexNets and GoogleNets and not really looking deeply under the hood (I speak from 1st hand experien

Sure, but most of the DNN people are just running glorified AlexNets and GoogleNets and not really looking deeply under the hood (I speak from 1st hand experience). For Caffe is open source. And cuDNN makes it fast. And if it doesn't, just jump and down like the above report did, and it will be fast in the next release. IMO if you're doing stuff for which cuDNN isn't good enough, you're probably capable of writing your own kernels, and that's awesome. Welcome to the Deep Learning equivalent of the 1%. Meanwhile, incorporate cuDNN into your production workflow to maximize comparative advantage. Or don't, whatever...