5 ms·

I don't know where this figure comes from: "Nvidia's official benchmarks show a 10x speed-up when using Caffe with cuDNN.". The graph in the original announceme

by benanne 12y ago

I don't know where this figure comes from: "Nvidia's official benchmarks show a 10x speed-up when using Caffe with cuDNN.". The graph in the original announcement ( http://devblogs.nvidia.com/parallelforall/accelerate-machine-learning-cudnn-deep-neural-network-library/ http://devblogs.nvidia.com/parallelforall/accelerate-machine... ) shows a 11x speed-up for using Caffe on the GPU versus CPU, and a 14x speed-up for using Caffe + cuDNN (again compared to CPU). Then again these figures are also kind of meaningless, as the specifics of the experiments were not specified. There are a lot of parameters that affect the performance of different implementations in various ways: the number of input feature maps, the number of filters, filter width/height, input width/height. The FFT-approach tends to do well for large filter sizes and lots of input feature maps, for example. I've played around with cudnn a bit using Theano. The Theano bindings are a work in progress, but so far they've already wrapped the convolution. Compared to conv_gemm (Theano's version of Caffe's GEMM convolution approach), it seems to be sometimes faster, sometimes slower. Soumith Chintala maintains a GitHub repository with benchmarks of various convolution implementations: https://github.com/soumith/convnet-benchmarks https://github.com/soumith/convnet-benchmarks His results aren't that spectacular either. In my own experiments cuDNN did pretty well for very small filter sizes (i.e. 3x3), often beating even the memory-hungry FFT approach). This is great because the top two scoring entries in the 2014 ImageNet competition made use of lots of convolutional layers with small filters. Of course the main benefit of cuDNN is that it will become faster over time, and that it will always be adapted to the latest NVIDIA GPUs without requiring code changes (provided that they keep maintaining it).

varelse 12y ago
You can probably figure out why 3x3 or smaller convolutions would be faster without much thought: hint FP performance is improving much faster than either internal bandwidth or CUDA kernel launch latency. And IMO this opens up the window for future GPUs to do even better. That said, when 9 TFLOPs is ~$1,100 (2 x GTX 980), I'm not too worried about memory usage in the long term.
- jimduk 12y ago
  Neophyte question - is FP necessary for a CNN as opposed to using 32 bits on some fixed encoding. If I used a 32 bit value on the order of +- 4.27 or +- 8.23, wouldn't that have enough accuracy. I'm assuming the weights and parameters don't go much above say 8 or 16 after all the Relu stuff.
  - benanne 12y ago
    Probably not. The noise in the training data itself, noise from dropout and various other sources are going to trump any quantization noise anyway, so you can use fairly inaccurate representations. This is why everyone's using gamer GPUs for this stuff, and not Tesla cards: single precision is enough, and much cheaper to come by. Although, I guess the cutoff might have to be a bit higher than 8.23. Maybe the neuron activations would never exceed that range, but some intermediate computations could. Supposedly the new Maxwell GPUs have some new instructions for working with half-precision floats ( https://developer.nvidia.com/sites/default/files/akamai/opengl/specs/GL_NV_shader_atomic_fp16_vector.txt https://developer.nvidia.com/sites/default/files/akamai/open... ). I wonder how complete this implementation is, because half-precision might be sufficient to train a convnet, and this would result in a significant speedup.
    - smhx 12y ago
      In these deep relu networks which are not renormalized in between, (like overfeat which has no normalization), some of the activations become pretty big in size! (in the order of 1e3). You also cant clip them and get away with it, you have to either renormalize the layers to do half-precision (and live with the extra cost) or stick to full-precision. I was doing fun stuff early this year that did fixed-precision nets (8-bit/16-bit). Things get very interesting :)
      - varelse 12y ago
        16-bit float has dynamic range from + or - 6.5e04 to 6.1e-05. http://en.wikipedia.org/wiki/Half-precision_floating-point_format http://en.wikipedia.org/wiki/Half-precision_floating-point_f... That's plenty IMO for most inputs and weights. Where it gets tricky is in accumulation. You could constrain the weights for each unit I guess, but this is the sort of work best done under the hood rather than by the data scientist IMO. I'd personally choose 32-bit accumulation just because it would drastically simplify code development. I've also worked with fixed precision elsewhere. It's awesome if you understand the dynamic range of your application. It's a migraine headache if you don't.

by benanne 12y ago

varelse 12y ago

You can probably figure out why 3x3 or smaller convolutions would be faster without much thought: hint FP performance is improving much faster than either internal bandwidth or CUDA kernel launch latency. And IMO this opens up the window for future GPUs to do even better. That said, when 9 TFLOPs is ~$1,100 (2 x GTX 980), I'm not too worried about memory usage in the long term.

jimduk 12y ago
Neophyte question - is FP necessary for a CNN as opposed to using 32 bits on some fixed encoding. If I used a 32 bit value on the order of +- 4.27 or +- 8.23, wouldn't that have enough accuracy. I'm assuming the weights and parameters don't go much above say 8 or 16 after all the Relu stuff.
- benanne 12y ago
  Probably not. The noise in the training data itself, noise from dropout and various other sources are going to trump any quantization noise anyway, so you can use fairly inaccurate representations. This is why everyone's using gamer GPUs for this stuff, and not Tesla cards: single precision is enough, and much cheaper to come by. Although, I guess the cutoff might have to be a bit higher than 8.23. Maybe the neuron activations would never exceed that range, but some intermediate computations could. Supposedly the new Maxwell GPUs have some new instructions for working with half-precision floats ( https://developer.nvidia.com/sites/default/files/akamai/opengl/specs/GL_NV_shader_atomic_fp16_vector.txt https://developer.nvidia.com/sites/default/files/akamai/open... ). I wonder how complete this implementation is, because half-precision might be sufficient to train a convnet, and this would result in a significant speedup.
  - smhx 12y ago
    In these deep relu networks which are not renormalized in between, (like overfeat which has no normalization), some of the activations become pretty big in size! (in the order of 1e3). You also cant clip them and get away with it, you have to either renormalize the layers to do half-precision (and live with the extra cost) or stick to full-precision. I was doing fun stuff early this year that did fixed-precision nets (8-bit/16-bit). Things get very interesting :)
    - varelse 12y ago
      16-bit float has dynamic range from + or - 6.5e04 to 6.1e-05. http://en.wikipedia.org/wiki/Half-precision_floating-point_format http://en.wikipedia.org/wiki/Half-precision_floating-point_f... That's plenty IMO for most inputs and weights. Where it gets tricky is in accumulation. You could constrain the weights for each unit I guess, but this is the sort of work best done under the hood rather than by the data scientist IMO. I'd personally choose 32-bit accumulation just because it would drastically simplify code development. I've also worked with fixed precision elsewhere. It's awesome if you understand the dynamic range of your application. It's a migraine headache if you don't.