3 ms·
If you want to use these in Torch7, my excellent colleague @soumith has released bindings at https://github.com/soumith/cudnn.torch https://github.com/soumith/c
by ajtulloch 12y ago
If you want to use these in Torch7, my excellent colleague @soumith has released bindings at https://github.com/soumith/cudnn.torch https://github.com/soumith/cudnn.torch. There are some benchmarks for the convolutional layers of a classic AlexNet model at https://github.com/soumith/convnet-benchmarks https://github.com/soumith/convnet-benchmarks.
In general, the question which is the fastest public implementation of spatial convolution is so heavily dependent on the kernel parameters (and even then, there are often knobs you can twiddle for a given implementation that can lead to substantial speedups), how much GPU memory you want to spend (GEMM and FFT based methods), so the best approach is probably just to benchmark a bunch of implementations for your network and just choose the best.
- benanne 12y agoThere are plans for Theano to build 'meta optimizers' that determine the fastest implementation separately for each layer in a network, and for each 'pass' (forward, backward w.r.t. weights, backward w.r.t. input): https://github.com/Theano/Theano/issues/2072 https://github.com/Theano/Theano/issues/2072 Sort of like how the FFTW library for FFT computation tries out a bunch of different 'plans' and then uses the fastest. I really hope this idea comes to fruition, as it would automate the "benchmark a bunch of implementations and choose the best" step, and make the process more granular.
- smhx 12y agometa-optimizers need to factor in a few more things like memory usage. Right now the FFT modules are insanely memory-hungry, and the versions written with reasonable memory usage (Michael Mathieu wrote one), are not as fast as batched CuFFT + CuBLAS