There are plans for Theano to build 'meta optimizers' that determine the fastest implementation separately for each layer in a network, and for each 'pass' (for

There are plans for Theano to build 'meta optimizers' that determine the fastest implementation separately for each layer in a network, and for each 'pass' (forward, backward w.r.t. weights, backward w.r.t. input): https://github.com/Theano/Theano/issues/2072 https://github.com/Theano/Theano/issues/2072 Sort of like how the FFTW library for FFT computation tries out a bunch of different 'plans' and then uses the fastest. I really hope this idea comes to fruition, as it would automate the "benchmark a bunch of implementations and choose the best" step, and make the process more granular.