4 ms·

Very good write up. If you want to trade speed and memory for accuracy, you can make a large lookup table for your sigmoidal function which should just about do

Very good write up. If you want to trade speed and memory for accuracy, you can make a large lookup table for your sigmoidal function which should just about double the speed of it. As an aside, and not to be too critical, because the post was great, but as (presumably) a non-native English speaker, you might do a spell-checker on your post. There are also some missing pronouns which make some sentences very Spanishy.

dfrodriguez143 13y ago
Definitely a native spanish speaker here :P. Because I wrote this on an iPython notebook it takes a little bit longer to spell-check. I will try not to be so lazy next time. Thanks for the tips.
- MaxGabriel 13y ago
  The only one I caught was "state of the are" in the last sentence.
lightcatcher 13y ago
A relatively small lookup table for the sigmoid function can also work well. Here are the various sigmoid approximations that Theano (a library used for deep learning research among other things) offers: http://deeplearning.net/software/theano/library/tensor/nnet/nnet.html#tensor.nnet.sigmoid http://deeplearning.net/software/theano/library/tensor/nnet/...
- gamegoblin 13y ago
  I usually use an array with a few thousand entries. In C this gives me a 2.5x speedup over the exact function with no important decrease in accuracy.
  - dfrodriguez143 13y ago
    Very interesting and simple improvement. Definitely will take a look at that.
  - mistercow 13y ago
    I wonder though if you might actually do better overall with a smaller lookup table and interpolation (or even just a polynomial approximation, which can be evaluated without branching), since large lookup tables can lead to bad cache behavior.
gallamine 13y ago
You could also try the Elliot Sigmoid activation function. I found it executed about 2x faster than the exponential sigmoid (in Matlab). Brief writeup: http://www.gallamine.com/blog/2013/01/21/a-sigmoid-function-with-out-the-exponential-elliot-sigmoid-activation-function/ http://www.gallamine.com/blog/2013/01/21/a-sigmoid-function-...

dfrodriguez143 13y ago

Definitely a native spanish speaker here :P. Because I wrote this on an iPython notebook it takes a little bit longer to spell-check. I will try not to be so lazy next time. Thanks for the tips.

MaxGabriel 13y ago
The only one I caught was "state of the are" in the last sentence.

lightcatcher 13y ago

A relatively small lookup table for the sigmoid function can also work well. Here are the various sigmoid approximations that Theano (a library used for deep learning research among other things) offers: http://deeplearning.net/software/theano/library/tensor/nnet/nnet.html#tensor.nnet.sigmoid http://deeplearning.net/software/theano/library/tensor/nnet/...

gamegoblin 13y ago
I usually use an array with a few thousand entries. In C this gives me a 2.5x speedup over the exact function with no important decrease in accuracy.
- dfrodriguez143 13y ago
  Very interesting and simple improvement. Definitely will take a look at that.
- mistercow 13y ago
  I wonder though if you might actually do better overall with a smaller lookup table and interpolation (or even just a polynomial approximation, which can be evaluated without branching), since large lookup tables can lead to bad cache behavior.

gallamine 13y ago

You could also try the Elliot Sigmoid activation function. I found it executed about 2x faster than the exponential sigmoid (in Matlab). Brief writeup: http://www.gallamine.com/blog/2013/01/21/a-sigmoid-function-with-out-the-exponential-elliot-sigmoid-activation-function/ http://www.gallamine.com/blog/2013/01/21/a-sigmoid-function-...