3 ms·
I seem to recall that before ReLU it was usually tanh.
by canjobear 5y ago
I seem to recall that before ReLU it was usually tanh.
- whatshisface 5y agoTanh belongs to a class of functions called "smooth steps" which I guess was being abbreviated as "step." Obviously the derivative of a step is zero everywhere it's defined so backprop wouldn't work.
- l33tman 5y agoAn interesting thing is that ReLu (which is absolutely the most common activation function) has a zero derivative over half the space its defined, and backprop still works (most of the time :). There are leaky ReLU to give some slope to the negative side, but it seems in most normal deep learning networks this is not required, there are always some neurons that aren't in the off regime to "catch the error".
- _0ffh 5y agoOne of a whole class of functions called sigmoidal, usally the sigmoid or the tanh function. These activation functions dominated the field for a very long time.