3 ms·
It doesn't depend on who you read. ReLUs (and their variants) are virtually universal now. LeCun advocated tanh but that was years and years ago before ReLUs ca
by argonaut 11y ago
It doesn't depend on who you read. ReLUs (and their variants) are virtually universal now. LeCun advocated tanh but that was years and years ago before ReLUs came out (they came out in 2011) and he advocated ReLUs as early as three years ago (http://www.cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf http://www.cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf). Empirically they give better results and are faster.
(I say "almost" because while I haven't seen a single state of the art result without ReLUs, I'm sure there is probably a random paper out there)
- tgflynn 11y agoAs far as I can tell LSTM's are still using a combination of logistic and tanh transfer functions, at least through 2014 and some of the most impressive deep learning results have been obtained with these. For example: http://arxiv.org/abs/1308.0850 http://arxiv.org/abs/1308.0850