4 ms·
It depends on who you read. LeCun certainly advocated tanh for a long time. ReLU seems to be currently favored, especially for deep nets but I'm not sure it's
by tgflynn 11y ago
It depends on who you read. LeCun certainly advocated tanh for a long time. ReLU seems to be currently favored, especially for deep nets but I'm not sure it's being used universally. In my own experiments I am seeing it train faster than logistic, but not necessarily give better results (maybe because you don't want the net to train too fast to reach the best optimum).
His overall point that the field is largely empirical with little solid theory to guide one's choices is, in my opinion, not wrong. It may be that a general theory doesn't exist and different network architectures and hyperparameters are needed for different types and sizes of problems.
- argonaut 11y agoIt doesn't depend on who you read. ReLUs (and their variants) are virtually universal now. LeCun advocated tanh but that was years and years ago before ReLUs came out (they came out in 2011) and he advocated ReLUs as early as three years ago (http://www.cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf http://www.cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf). Empirically they give better results and are faster. (I say "almost" because while I haven't seen a single state of the art result without ReLUs, I'm sure there is probably a random paper out there)
- tgflynn 11y agoAs far as I can tell LSTM's are still using a combination of logistic and tanh transfer functions, at least through 2014 and some of the most impressive deep learning results have been obtained with these. For example: http://arxiv.org/abs/1308.0850 http://arxiv.org/abs/1308.0850