3 ms·
It's funny that you mention Relu. People have recently trained Imagenet networks using sigmoid/tanh (e.g. the activations that were used decades ago) on GPUs an
by argonaut 9y ago
It's funny that you mention Relu. People have recently trained Imagenet networks using sigmoid/tanh (e.g. the activations that were used decades ago) on GPUs and they train just fine. They train a bit slower is all. Not the breakthrough you're making it out to be. Relus were a very useful stop-gap in 2012 when GPUs weren't as fast.
- Eliezer 9y agoNow that we know how to initialize the weights so as to have the layer activations be something like sane, yes, we can use sigmoid/tanh. If you don't know modern clever ways of initializing weights then multi-layered sigmoid/tanh causes your activations and gradients to die out fast in deep networks, and ReLU is a godsend.