It's not just ReLUs that have helped, its also better random initialization before starting training, such as using Xavier initialization (http://andyljones.tum

It's not just ReLUs that have helped, its also better random initialization before starting training, such as using Xavier initialization (http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization http://andyljones.tumblr.com/post/110998971763/an-explanatio...) Also, batch normalization helps with convergence as well. In addition, LSTMs work when dealing with recurrent neural nets.