Hinton nicely summarized why neural nets used to not work - https://www.youtube.com/watch?v=IcOMKXAw5VA&feature=youtu.be&t=21m29s https://www.youtube.com/watch?

Hinton nicely summarized why neural nets used to not work - https://www.youtube.com/watch?v=IcOMKXAw5VA&feature=youtu.be&t=21m29s https://www.youtube.com/watch?v=IcOMKXAw5VA&feature=youtu.be... Quoted: 1. Our labeled datasets were thousands of times too small. 2. Our computers were millions of times too slow. 3. We initialized the weights in a stupid way. 4. We used the wrong type of non-linearity. The pre-training helped with initialization, but later it turned out that just initializing the weights with correct scales for each layer (to deal with dissapearing/exploding gradient effect) worked almost as well with enough data.