3 ms·
At this point I'm going to point you to [1]. They interview Ilya Sutskever (ex Toronto, ex Google Brain, now director of OpenAI). He knows a little about neural
by nl 11y ago
At this point I'm going to point you to [1]. They interview Ilya Sutskever (ex Toronto, ex Google Brain, now director of OpenAI). He knows a little about neural networks[2] and possibly knows as much as anyone else about gradient descent.
In [1] he talks a lot about how there is no theoretical basis to think that a deep neural network should converge, and prior to around 2006 the accepted wisdom was that networks deep enough to outperform other methods of machine learning were useless because they couldn't be trained. Then they discovered that using larger random values for initialization means that they do converge, but that there is no theoretical basis to explain this at all.
[1] http://www.thetalkingmachines.com/blog/2015/1/15/machine-learning-and-magical-thinking http://www.thetalkingmachines.com/blog/2015/1/15/machine-lea...
[2] https://scholar.google.com.au/citations?user=x04W_mMAAAAJ&hl=en&oi=ao https://scholar.google.com.au/citations?user=x04W_mMAAAAJ&hl...
- Houshalter 11y agoI believe that has to do with the vanishing gradient issue which has been studied. As for why nets converge at all, there's this paper whichI believe tries to establish some theory why bigger nets don't have such a big problem with local optima: http://arxiv.org/abs/1412.0233 http://arxiv.org/abs/1412.0233
- nl 11y agoYes, exactly! Trying to understand why this happens is a very active area of research.