3 ms·
As a counterpoint, the Blessing of Dimensionality is that local minima are mostly as good as the global minima. In David Silver's RLDM 2015 talk @20 mins in h
by deepnet 11y ago
As a counterpoint, the Blessing of Dimensionality is that local minima are mostly as good as the global minima.
In David Silver's RLDM 2015 talk @20 mins in
http://videolectures.net/rldm2015_silver_reinforcement_learning/ http://videolectures.net/rldm2015_silver_reinforcement_learn...
he recounts this as his experience.
the paper The Loss Surfaces of Multilayer Networks goes in depth
http://arxiv.org/abs/1412.0233 http://arxiv.org/abs/1412.0233
[edit] and the paper
Qualitative Characterisation of Neural Net Optimisation
http://arxiv.org/abs/1412.6544v4 http://arxiv.org/abs/1412.6544v4
- Fede_V 11y agoLocal minima are only nearly as good as global minima in very very deep networks.
- deepnet 11y agoAnd yet Goodfellow et al's paper only uses 2 hidden layers and 1 softmax output layer (on MNIST) which is not so very deep but the effect is still present. (Qualitative Characterisation of Neural Net Optimisation ICML 2015) http://arxiv.org/abs/1412.6544v4 http://arxiv.org/abs/1412.6544v4 On p.6 Goodfellow et al. compare the neural nets to linear solutions by removing the non-linear layers, so that a single linear function is learned and find for Deep Linear Networks that : " All minima are global minima, and are linked to each other in a continuous manifold. " Which is pretty exciting because the received wisdom was local-minima are a really big problem.