4 ms·

Local minima are only nearly as good as global minima in very very deep networks.

• deepnet 11y ago
  And yet Goodfellow et al's paper only uses 2 hidden layers and 1 softmax output layer (on MNIST) which is not so very deep but the effect is still present. (Qualitative Characterisation of Neural Net Optimisation ICML 2015) http://arxiv.org/abs/1412.6544v4 http://arxiv.org/abs/1412.6544v4 On p.6 Goodfellow et al. compare the neural nets to linear solutions by removing the non-linear layers, so that a single linear function is learned and find for Deep Linear Networks that : " All minima are global minima, and are linked to each other in a continuous manifold. " Which is pretty exciting because the received wisdom was local-minima are a really big problem.