3 ms·
Is the claim that there aren’t local many local minima for high dimensional problems eg in neural network loss functions?
by jonathan_landy 2y ago
Is the claim that there aren’t local many local minima for high dimensional problems eg in neural network loss functions?
- Imnimo 2y agoYes. To be more specific, it's that nearly all points where the derivative is zero are saddle points rather than minima. Note that some portion of this nice behavior seems to be due to design choices in modern architectures, like residual connections, rather than being a general fact about all high dimensional problems. https://arxiv.org/pdf/1712.09913 https://arxiv.org/pdf/1712.09913 This paper has some nice visualizations.