7 ms·
I think you're right, but the issue might be local minima which a better optimiser wouldn't help with much. A reason a larger network might work better is that
by WithinReason 2y ago
I think you're right, but the issue might be local minima which a better optimiser wouldn't help with much. A reason a larger network might work better is that there are fewer local minima in a higher dimension too.
- jxy 2y ago> there are fewer local minima in a higher dimension Is it actually proven, or another hypothesis? What is the reason behind this?
- jcrites 2y agoJust reasoning about this from first principles, but intuitively, the more dimensions you have, the more likely that you are to find a gradient in some dimension. In an N-dimensional space, a local minimum needs to be a minimum in all N dimensions, right? Otherwise the algorithm will keep exploring down the gradient. (Not an expert on this stuff.) The more dimensions there are, the more likely it seems to be that a gradient exists down to some greater minimum from any given point.