3 ms·
This is only tangentially my field, so pure speculation. I suppose it's possible that generalized minima are numerically more common than overfitted minima in
by benlivengood 6y ago
This is only tangentially my field, so pure speculation.
I suppose it's possible that generalized minima are numerically more common than overfitted minima in an over-parameterized model, so probabilistically SGD will find a more general minima than not, regardless of regularization.
- hervature 6y agoI think the general consensus (from my interactions) is that a local minima requires the gradient to vanish. When you have many dimensions, it's unlikely that they are all 0. Coupled with modern optimization methods (primarily momentum), this encourages the result to be in a shallow valley as opposed to a spiky minima. The leap of faith is equating shallow=general and spiky=overfitted.