4 ms·
In practice, you don't want the global optimum because you can't put all possible inputs in the training data and need your system to "generalize" instead. Glob
by staunton 2y ago
In practice, you don't want the global optimum because you can't put all possible inputs in the training data and need your system to "generalize" instead. Global optimum would mean overfitting.
- WithinReason 2y agoRegularisation should not be done with the optimiser but with the loss function and the architecture.
- redox99 2y agoRegularization only helps you so much.
- staunton 2y agoMaybe it should not be done but the large neutral networks this decade absolutely rely on this. A network at the global minimum of any of the (regularized) loss functions that are used these days would be waaay overfitted.
- uoaei 2y agoThe entire reason SGD works is because the stochastic nature of updates on minibatches is an implicit regularizer. This one perspective built the foundations for all of modern machine learning. I completely agree that the most effective regularization is inductive bias in the architecture. But bang for buck, given all the memory/compute savings it accomplishes, SGD is the exemplar of implicit regularization techniques.
- badrunaway 2y agoCan someone explain this? Isn't it possible for the global optimum to be also be the right generalisation optimum?
- rcxdude 2y agoit's possible, but unlikely. The issue is your training examples are essentially a noisy representation of the general function you are trying to get it to learn. Generally any representation that fits too well will be incorporating the noise and that will distort the general function (in the case of NN it'll generally mean memorising the input data). Most function-fitting approaches are vulnerable to this.
- G3rn0ti 2y agoHm. I see. But, ultimately, overfitting is a consequence of too many parameters absorbing the noise. Perhaps one could fit smaller models and add artificial noise.
- nextaccountic 2y agoThe global optimum would be taken in reference to the training data (because that's all you have to set the weights). Unless the training data represents all real world data perfectly, fully optimizing for it will pessimize the model in relation to some set of real world data.