3 ms·
Hence, the interesting question is why deep learning models don't overfit all the time.
by disgruntledphd2 4y ago
Hence, the interesting question is why deep learning models don't overfit all the time.
- DougBTX 4y agoThis is from 2019 so I suspect there’s a more recent paper now, but this is interesting: https://openai.com/blog/deep-double-descent/ https://openai.com/blog/deep-double-descent/
- disgruntledphd2 4y agoAgreed, but it doesn't really explain it in any real sense.
- mr_toad 4y ago“However, in the over-parameterized regime, there are many models that fit the train set and there exist such good models.” One explanation is that a very large model effectively fits several models and penalises over-fit models itself.
- skinner_ 4y agoWork is ongoing, this is a whole field of research now, a good keyword to search for is "implicit regularization".
- marcosdumay 4y agoI really like how AI research is all about empirical analysis of mathematics.
- gnramires 4y agoThe answer is really very simple: because we use training and test sets, and generally use iterative training methods. As soon as test set error plateaus or starts to degrade, you stop training. This is called early stopping and can be considered a form of regularization. In theory it would seem a neural network (with early stopping "correctly" implemented) can never overfit, although I suppose in practice it can vary a little (because of test variance, too large step sizes, etc.). It's possible to train huge overparametrized networks this way. One big problem I can see with this approach (maybe compared to over forms of regularization) is that it might be very space/memory inefficient. But if you've got a big budget I think it's a surefire way to make sure your model performs as well as possible (without having to tune hyperparameters too much or deal with local minima that probably happens at near-capacity networks).