4 ms·
The linear regression is somewhat interesting, but also points out that the double descent might have a somewhat strange cause. What technically happens is that
by contravariant 4y ago
The linear regression is somewhat interesting, but also points out that the double descent might have a somewhat strange cause. What technically happens is that as you have more parameters you not only optimize for the model fit, but also for the size of the parameters.
If you force the model to fit your training data perfectly then it is no wonder that you can only start to optimize the size of your parameters after you have enough leeway to easily fit the entire training set.
However another way to achieve the same effect is to make the assumption that the parameters should be small(ish) explicit. Fitting the whole training set is nice but ultimately pointless if you're just fitting noise. If the actual data is a simple polynomial + random noise then fitting all the noise will give a worse estimate.
- gnramires 4y agoI think you're right, cross-validation should enable you to detect overfitting. When you see severe overfitting, you can try a different model (say a statistical model including noise), or add regularization hyperparameters, which you can tune via cross validation (or simply re-run the fit a number of times, because information contained in hyperparameters alone tends to be very low, hence not run risk of overfitting in the hyperparameter tuning process).