Most of the DL models we use are already above the "interpolation threshold" because they use millions of parameters which implies that they are already being t

Most of the DL models we use are already above the "interpolation threshold" because they use millions of parameters which implies that they are already being trained in the "double descent" regime. Those papers and more [0] argue to explain why using millions of parameters still doesn't overfit our models that are trained on much less data (opposing the traditional Vapnik machine learning argument that your model paramters should not exceed number of data points otherwise you see overfit) because the models still manage to perform well on test data. In this paper, I think the authors are not focusing on overfitting on the data in the traditional Vapnik sense, but overfitting on unimportant information within that data. Noise injection and data augmentation are the techniques we use to de-correlate signal from noise in the data so that the networks can focus on signal and not on noise. Here sleep and dreams are argued to be that source of "noise" for our brain to decorrelate "signal" from the tasks we learn while awake. [0] https://arxiv.org/abs/1903.07571 https://arxiv.org/abs/1903.07571