5 ms·
> ... all DNNs face the issue of overfitting as they learn, which is when performance on one data set increases but the network's performance fails to generaliz
by cs702 6y ago
> ... all DNNs face the issue of overfitting as they learn, which is when performance on one data set increases but the network's performance fails to generalize (often measured by the divergence of performance on training vs testing data sets). This ubiquitous problem in DNNs is often solved by experimenters via "noise injections" in the form of noisy or corrupted inputs. The goal of this paper is to argue that the brain faces a similar challenge of overfitting, and that nightly dreams evolved to combat the brain's overfitting during its daily learning.
Actually, there's compelling evidence that overfitting is a necessary step for achieving state-of-the-art performance with DNNs!
Many state-of-the-art deep learning models today are trained until they achieve ~100% accuracy on the training data, and then we continue to train them because once they go past this "interpolation threshold" they continue learning to generalize better to unseen data. This is known as the "double descent" phenomenon. See, for example:
https://openai.com/blog/deep-double-descent/ https://openai.com/blog/deep-double-descent/
https://arxiv.org/abs/1912.02292 https://arxiv.org/abs/1912.02292
https://arxiv.org/abs/1903.08560 https://arxiv.org/abs/1903.08560
The author makes no mention of double descent and the need for overfitting. In fact, he seems completely unaware of it.
--
EDIT: Also, see this comment https://news.ycombinator.com/item?id=23957501 https://news.ycombinator.com/item?id=23957501 elsewhere on this page.
- SoerenMind 6y agoThis paper argues that dreams are a form of data augmentation. Data augmentation is still useful even given double descent. Double descent is theoretically interesting but every DL practitioner will tell you that overfitting is still a problem.
- seesawtron 6y agoMost of the DL models we use are already above the "interpolation threshold" because they use millions of parameters which implies that they are already being trained in the "double descent" regime. Those papers and more [0] argue to explain why using millions of parameters still doesn't overfit our models that are trained on much less data (opposing the traditional Vapnik machine learning argument that your model paramters should not exceed number of data points otherwise you see overfit) because the models still manage to perform well on test data. In this paper, I think the authors are not focusing on overfitting on the data in the traditional Vapnik sense, but overfitting on unimportant information within that data. Noise injection and data augmentation are the techniques we use to de-correlate signal from noise in the data so that the networks can focus on signal and not on noise. Here sleep and dreams are argued to be that source of "noise" for our brain to decorrelate "signal" from the tasks we learn while awake. [0] https://arxiv.org/abs/1903.07571 https://arxiv.org/abs/1903.07571
- cs702 6y agoNot exactly. Those overparametrized models first overfit to the training data in the classical sense, and generalize worse to unseen data, and then, if we keep training past the interpolation threshold, the models start generalizing better. That's why the phenomenon is being called double descent. Injecting noise via dropout or mixup obscures this learning dynamic.
- blackbear_ 6y agoYou should be careful over-interpreting this stuff. Double descent is still poorly understood, and some [1] argue it is an artifact caused by wrongly assuming that the model complexity is a linear function of the number of parameters. I would also argue that "the need for overfitting" is a consequence of broken benchmarks, rather than a feature of deep learning. Why else would adversarial examples arise? [1] https://deepai.org/publication/rethinking-parameter-counting-in-deep-models-effective-dimensionality-revisited https://deepai.org/publication/rethinking-parameter-counting...
- jules 6y agoIsn't double descent explained by the following? The network contains many more parameters than data points. Therefore there is an entire region of lowest training error. A random point sonewhere in the middle of that region probably generalises better than at the edge. Once SGD enters this region at the edge, generalisation can still occur because the randomness will most likely cause it to random walk inside the region. To test this hypothesis you could run gradient descent with line search instead of SGD, and then you should not see this extra generalisation. Then if you add a bit of randomness to gradient descent you should see this extra generalisation again, if this hypothesis is correct. Also, under this hypothesis you'd predict that the speed at which generalisation improves depends on the batch size.
- Kinrany 6y agoMight be a stupid question, but can we skip the random walk and just pick the middle?
- pigscantfly 6y agoNot a stupid question at all, but one problem is that the boundaries of a zero-train-loss region are not well characterized and evaluating the validation loss even at a single point is computationally expensive. The centroid of one of these regions might not even be inside it (eg. donut shape but in higher dimensions) Interesting discussion though -- probably worth a few papers if someone were to investigate further.
- trott 6y ago> Actually, there's compelling evidence that overfitting is a necessary step for achieving state-of-the-art performance with DNNs! I think the term you are looking for is "overparametrization". Overfitting is, by definition, worse generalization performance despite better training performance.