5 ms·
Understanding “Deep Double Descent”
- dzdt 7y agoThis is a really good read! I am not deep in machine learning research but the exposition and diagrams make the point clearly. I really feel like this us advancing deep learning as a science.
- eutectic 7y agoWhat happens if you take a large model and regularize it to have the same training error as a smaller model? Do you get the same benefits?
- jackson1372 7y agoThe reason you want to over-parameterize your model is that it protects you from "bad bounce" learning trajectories. You effectively spread out your overfitting risk until it's pretty close to 0. Or at least that's the way I like to think of it. The next step is to better compress the resulting model in a simpler, less computationally costly network.
- deleted 7y ago[deleted]
- gyuserbti 7y agoAre you suggesting dd is about local minima sort of? Like if you extended the risk: parametrization curve out you'd start to see overfitting again?
- _0ffh 7y agoWow, that implies that given a large enough model, early stopping is actually a mistake! Who’d a thunk it?
- gwern 7y agoBut maybe it's not a big one (the reduction on the other side is still fairly small), and it may not be one at all (do you have the compute budget? look at how many epoches it takes to reach the other side and actually realize any gains!). More theoretically interesting than practical advice, I'd say.
- _0ffh 7y agoYup, I've seen on the graph it looks like at least an order of magnitude more training time until it starts to get interesting. Obviously this is only one example, so that factor might vary mightily. It might be an option for someone who is really serious about squeezing out the last drop of generalisation performance. I guess we'll see how things work out, because some folk are bound to try.
- liaukovv 7y ago"Understanding" part seems rather optimistic.
- lostmsu 7y agoCan someone explain what is "interpolation threshold"? Both articles talk about it, but none defines.
- deleted 7y ago[deleted]
- preetum 7y agoHi, we define the interpolation threshold in Section 2 of the full paper (https://arxiv.org/abs/1912.02292 https://arxiv.org/abs/1912.02292) as the point when the "Effective Model Complexity" = # of train samples. Where the "Effective Model Complexity (EMC)" of a model + training procedure (w.r.t an input distribution) is the maximum number of samples from the distribution that the model+training can fit to ~0 train error. Our experiments are consistent with the hypothesis that the double-descent peak occurs when EMC = n; that is, when the model+training is just barely able to fit the train set.
- joe_the_user 7y agoIt seems like a phenomena of this sort would have to depend on the data set you are dealing with. It might be true for "all typical data sets in important domains" but it would still seem like basic point would hold.
- preetum 7y agoOh yes, absolutely. We only make claims for "natural distributions and models" (^). It's almost certainly possible to break this by pathological choice of a data distribution or model/initialization/optimization scheme. But, I don't think this is interesting -- what I think is interesting is that this seems to hold true in real life, in "natural settings". --- (^) Whatever "natural distributions" means... (which I think is a good research direction in itself).
- jackson1372 7y agoIsn't the explanation for this that the world actually does work in some way or other and that it's not just infinite chaos and so if you keep throwing parameters at some problem, you will eventually stumble upon the "real" structure, but that's no guarantee of when that occurs, and with which parameters?
- ganzuul 7y agoThis observation seems to come up in discussions regarding the hot topic of the https://en.wikipedia.org/wiki/Neural_tangent_kernel https://en.wikipedia.org/wiki/Neural_tangent_kernel Please correct me if I'm wrong, but I think it means to say that you can in theory conjure specialized kernel methods out of 'infinitely' over-parametrized neural networks. At the moment this all gives unimpressive performance, but it is theoretically promising and could give statisticians interpretable NN-derived models.
- gyuserbti 7y agoMy initial intuition is there's limitations in test samples that are used, in the sense they only have so much information. At some point overfitting is likely to manifest not in test risk per se, but in random variations over alternate test samples. Eg overfitting would evidence in susceptibility to adversarial regimes not cross validation risk. I've always been skeptical of cross validation based inference though and admit it's a fascinating phenomenon in the paper. It just seems, informationally speaking, to be proposing something akin to free energy: that more data is worse and if you just increase your model complexity you can magically infer truth. It seems more likely to be an error in the inferential paradigm.