3 ms·
>For example, getting good performance on a dataset with deep learning also involves a lot of guesswork. But, if your training loss is way lower than your test
by noosphr 5mo ago
>For example, getting good performance on a dataset with deep learning also involves a lot of guesswork. But, if your training loss is way lower than your test loss, you're in the "overfitting" regime, and you're wasting your time if you try to increase the capacity of your model.
https://arxiv.org/abs/1912.02292 https://arxiv.org/abs/1912.02292
- appplication 5mo agoGenerally, posting a link-only reply without further elaboration comes across as a bit rude. Are you providing support for the above point? Refuting it? You felt compelled to comment, a few words to indicate what you’re actually trying to say would go a long way.
- noosphr 5mo ago>We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better.
- ForceBru 5mo agoRight, isn't double descent one of the reasons why modern Extremely Large Language Models work at all? I think I heard somewhere that basically all today's "smart" (reasoning, solving math problems, etc) LLMs are trained in the "double descent" territory (whatever this means, I'm not entirely sure).
- SiempreViernes 5mo agoNo, double descent is a symptom of whatever it is that makes the deep models work at all. It's just the name for something you see happen when it works. The reason it works has something to do with how all those extra dimensions work as a regularisation term in the fit.
- mxwsn 5mo agoNo, there are more training tokens than parameters in LLMs. They are in the classical first descent setting.
- smallerize 5mo agoDoes this mean that if your model is "overfitting", the solution is to train for even more epochs?
- noosphr 5mo agoMaybe. Just means that the conventional wisdom was wrong and substantially over training can be a good thing. No one I knew at the time suspected that, including the people who wrote the paper.