3 ms·
Have you seen the latest research on double descent? Here's a good intro, with references to some of the foundational work: https://openai.com/blog/deep-double-
by jacobbuckman 7y ago
Have you seen the latest research on double descent? Here's a good intro, with references to some of the foundational work: https://openai.com/blog/deep-double-descent/ https://openai.com/blog/deep-double-descent/
It seems bias-variance doesn't apply to neural networks at all! So your intuitions are good, but there's definitely more to the story.
- manthideaal 7y agoAbout the work you cite, I think that double descent is simply because the extra number of parameters (low bias) used produces a large variance when the input data is small, but as more data is introduced the extra parameters don't play any role, that is they are prunned. So the high bias is relative to the quantity of availabe information (training data). So the second descent starts when the extra parameters are prunned, in practice their coefficients tends to zero, the system learns that those coefficient don't play any role.