4 ms·
> Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books. I’m curious for references or citations to th
by aray 5y ago
> Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books.
I’m curious for references or citations to this. When I was going over double descent I tried to find citations like this (just in a couple places like ML/stats textbooks).
- tomrod 5y agoThere are a handful of papers in the 90s that show this, but it wasn't recognized for what it is. Double descent is REALLY crazy to me, coming from a classical background.
- pishpash 5y agoOver-parameterization for regularization is really old. The pseudoinverse min-norm solution for under-determined linear systems even has that flavor.
- tomrod 5y agoSure, but that's identification approaches in econometrics and matrix analysis contexts. Using that for neural networks is new-ish in the zeitgeist, which did not exist in the 1990s as it does today.
- moyix 5y agoHere's one that lists some older references: https://arxiv.org/abs/2004.04328 https://arxiv.org/abs/2004.04328