4 ms·

> Right now, we are routinely creating neural networks that have a number of parameters more than the number of training samples. This says that the books have

> Right now, we are routinely creating neural networks that have a number of parameters more than the number of training samples. This says that the books have to be rewritten. Confused by this statement. Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books. > In their new proof, the pair show that overparameterization is necessary for a network to be robust. What is important to note here is that many of papers this paper cites prove or show this result in certain network architectures. This paper adds universality. > The proof is very elementary — no heavy math, and it says something very general The most elementary part was clever use of Hoeffding's inequality. Some people are really fast readers haha. I don't even know how you pick up the fact that isoperimetry holds in manifold settings with positive curvature while also playing with all those norms and inequalities. A few years ago I mentioned on here all the maths that I knew or wanted to know to read more papers, and others critiqued that the list was too long. Well, this is why!

aray 5y ago
> Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books. I’m curious for references or citations to this. When I was going over double descent I tried to find citations like this (just in a couple places like ML/stats textbooks).
- tomrod 5y ago
  There are a handful of papers in the 90s that show this, but it wasn't recognized for what it is. Double descent is REALLY crazy to me, coming from a classical background.
  - pishpash 5y ago
    Over-parameterization for regularization is really old. The pseudoinverse min-norm solution for under-determined linear systems even has that flavor.
    - tomrod 5y ago
      Sure, but that's identification approaches in econometrics and matrix analysis contexts. Using that for neural networks is new-ish in the zeitgeist, which did not exist in the 1990s as it does today.
- moyix 5y ago
  Here's one that lists some older references: https://arxiv.org/abs/2004.04328 https://arxiv.org/abs/2004.04328
throwmeawaysoon 5y ago
carbocation 5y ago
Daniela Witten has a nice twitter thread on the topic of double descent in classical settings: https://twitter.com/daniela_witten/status/1292293104855158784 https://twitter.com/daniela_witten/status/129229310485515878...
hikarudo 5y ago
Here's a very good visual explanation of double descent: https://mlu-explain.github.io/double-descent/ https://mlu-explain.github.io/double-descent/
- oxff 5y ago
  > ... "the model perfectly memorizes ... the training data." Can you call this learning?
jfmc 5y ago
> Right now, we are routinely creating neural networks that have a number of parameters more than the number of training samples. This says that the books have to be rewritten. I also believe that this statement is weird. I have a very shallow knowledge of ML, but I can imagine that in a convolutional neural network a training sample interacts with lots of parameters. This 'one training sample <-> one parameter' correspondence seems wrong to me.
- asfarley 5y ago
  One CNN parameter would interact with the total number of training samples, but that doesn’t tell you about the ratio of total training samples to total CNN parameters.

aray 5y ago

> Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books. I’m curious for references or citations to this. When I was going over double descent I tried to find citations like this (just in a couple places like ML/stats textbooks).

tomrod 5y ago
There are a handful of papers in the 90s that show this, but it wasn't recognized for what it is. Double descent is REALLY crazy to me, coming from a classical background.
- pishpash 5y ago
  Over-parameterization for regularization is really old. The pseudoinverse min-norm solution for under-determined linear systems even has that flavor.
  - tomrod 5y ago
    Sure, but that's identification approaches in econometrics and matrix analysis contexts. Using that for neural networks is new-ish in the zeitgeist, which did not exist in the 1990s as it does today.
moyix 5y ago
Here's one that lists some older references: https://arxiv.org/abs/2004.04328 https://arxiv.org/abs/2004.04328

throwmeawaysoon 5y ago

carbocation 5y ago

Daniela Witten has a nice twitter thread on the topic of double descent in classical settings: https://twitter.com/daniela_witten/status/1292293104855158784 https://twitter.com/daniela_witten/status/129229310485515878...

hikarudo 5y ago

Here's a very good visual explanation of double descent: https://mlu-explain.github.io/double-descent/ https://mlu-explain.github.io/double-descent/

oxff 5y ago
> ... "the model perfectly memorizes ... the training data." Can you call this learning?

jfmc 5y ago

> Right now, we are routinely creating neural networks that have a number of parameters more than the number of training samples. This says that the books have to be rewritten. I also believe that this statement is weird. I have a very shallow knowledge of ML, but I can imagine that in a convolutional neural network a training sample interacts with lots of parameters. This 'one training sample <-> one parameter' correspondence seems wrong to me.

asfarley 5y ago
One CNN parameter would interact with the total number of training samples, but that doesn’t tell you about the ratio of total training samples to total CNN parameters.