4 ms·

I get so pissed every time the double descent paper gets brought up, maybe because of the hubris in its abstract. No "questions about the mathematical foundatio

I get so pissed every time the double descent paper gets brought up, maybe because of the hubris in its abstract. No "questions about the mathematical foundation of machine learning" were raised by deep learning and a few silly experiments are far from showing "limits of classical analyses". All the paper shows is (1) stupid ways of counting model complexity and (2) that gradient descent is flawed. Nobody in their right mind believes that increasing the number of hidden neurons can result in a network that is worse on the test set. Since the bigger network contains the smaller network, it is perfectly capable of achieving the same performance, so the only reason why this does not happen is that SGD cannot find it. But of course "SGD cannot always find good solutions" is surprising to nobody, so let's just shit on decades of serious work to get our little paper out. Sorry for the rant.

mjburgess 6y ago
The interesting contribution was forcing interpolation on the training set, right? Its one of those papers where if you had a real handle on how everything works, wasnt surprising. But it was more a rebuke on some cargo cult ideas. Lots of impactful papers have that form, eg., Gettier's paper on justified true belief.
unishark 6y ago
I'm not clear why you are blaming SGD. Maybe I missed the point. In principle SDG might well find the global optimal. The problem is this optimal is achieved only for training data. It could certainly perform worse on the test set. Maybe you are referring to the entire training process? The general idea is from the days of SVM's where the optimization method was convex. Though personally I do find a lot this modern "experimental" research quite hokey. I don't think this is something academics should be getting research funding to pursue. This is engineers building intuition about how to tune their product.
- GregarianChild 6y ago
  In practise, how do you know if SDG has converged to a global optimum, and do you let it run long enough in practise until covergence?
  - unishark 6y ago
    You don't know. The point was even if you did get there it could still be an overfit model you don't want, since it's based on a training data set, not the true statistics of the distribution the samples come from.
PartiallyTyped 6y ago
Let's for a second assume that we bound the weights within [-1,1], considering that we are using floats to represent them which are a subset of real numbers, by increasing the number of hidden neurons, you increase the class of models that you can select from, any single bit change in the weights means a different model. By increasing the class of models in your search space, you increase the number of models that perform good on training, but perform worse on test, i.e. underestimate true loss. Exactly because you select a single model and you have more models that underestimate the true (read test) error, you *may* be more likely to get models with worse performance on the test set compared to using a smaller network exactly because in the smaller network you are searching within a smaller distribution.
Nimitz14 6y ago
I really do not understand your post. One doesn't train on the test set, so of course it's reasonable to think that increasing the number of parameters will cause more overfitting.
- blackbear_ 6y ago
  And yet that is not what is observed in practice. See figures 1 and 2 of that paper [1]. What I am complaining about is that the authors are confusing "what can be represented" by a neural network with "what can be easily learned" via SGD. I argued that the peak in test loss in those figures is observed becuse SGD struggles to find a solution that generalizes well, not because those networks are intrinsically less powerful (as the paper seems to imply). [1] https://arxiv.org/pdf/1812.11118.pdf https://arxiv.org/pdf/1812.11118.pdf
  - Nimitz14 6y ago
    > And yet that is not what is observed in practice. See figures 1 and 2 of that paper Yeah... that's why the paper is a good contribution contrary to what you're saying. Not sure why you're repeating this information. > What I am complaining about is that the authors are confusing "what can be represented" by a neural network with "what can be easily learned" via SGD. I argued that the peak in test loss in those figures is observed becuse SGD struggles to find a solution that generalizes well, not because those networks are intrinsically less powerful (as the paper seems to imply). I mean.. those models are by definition less powerful as they have less parameters. The (to me) main point of the paper is to point out a symptom which is interesting. Their explanation for the symptom being (maybe) wrong doesn't detract from the important work of showing the symptom exists. A legitimate criticism would be that there have been earlier papers showing the same thing. Sidenote: It seems having a larger model does make it easier for SGD to find good solutions[1] [1] https://www.youtube.com/watch?v=kcVWAKf7UAg https://www.youtube.com/watch?v=kcVWAKf7UAg
eugenhotaj 6y ago
> Since the bigger network contains the smaller network, it is perfectly capable of achieving the same performance, so the only reason why this does not happen is that SGD cannot find it. This is maybe true in the limit of infinite data, but not true in any practical sense, and I don't think it has anything to do with SGD. E.g. polynomial basis functions also have this property, but you can't use an arbitrarily large polynomial order or you'll eventually overfit. You can get a closed-form solution for polynomial regression problems, so no SGD involved.
moultano 6y ago
The fact that over parametrized models can generalize better than under parameterized models, and that both are better than models that can just barely interpolate is a genuinely new insight that was not predicted by any prior theory.

mjburgess 6y ago

The interesting contribution was forcing interpolation on the training set, right? Its one of those papers where if you had a real handle on how everything works, wasnt surprising. But it was more a rebuke on some cargo cult ideas. Lots of impactful papers have that form, eg., Gettier's paper on justified true belief.

unishark 6y ago

I'm not clear why you are blaming SGD. Maybe I missed the point. In principle SDG might well find the global optimal. The problem is this optimal is achieved only for training data. It could certainly perform worse on the test set. Maybe you are referring to the entire training process? The general idea is from the days of SVM's where the optimization method was convex. Though personally I do find a lot this modern "experimental" research quite hokey. I don't think this is something academics should be getting research funding to pursue. This is engineers building intuition about how to tune their product.

GregarianChild 6y ago
In practise, how do you know if SDG has converged to a global optimum, and do you let it run long enough in practise until covergence?
- unishark 6y ago
  You don't know. The point was even if you did get there it could still be an overfit model you don't want, since it's based on a training data set, not the true statistics of the distribution the samples come from.

PartiallyTyped 6y ago

Let's for a second assume that we bound the weights within [-1,1], considering that we are using floats to represent them which are a subset of real numbers, by increasing the number of hidden neurons, you increase the class of models that you can select from, any single bit change in the weights means a different model. By increasing the class of models in your search space, you increase the number of models that perform good on training, but perform worse on test, i.e. underestimate true loss. Exactly because you select a single model and you have more models that underestimate the true (read test) error, you *may* be more likely to get models with worse performance on the test set compared to using a smaller network exactly because in the smaller network you are searching within a smaller distribution.

Nimitz14 6y ago

I really do not understand your post. One doesn't train on the test set, so of course it's reasonable to think that increasing the number of parameters will cause more overfitting.

blackbear_ 6y ago
And yet that is not what is observed in practice. See figures 1 and 2 of that paper [1]. What I am complaining about is that the authors are confusing "what can be represented" by a neural network with "what can be easily learned" via SGD. I argued that the peak in test loss in those figures is observed becuse SGD struggles to find a solution that generalizes well, not because those networks are intrinsically less powerful (as the paper seems to imply). [1] https://arxiv.org/pdf/1812.11118.pdf https://arxiv.org/pdf/1812.11118.pdf
- Nimitz14 6y ago
  > And yet that is not what is observed in practice. See figures 1 and 2 of that paper Yeah... that's why the paper is a good contribution contrary to what you're saying. Not sure why you're repeating this information. > What I am complaining about is that the authors are confusing "what can be represented" by a neural network with "what can be easily learned" via SGD. I argued that the peak in test loss in those figures is observed becuse SGD struggles to find a solution that generalizes well, not because those networks are intrinsically less powerful (as the paper seems to imply). I mean.. those models are by definition less powerful as they have less parameters. The (to me) main point of the paper is to point out a symptom which is interesting. Their explanation for the symptom being (maybe) wrong doesn't detract from the important work of showing the symptom exists. A legitimate criticism would be that there have been earlier papers showing the same thing. Sidenote: It seems having a larger model does make it easier for SGD to find good solutions[1] [1] https://www.youtube.com/watch?v=kcVWAKf7UAg https://www.youtube.com/watch?v=kcVWAKf7UAg

eugenhotaj 6y ago

> Since the bigger network contains the smaller network, it is perfectly capable of achieving the same performance, so the only reason why this does not happen is that SGD cannot find it. This is maybe true in the limit of infinite data, but not true in any practical sense, and I don't think it has anything to do with SGD. E.g. polynomial basis functions also have this property, but you can't use an arbitrarily large polynomial order or you'll eventually overfit. You can get a closed-form solution for polynomial regression problems, so no SGD involved.

moultano 6y ago

The fact that over parametrized models can generalize better than under parameterized models, and that both are better than models that can just barely interpolate is a genuinely new insight that was not predicted by any prior theory.