3 ms·

You should be careful over-interpreting this stuff. Double descent is still poorly understood, and some [1] argue it is an artifact caused by wrongly assuming t

You should be careful over-interpreting this stuff. Double descent is still poorly understood, and some [1] argue it is an artifact caused by wrongly assuming that the model complexity is a linear function of the number of parameters. I would also argue that "the need for overfitting" is a consequence of broken benchmarks, rather than a feature of deep learning. Why else would adversarial examples arise? [1] https://deepai.org/publication/rethinking-parameter-counting-in-deep-models-effective-dimensionality-revisited https://deepai.org/publication/rethinking-parameter-counting...

jules 6y ago
Isn't double descent explained by the following? The network contains many more parameters than data points. Therefore there is an entire region of lowest training error. A random point sonewhere in the middle of that region probably generalises better than at the edge. Once SGD enters this region at the edge, generalisation can still occur because the randomness will most likely cause it to random walk inside the region. To test this hypothesis you could run gradient descent with line search instead of SGD, and then you should not see this extra generalisation. Then if you add a bit of randomness to gradient descent you should see this extra generalisation again, if this hypothesis is correct. Also, under this hypothesis you'd predict that the speed at which generalisation improves depends on the batch size.
- Kinrany 6y ago
  Might be a stupid question, but can we skip the random walk and just pick the middle?
  - pigscantfly 6y ago
    Not a stupid question at all, but one problem is that the boundaries of a zero-train-loss region are not well characterized and evaluating the validation loss even at a single point is computationally expensive. The centroid of one of these regions might not even be inside it (eg. donut shape but in higher dimensions) Interesting discussion though -- probably worth a few papers if someone were to investigate further.
    - chillee 6y ago
      This just sounds like Stochastic Weight Averaging, which works quite well: https://arxiv.org/abs/1803.05407 https://arxiv.org/abs/1803.05407

jules 6y ago

Isn't double descent explained by the following? The network contains many more parameters than data points. Therefore there is an entire region of lowest training error. A random point sonewhere in the middle of that region probably generalises better than at the edge. Once SGD enters this region at the edge, generalisation can still occur because the randomness will most likely cause it to random walk inside the region. To test this hypothesis you could run gradient descent with line search instead of SGD, and then you should not see this extra generalisation. Then if you add a bit of randomness to gradient descent you should see this extra generalisation again, if this hypothesis is correct. Also, under this hypothesis you'd predict that the speed at which generalisation improves depends on the batch size.

Kinrany 6y ago
Might be a stupid question, but can we skip the random walk and just pick the middle?
- pigscantfly 6y ago
  Not a stupid question at all, but one problem is that the boundaries of a zero-train-loss region are not well characterized and evaluating the validation loss even at a single point is computationally expensive. The centroid of one of these regions might not even be inside it (eg. donut shape but in higher dimensions) Interesting discussion though -- probably worth a few papers if someone were to investigate further.
  - chillee 6y ago
    This just sounds like Stochastic Weight Averaging, which works quite well: https://arxiv.org/abs/1803.05407 https://arxiv.org/abs/1803.05407