3 ms·

The paper “Understanding Deep Learning Requires Rethinking Generalization” was where this was first pointed out, I think. They shuffled the labels on their dat

The paper “Understanding Deep Learning Requires Rethinking Generalization” was where this was first pointed out, I think. They shuffled the labels on their datasets, so there can’t possibly be anything to learn, yet got zero training loss, meaning the network must be severely overfitting. Yet the same network trained with the actual labels shows quite good generalization. So the usual intuition about overfitting and the bias-variance tradeoff doesn’t seem to apply.

elcomet 8y ago
This seems quite intuitive to me: When you have nothing to learn, you need to memorize the data. But when there is structure, it is easier to memorize the structure, so the network will learn this first (and will memorize after).
- srean 8y ago
  But thats the crux of the question: why and how does it not just memorize when we know it can do so easily ?
  - elcomet 8y ago
    Maybe just because it's easier to find patterns than to memorize (if you have a lot of data).
    - currymj 8y ago
      That sounds like it’s probably right to me. But so do lots of things that turn out to be wrong. I wish we had a better grasp of what is happening, not just plausible stories. I’m already sick of doing alchemical tinkering to find a model that works.
      - elcomet 8y ago
        Yeah, I agree it would be nice to have some theoretical guarantees on the architecture we need based on the problem, and the size of the data
charleshmartin 8y ago
I would say “Understanding Deep Learning Requires REMEMBERING Generalization” https://calculatedcontent.com/2018/04/01/rethinking-or-remembering-generalization-in-neural-networks/ https://calculatedcontent.com/2018/04/01/rethinking-or-remem... https://arxiv.org/abs/1710.09553 https://arxiv.org/abs/1710.09553 We can understand this using the traditional theory of Statistical Mechanics of Generalization Briefly, shuffling the labels corresponds to decreasing the effective load on the Neural Network, which pushes the system into the spin glass phase

elcomet 8y ago

This seems quite intuitive to me: When you have nothing to learn, you need to memorize the data. But when there is structure, it is easier to memorize the structure, so the network will learn this first (and will memorize after).

srean 8y ago
But thats the crux of the question: why and how does it not just memorize when we know it can do so easily ?
- elcomet 8y ago
  Maybe just because it's easier to find patterns than to memorize (if you have a lot of data).
  - currymj 8y ago
    That sounds like it’s probably right to me. But so do lots of things that turn out to be wrong. I wish we had a better grasp of what is happening, not just plausible stories. I’m already sick of doing alchemical tinkering to find a model that works.
    - elcomet 8y ago
      Yeah, I agree it would be nice to have some theoretical guarantees on the architecture we need based on the problem, and the size of the data

charleshmartin 8y ago

I would say “Understanding Deep Learning Requires REMEMBERING Generalization” https://calculatedcontent.com/2018/04/01/rethinking-or-remembering-generalization-in-neural-networks/ https://calculatedcontent.com/2018/04/01/rethinking-or-remem... https://arxiv.org/abs/1710.09553 https://arxiv.org/abs/1710.09553 We can understand this using the traditional theory of Statistical Mechanics of Generalization Briefly, shuffling the labels corresponds to decreasing the effective load on the Neural Network, which pushes the system into the spin glass phase