7 ms·

Understanding deep learning requires rethinking generalization

165 points by tmfi 6y ago

clircle 6y ago
> Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. I'd say it's more about the simplicity of the task and quality of the data.
deleted 6y ago
[deleted]
magicalhippo 6y ago
The experiments we conducted emphasize that the effective capacity of several successful neural network architectures is large enough to shatter the training data. Consequently, these models are in principle rich enough to memorize the training data. So they're fitting elephants[1]. I've been trying to use DeepSpeech[2] lately for a project, would be interesting to see the results for that. I guess it could also be a decent test for your model? Retrain it with random labels and if it succeeds the model is just memorizing, so either reduce model complexity or add more training data? [1]: https://www.johndcook.com/blog/2011/06/21/how-to-fit-an-elephant/ https://www.johndcook.com/blog/2011/06/21/how-to-fit-an-elep... [2]: https://github.com/mozilla/DeepSpeech https://github.com/mozilla/DeepSpeech
- mxwsn 6y ago
  Large model capacity enough to perfectly memorize/interpolate data may not be a bad thing. A phenomenon known as "deep double descent" says that increasing modeling capacity relative to the dataset size can reduce generalization error, even after the model achieves perfect training performance (see work by Mikhail Belkin [0] and empirical demonstrations on large deep learning tasks by researchers from Harvard/OpenAI [1]). Other work argues that memorization is critical to good performance on real-world tasks where the data distribution is often long-tailed [2]: to perform well at tasks where the training data set only has 1 or 2 examples, it's best to memorize those labels, rather than extrapolate from other data (which a lower capacity model may prefer to do). [0]: https://arxiv.org/abs/1812.11118 https://arxiv.org/abs/1812.11118 [1]: https://openai.com/blog/deep-double-descent/ https://openai.com/blog/deep-double-descent/ [2]: https://arxiv.org/abs/2008.03703 https://arxiv.org/abs/2008.03703
  - derbOac 6y ago
    I think there's something critical about the implicit data universe being considered in the test and training data, and in these randomized datasets. Memorizing elephants isn't necessarily a bad thing if you can be assured there are actually elephants in the data, or if your job is to reproduce some data that has some highly non-random, low entropy (in an abstract sense) features. I think where the phenomenon in this paper, and deep double descent starts to clash with intuition, is the more realistic case where the adversarial data universe is structured, not random, but not conforming to the observed training target label alphabet (to borrow a term loosely from the IT literature). That is, it's interesting to know that these models can perfectly reproduce random data, but generalizing from training to test data isn't interesting in a real-world sense if both are constrained by some implicit features of the data universe involved in the modeling process (e.g., that the non-elephant data only differs randomly from the elephant data, or doesn't contain any nonelephants that aren't represented in the training data). So then you end up with this: https://www.theverge.com/2020/9/20/21447998/twitter-photo-preview-white-black-faces https://www.theverge.com/2020/9/20/21447998/twitter-photo-pr... I guess it seems to me there's a lot of implicit assumptions about the data space and what's actually being inferred in a lot of these DL models. The insight about SGD is useful, but maybe only underscores certain things, and seems to get lost in some of the discussion about DDD. Even Rademacher complexity isn't taken with regard to the entire dataspace, just over a uniformly random sampling of it -- so it will underrepresent certain corners of the data space that are highly nonuniform, low entropy, which is exactly where the trouble lies. There's lots of fascinating stuff in this area of research, lots to say, glad to see it here on HN again.
  - medlyyy 6y ago
    Thanks for your [2] link paper, it's an interesting one. If you think about it, the high-level idea makes some intuitive sense. Since NNs are known to be "equivalent" to kernel methods - which are, to oversimplify, essentially nearest-neighbour + a similarity function - then the ability of NNs to memorise specific training examples can be analogised to adding another "neighbour" to interpolate from. So maybe it's not too surprising that NNs which can do this can have better generalisation performance. (Although it is still surprising, since I certainly wouldn't predict it a priori!) Really, what is the difference between a "feature" and "memorising a data point"? It seems only a matter of scale - how much of the input space is "relevant" to the learned representation.
- caddemon 6y ago
  Would that necessarily imply it is memorizing on the non-random labels though? I know analogies to human learning are overdone, but I definitely have seen humans fall back on memorization when they are struggling to "actually learn" some topic. So genuinely asking from a technical/ML perspective - is it possible a network could optimize training loss without memorization when possible, but as that fails end up just memorizing?
  - hervature 6y ago
    > is it possible a network could optimize training loss without memorization when possible What does that even mean? ML is simply a set of tools to minimize (f(x_i) - y_i)^2 (if you don't like squared loss, pick whatever you prefer). In particular, f() here is our neural network. There is no loss without previous data. The only thing the network is trying to do is "memorize" old data.
    - caddemon 6y ago
      It depends of course how you are defining memorization, but the network doesn't necessarily need to use the entirety of every input to do what you are describing. I would think what people mean when they say "it isn't learning, just memorizing" is that the vast majority of information about all previous inputs is being directly encoded in the network. The person I was responding to mentioned training on random labels, and if training still goes well the network must be a memorizer. But I don't see why it couldn't be the case that a network is able to act as a memorizer, but doesn't if there are certain patterns it can generalize on in the training data. Also, there is no human learning without previous data either, but I wouldn't characterize all of human learning as memorization.
      - dumb1224 6y ago
        I don't understand the random label training part. Presumably you train on randomised labels which have no relationship with the input but surely it won't generalise well at all given the small probability of predicting the labels correctly by chance (The setup for a Permutation test am I wrong)?
- bloaf 6y ago
  That's not my reading. I think they are saying that models which *can* over-fit the data in both theory and practice, appear not do so when there are in fact generalizations in the data.
  - magicalhippo 6y ago
    Ah hmm yes, good point. I forgot this piece from the article: We observe a steady deterioration of the generalization error as we increase the noise level. This shows that neural networks are able to capture the remaining signal in the data while at the same time fit the noisy part using brute-force. Since they score well on the test data, they must have generalized to some degree. But since they're just as good at training on random input they also have the capacity to just memorize the training data.
- trowngon 6y ago
  Irrespective of the subject Deepspeech is very old archtecture with suboptimal results. You'd better try any recent conformer implementations (flashlight, nemo, wenet, etc) or wav2vec.
  - magicalhippo 6y ago
    I ended up with DeepSpeech since it was very easy to get started with, and it has support for fairly low-latency inferencing which is very important for my project. I will take a look at the ones you suggested though, thanks for the heads-up!
blt 6y ago
Please add the (still) to the HN post title. The original version of the paper without (still) in the title is several years old.
- davnn 6y ago
  2016 for v1 to be exact, link: https://arxiv.org/abs/1611.03530 https://arxiv.org/abs/1611.03530 and discussion: https://news.ycombinator.com/item?id=13566917 https://news.ycombinator.com/item?id=13566917
- p1esk 6y ago
  I'm confused about "still". Skimming this version I don't see anything new added to the original paper.
  - visarga 6y ago
    Exactly, the paper "still" applies.
benlivengood 6y ago
This is only tangentially my field, so pure speculation. I suppose it's possible that generalized minima are numerically more common than overfitted minima in an over-parameterized model, so probabilistically SGD will find a more general minima than not, regardless of regularization.
- hervature 6y ago
  I think the general consensus (from my interactions) is that a local minima requires the gradient to vanish. When you have many dimensions, it's unlikely that they are all 0. Coupled with modern optimization methods (primarily momentum), this encourages the result to be in a shallow valley as opposed to a spiky minima. The leap of faith is equating shallow=general and spiky=overfitted.
vonsydov 6y ago
The whole point of neural networks was that you don't need to think hard about generalizations.
- deleted 6y ago
  [deleted]
sdenton4 6y ago
I tend to think this is a result of classification's information density being too low. You can 'learn' a classification problem in the same way with a hash function: take a hash of each image, and memorize the hash and label. Then you only need to 'learn' a very tiny amount of data relative to the size of the dataset to get zero loss. A series of random projections can also function as a crude hash function, and this is likely how NN memorization works. Generative models, on the other hand, don't allow this kind of data reduction trick. If you need to predict (say) every sample of an audio stream conditioned on the previous audio samples, you really do need to memorize the whole dataset (not just a hash of each item) to get to zero loss, because the information density of the output is still very high. And then you've got BERT, which is basically a generative model for language, used for downstream tasks. The information-dense task probably helps with the memorization 'problem' so good features are learned that adapt nicely to other tasks. (and as others have said, memorization may not be a problem in practice. Sometimes it's actually the right thing to do.)
- DavidSJ 6y ago
  The thing is, despite the fact that deep neural networks can effectively learn the hash function you describe (i.e. they have the capacity to do so, and they will do so if "forced" to with random labels), they "prefer" not to. They learn a generalizable pattern in the data. That's the mystery this paper points to.
  - sdenton4 6y ago
    Yeah, I've also done a bunch of work on hard classification problems where generalization pretty much fails. I think they're /probably/ doing the right thing in the infinite training data regime. And they're definitely doing interesting things in the generative case. But ultimately they look for a convenient 'nearby' solution: that's the nature of stochastic gradient descent. If you have two solutions and one of them is the hash function, it would be helpful to /know/ that the hash function is harder to get to than the 'correct' solution.
    - anonymousDan 6y ago
anonymousDan 6y ago
I would be interested to know if there is any effect on the time to convergence between the two setups (i.e. training with real labels Vs random labels). Is it in any way easier/harder to memorize everything Vs extracting a generalizable representation? Edit: Ah, I see they mention they not only investigated non-convergence but also whether training slows down. So randomising/corrupting does impact the convergence rate. This means it is more work/effort to memorize things which I guess is interesting.

165 points by tmfi 6y ago

> Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. I'd say it's more about the simplicity of the task and quality of the data.

deleted 6y ago

[deleted]

magicalhippo 6y ago

The experiments we conducted emphasize that the effective capacity of several successful neural network architectures is large enough to shatter the training data. Consequently, these models are in principle rich enough to memorize the training data. So they're fitting elephants[1]. I've been trying to use DeepSpeech[2] lately for a project, would be interesting to see the results for that. I guess it could also be a decent test for your model? Retrain it with random labels and if it succeeds the model is just memorizing, so either reduce model complexity or add more training data? [1]: https://www.johndcook.com/blog/2011/06/21/how-to-fit-an-elephant/ https://www.johndcook.com/blog/2011/06/21/how-to-fit-an-elep... [2]: https://github.com/mozilla/DeepSpeech https://github.com/mozilla/DeepSpeech

mxwsn 6y ago
Large model capacity enough to perfectly memorize/interpolate data may not be a bad thing. A phenomenon known as "deep double descent" says that increasing modeling capacity relative to the dataset size can reduce generalization error, even after the model achieves perfect training performance (see work by Mikhail Belkin [0] and empirical demonstrations on large deep learning tasks by researchers from Harvard/OpenAI [1]). Other work argues that memorization is critical to good performance on real-world tasks where the data distribution is often long-tailed [2]: to perform well at tasks where the training data set only has 1 or 2 examples, it's best to memorize those labels, rather than extrapolate from other data (which a lower capacity model may prefer to do). [0]: https://arxiv.org/abs/1812.11118 https://arxiv.org/abs/1812.11118 [1]: https://openai.com/blog/deep-double-descent/ https://openai.com/blog/deep-double-descent/ [2]: https://arxiv.org/abs/2008.03703 https://arxiv.org/abs/2008.03703
- derbOac 6y ago
  I think there's something critical about the implicit data universe being considered in the test and training data, and in these randomized datasets. Memorizing elephants isn't necessarily a bad thing if you can be assured there are actually elephants in the data, or if your job is to reproduce some data that has some highly non-random, low entropy (in an abstract sense) features. I think where the phenomenon in this paper, and deep double descent starts to clash with intuition, is the more realistic case where the adversarial data universe is structured, not random, but not conforming to the observed training target label alphabet (to borrow a term loosely from the IT literature). That is, it's interesting to know that these models can perfectly reproduce random data, but generalizing from training to test data isn't interesting in a real-world sense if both are constrained by some implicit features of the data universe involved in the modeling process (e.g., that the non-elephant data only differs randomly from the elephant data, or doesn't contain any nonelephants that aren't represented in the training data). So then you end up with this: https://www.theverge.com/2020/9/20/21447998/twitter-photo-preview-white-black-faces https://www.theverge.com/2020/9/20/21447998/twitter-photo-pr... I guess it seems to me there's a lot of implicit assumptions about the data space and what's actually being inferred in a lot of these DL models. The insight about SGD is useful, but maybe only underscores certain things, and seems to get lost in some of the discussion about DDD. Even Rademacher complexity isn't taken with regard to the entire dataspace, just over a uniformly random sampling of it -- so it will underrepresent certain corners of the data space that are highly nonuniform, low entropy, which is exactly where the trouble lies. There's lots of fascinating stuff in this area of research, lots to say, glad to see it here on HN again.
- medlyyy 6y ago
  Thanks for your [2] link paper, it's an interesting one. If you think about it, the high-level idea makes some intuitive sense. Since NNs are known to be "equivalent" to kernel methods - which are, to oversimplify, essentially nearest-neighbour + a similarity function - then the ability of NNs to memorise specific training examples can be analogised to adding another "neighbour" to interpolate from. So maybe it's not too surprising that NNs which can do this can have better generalisation performance. (Although it is still surprising, since I certainly wouldn't predict it a priori!) Really, what is the difference between a "feature" and "memorising a data point"? It seems only a matter of scale - how much of the input space is "relevant" to the learned representation.
caddemon 6y ago
Would that necessarily imply it is memorizing on the non-random labels though? I know analogies to human learning are overdone, but I definitely have seen humans fall back on memorization when they are struggling to "actually learn" some topic. So genuinely asking from a technical/ML perspective - is it possible a network could optimize training loss without memorization when possible, but as that fails end up just memorizing?
- hervature 6y ago
  > is it possible a network could optimize training loss without memorization when possible What does that even mean? ML is simply a set of tools to minimize (f(x_i) - y_i)^2 (if you don't like squared loss, pick whatever you prefer). In particular, f() here is our neural network. There is no loss without previous data. The only thing the network is trying to do is "memorize" old data.
  - caddemon 6y ago
    It depends of course how you are defining memorization, but the network doesn't necessarily need to use the entirety of every input to do what you are describing. I would think what people mean when they say "it isn't learning, just memorizing" is that the vast majority of information about all previous inputs is being directly encoded in the network. The person I was responding to mentioned training on random labels, and if training still goes well the network must be a memorizer. But I don't see why it couldn't be the case that a network is able to act as a memorizer, but doesn't if there are certain patterns it can generalize on in the training data. Also, there is no human learning without previous data either, but I wouldn't characterize all of human learning as memorization.
    - dumb1224 6y ago
      I don't understand the random label training part. Presumably you train on randomised labels which have no relationship with the input but surely it won't generalise well at all given the small probability of predicting the labels correctly by chance (The setup for a Permutation test am I wrong)?
bloaf 6y ago
That's not my reading. I think they are saying that models which *can* over-fit the data in both theory and practice, appear not do so when there are in fact generalizations in the data.
- magicalhippo 6y ago
  Ah hmm yes, good point. I forgot this piece from the article: We observe a steady deterioration of the generalization error as we increase the noise level. This shows that neural networks are able to capture the remaining signal in the data while at the same time fit the noisy part using brute-force. Since they score well on the test data, they must have generalized to some degree. But since they're just as good at training on random input they also have the capacity to just memorize the training data.
trowngon 6y ago
Irrespective of the subject Deepspeech is very old archtecture with suboptimal results. You'd better try any recent conformer implementations (flashlight, nemo, wenet, etc) or wav2vec.
- magicalhippo 6y ago
  I ended up with DeepSpeech since it was very easy to get started with, and it has support for fairly low-latency inferencing which is very important for my project. I will take a look at the ones you suggested though, thanks for the heads-up!

blt 6y ago

Please add the (still) to the HN post title. The original version of the paper without (still) in the title is several years old.

davnn 6y ago
2016 for v1 to be exact, link: https://arxiv.org/abs/1611.03530 https://arxiv.org/abs/1611.03530 and discussion: https://news.ycombinator.com/item?id=13566917 https://news.ycombinator.com/item?id=13566917
p1esk 6y ago
I'm confused about "still". Skimming this version I don't see anything new added to the original paper.
- visarga 6y ago
  Exactly, the paper "still" applies.

benlivengood 6y ago

This is only tangentially my field, so pure speculation. I suppose it's possible that generalized minima are numerically more common than overfitted minima in an over-parameterized model, so probabilistically SGD will find a more general minima than not, regardless of regularization.

hervature 6y ago
I think the general consensus (from my interactions) is that a local minima requires the gradient to vanish. When you have many dimensions, it's unlikely that they are all 0. Coupled with modern optimization methods (primarily momentum), this encourages the result to be in a shallow valley as opposed to a spiky minima. The leap of faith is equating shallow=general and spiky=overfitted.

vonsydov 6y ago

The whole point of neural networks was that you don't need to think hard about generalizations.

deleted 6y ago
[deleted]

sdenton4 6y ago

I tend to think this is a result of classification's information density being too low. You can 'learn' a classification problem in the same way with a hash function: take a hash of each image, and memorize the hash and label. Then you only need to 'learn' a very tiny amount of data relative to the size of the dataset to get zero loss. A series of random projections can also function as a crude hash function, and this is likely how NN memorization works. Generative models, on the other hand, don't allow this kind of data reduction trick. If you need to predict (say) every sample of an audio stream conditioned on the previous audio samples, you really do need to memorize the whole dataset (not just a hash of each item) to get to zero loss, because the information density of the output is still very high. And then you've got BERT, which is basically a generative model for language, used for downstream tasks. The information-dense task probably helps with the memorization 'problem' so good features are learned that adapt nicely to other tasks. (and as others have said, memorization may not be a problem in practice. Sometimes it's actually the right thing to do.)

DavidSJ 6y ago
The thing is, despite the fact that deep neural networks can effectively learn the hash function you describe (i.e. they have the capacity to do so, and they will do so if "forced" to with random labels), they "prefer" not to. They learn a generalizable pattern in the data. That's the mystery this paper points to.
- sdenton4 6y ago
  Yeah, I've also done a bunch of work on hard classification problems where generalization pretty much fails. I think they're /probably/ doing the right thing in the infinite training data regime. And they're definitely doing interesting things in the generative case. But ultimately they look for a convenient 'nearby' solution: that's the nature of stochastic gradient descent. If you have two solutions and one of them is the hash function, it would be helpful to /know/ that the hash function is harder to get to than the 'correct' solution.
  - anonymousDan 6y ago

anonymousDan 6y ago

I would be interested to know if there is any effect on the time to convergence between the two setups (i.e. training with real labels Vs random labels). Is it in any way easier/harder to memorize everything Vs extracting a generalizable representation? Edit: Ah, I see they mention they not only investigated non-convergence but also whether training slows down. So randomising/corrupting does impact the convergence rate. This means it is more work/effort to memorize things which I guess is interesting.