3 ms·

The thing is, despite the fact that deep neural networks can effectively learn the hash function you describe (i.e. they have the capacity to do so, and they wi

The thing is, despite the fact that deep neural networks can effectively learn the hash function you describe (i.e. they have the capacity to do so, and they will do so if "forced" to with random labels), they "prefer" not to. They learn a generalizable pattern in the data. That's the mystery this paper points to.

sdenton4 6y ago
Yeah, I've also done a bunch of work on hard classification problems where generalization pretty much fails. I think they're /probably/ doing the right thing in the infinite training data regime. And they're definitely doing interesting things in the generative case. But ultimately they look for a convenient 'nearby' solution: that's the nature of stochastic gradient descent. If you have two solutions and one of them is the hash function, it would be helpful to /know/ that the hash function is harder to get to than the 'correct' solution.
- anonymousDan 6y ago
  But isn't that shown by the fact it takes training longer to converge in their experiments with random labels vs real labels?
  - sdenton4 6y ago
    Yeah, not a bad idea; for any given problem, just scramble the labels and compare training time... For my pet problem, there's quite different performance across classes. It would be tougher to do the comparison for a subset of classes, but could still probably design some kind of experiment.
joe_the_user 6y ago
The paper says that: "Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data" What is a generalization of a random labeling? Does the network become psychic? Is this saying "garbage works fine by these measures"?
- visarga 6y ago
  The network learns how to map data into an intermediate representation, then learns a hash output for each of these intermediate values. So it learns two things compared to when the intermediate values correspond to the task.
- DavidSJ 6y ago
  Random labeling can’t generalize. But in the usual case, the labels are nonrandom and the learned map generalizes.
joe_the_user 6y ago
That's the mystery this paper points to I'm still honestly confused here. Yes, the success of neural networks generally shows they more than memorize. Some of the experiments here show they can just memorize too. I guess some describe something in-between. But all this seems obvious. Is there something that's being quantified here?
- DavidSJ 6y ago
  You can measure the generalization gap: the difference between training and test performance. With good generalization that gap will be small. When fitting random labels the gap will be large. Classical statistical learning theory (from Vapnik and Chervonenkis, among others), predicts that low capacity models will generalize (have small generalization gaps). It makes no promises for large capacity models, i.e. those which can fit random labels. Yet here we are.

sdenton4 6y ago

Yeah, I've also done a bunch of work on hard classification problems where generalization pretty much fails. I think they're /probably/ doing the right thing in the infinite training data regime. And they're definitely doing interesting things in the generative case. But ultimately they look for a convenient 'nearby' solution: that's the nature of stochastic gradient descent. If you have two solutions and one of them is the hash function, it would be helpful to /know/ that the hash function is harder to get to than the 'correct' solution.

anonymousDan 6y ago
But isn't that shown by the fact it takes training longer to converge in their experiments with random labels vs real labels?
- sdenton4 6y ago
  Yeah, not a bad idea; for any given problem, just scramble the labels and compare training time... For my pet problem, there's quite different performance across classes. It would be tougher to do the comparison for a subset of classes, but could still probably design some kind of experiment.

joe_the_user 6y ago

The paper says that: "Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data" What is a generalization of a random labeling? Does the network become psychic? Is this saying "garbage works fine by these measures"?

visarga 6y ago
The network learns how to map data into an intermediate representation, then learns a hash output for each of these intermediate values. So it learns two things compared to when the intermediate values correspond to the task.
DavidSJ 6y ago
Random labeling can’t generalize. But in the usual case, the labels are nonrandom and the learned map generalizes.

joe_the_user 6y ago

That's the mystery this paper points to I'm still honestly confused here. Yes, the success of neural networks generally shows they more than memorize. Some of the experiments here show they can just memorize too. I guess some describe something in-between. But all this seems obvious. Is there something that's being quantified here?

DavidSJ 6y ago
You can measure the generalization gap: the difference between training and test performance. With good generalization that gap will be small. When fitting random labels the gap will be large. Classical statistical learning theory (from Vapnik and Chervonenkis, among others), predicts that low capacity models will generalize (have small generalization gaps). It makes no promises for large capacity models, i.e. those which can fit random labels. Yet here we are.