3 ms·
Yeah, I've also done a bunch of work on hard classification problems where generalization pretty much fails. I think they're /probably/ doing the right thing in
by sdenton4 6y ago
Yeah, I've also done a bunch of work on hard classification problems where generalization pretty much fails. I think they're /probably/ doing the right thing in the infinite training data regime. And they're definitely doing interesting things in the generative case.
But ultimately they look for a convenient 'nearby' solution: that's the nature of stochastic gradient descent. If you have two solutions and one of them is the hash function, it would be helpful to /know/ that the hash function is harder to get to than the 'correct' solution.
- anonymousDan 6y agoBut isn't that shown by the fact it takes training longer to converge in their experiments with random labels vs real labels?
- sdenton4 6y agoYeah, not a bad idea; for any given problem, just scramble the labels and compare training time... For my pet problem, there's quite different performance across classes. It would be tougher to do the comparison for a subset of classes, but could still probably design some kind of experiment.