4 ms·

I disagree: some "collective overfitting" happens when everyone evaluates on the same dataset (and often, the same test set). There's a neat set of papers by R

I disagree: some "collective overfitting" happens when everyone evaluates on the same dataset (and often, the same test set). There's a neat set of papers by Recht et al. showing results are slightly overfit to the test partitions of ImageNet and CIFAR-10: rotating examples between the train and test partitions causes systems to perform up to 10-15% worse. https://arxiv.org/abs/1806.00451 https://arxiv.org/abs/1806.00451 https://arxiv.org/abs/1902.10811 https://arxiv.org/abs/1902.10811 There's another neat bit of work involving MNIST. The original dataset (from the mid-90s) had 60,000 test examples, but the distributed versions that virtually everyone uses has only 10,000 test examples. Performance on these held-out examples is, unsurprisingly, a bit worse: https://arxiv.org/pdf/1905.10498.pdf https://arxiv.org/pdf/1905.10498.pdf

dontreact 5y ago
I think it's super important to separate the following two situations, both of which I suppose are fair to call overfitting Situation A: Models are slightly overfit to some portions of the test set. But the following holds IF PerformanceOnBenchmark(Model A) > PerformanceOnBenchmark(Model B) Then PerformanceOnSimilarDaset(Model A) > PerformanceOnSimilarDataset(Model B) Therefore progress on the benchmark is predictive of progress in general. Situation B: The relation does not hold, and therefore progress on the benchmark does not predict general progress. This almost always happen if you train a deep neural network long enough: train performance goes up, but test performance goes down. If you look at figure 2 of the first paper you sent, you will note that it shows we are in situation A and not situation B. Situation A overfitting = diminishing returns on improvements on benchmark, but the benchmark is still useful. Situation B overfitting = the benchmark is now useless
- sdenton4 5y ago
  >> "This almost always happen if you train a deep neural network long enough: train performance goes up, but test performance goes down." This is a problem that is more common for classification problems, I think. Generative and self-supervised models (trained with augmentation) tend to just get better forever (with some asymptote) because memorization isn't a viable strategy. I personally think image classification is mostly a silly problem to judge new algos on as a result, and leads to all kinds of nonsense as people try to extrapolate meaning from new results.
  - dontreact 5y ago
    Nearly all useful machine learning is supervised, still. And if you are using a neural network, it will eventually memorize. This is fine though, we have early stopping :)
    - sdenton4 5y ago
      Eh, I work in audio ml, where most of the interesting and useful work is in conditioned generative models. Compression, TTS, source separation. The bias towards classification is a side effect of people starting at imagenet too long; I really think it's holding the field back.

dontreact 5y ago

I think it's super important to separate the following two situations, both of which I suppose are fair to call overfitting Situation A: Models are slightly overfit to some portions of the test set. But the following holds IF PerformanceOnBenchmark(Model A) > PerformanceOnBenchmark(Model B) Then PerformanceOnSimilarDaset(Model A) > PerformanceOnSimilarDataset(Model B) Therefore progress on the benchmark is predictive of progress in general. Situation B: The relation does not hold, and therefore progress on the benchmark does not predict general progress. This almost always happen if you train a deep neural network long enough: train performance goes up, but test performance goes down. If you look at figure 2 of the first paper you sent, you will note that it shows we are in situation A and not situation B. Situation A overfitting = diminishing returns on improvements on benchmark, but the benchmark is still useful. Situation B overfitting = the benchmark is now useless

sdenton4 5y ago
>> "This almost always happen if you train a deep neural network long enough: train performance goes up, but test performance goes down." This is a problem that is more common for classification problems, I think. Generative and self-supervised models (trained with augmentation) tend to just get better forever (with some asymptote) because memorization isn't a viable strategy. I personally think image classification is mostly a silly problem to judge new algos on as a result, and leads to all kinds of nonsense as people try to extrapolate meaning from new results.
- dontreact 5y ago
  Nearly all useful machine learning is supervised, still. And if you are using a neural network, it will eventually memorize. This is fine though, we have early stopping :)
  - sdenton4 5y ago
    Eh, I work in audio ml, where most of the interesting and useful work is in conditioned generative models. Compression, TTS, source separation. The bias towards classification is a side effect of people starting at imagenet too long; I really think it's holding the field back.