4 ms·

> The results of Recht et al. do not show that people are performing validation on the test set. As I read #3, the hypothesis doesn't depend on individual rese

> The results of Recht et al. do not show that people are performing validation on the test set. As I read #3, the hypothesis doesn't depend on individual researchers acting unethically by validating against the test set. Instead, I read it as an analogy to significance bias in other sciences: machine learning models that don't perform as well on the validation set simply aren't published, so the field as a whole over-fits, as if validation is performed on the test set. In the paper you link, the authors themselves do explicitly note some test-set shenanigans: > But this assumption [that the models are independent of the test set] is undermined by the common practice of tuning model hyperparameters directly on the test set, which introduces dependencies between the model $\ˆf$ and the test set S. In the extreme case, this can be seen as training directly on the test set

jph00 8y ago
The authors made various claims or implications not backed up by their experiment. However, it absolutely does not answer the question that is asked in the title of the paper, and the process they use is incapable of answering that question. If you go back and read the original CIFAR10 paper, you'll see that the process they carefully went through meant that they curated the most suitable images for each category. By definition, what's left over (which is what the Recht et al paper chose from) is less good images, which are of course therefore harder to classify. All the experiment measures is how good they are at matching the distribution of the original dataset. The answer, they discovered, is: not very.