This sort of "shortcut" exploits a well known flaw in the training any machine learning or regression model using any algorithm. It is the problem of overfitti

This sort of "shortcut" exploits a well known flaw in the training any machine learning or regression model using any algorithm. It is the problem of overfitting. The problem is that given enough parameters you can tweak those parameters so that your model exactly fits a target data set (including the random noise). The reason for excluding and limiting access to the test set is so that parameters can't be modified such that the model is adjusted to the specific test data set. It is a well known problem and exploit that the rules are designed to guard against. Here is the wiki page on overfitting: https://en.wikipedia.org/wiki/Overfitting https://en.wikipedia.org/wiki/Overfitting Say the test data set, by chance, has 1% more dogs than the training set. By tweaking your algorithm to guess "dog" an extra 1% of the time you may be able to get .1% increase in your success rate. Also, that tweaking isn't a manual process it's part of the training process based on feedback from the results of your test set score. The improvement is enough to "win", but it's not because your algorithm is better. That goes along with the argument that the data set is at its end of life. Maybe we're at the point that gaming the system is the only way to eek out the .1% needed to "win". In that case it's time to move on to a tougher test. EDIT: I'd like to point out that the ImageNet competition is continually on top of the "time to make it more challenging" aspect. They introduced localization in 2011 (identifying not just what, but where items are). The 2015 competition includes, for the first time, recognition and localization tests in video clips.