This doesn't affect the actual contest results on the test set. Anyone who wants to overfit to the validation set can do so, getting a nice temporary leaderboar

This doesn't affect the actual contest results on the test set. Anyone who wants to overfit to the validation set can do so, getting a nice temporary leaderboard spot at the price of test set generalization, which is what determines the winners. That said it'd obviously be nice to build leaderboards that are robust to this type of gaming. There's been some recent work, mostly driven by Moritz Hardt AFAIK (briefly referenced in the submission), using ideas from differential privacy to create leaderboards that leak only a bounded amount of information about the validation set: http://arxiv.org/abs/1502.04585 http://arxiv.org/abs/1502.04585.