Two things - firstly you need to understand the concept of overfitting : http://en.wikipedia.org/wiki/Overfitting http://en.wikipedia.org/wiki/Overfitting . If

Two things - firstly you need to understand the concept of overfitting : http://en.wikipedia.org/wiki/Overfitting http://en.wikipedia.org/wiki/Overfitting . If teams were allowed to train on the full dataset, it would be possible to get a 100% score, yet still not have a model that was useful on any images that were not in the training set. Furthermore, if you allowed infinite submissions, teams could just train a million models with slightly different hyperparameters, and submit the one that did best (which may be what Baidu was trying to do here). This is a problem because now there is the possibility that you are overfitting the test data - there would be no way to tell if the accuracy generalised to other images without coming up with more labelled data, i.e. making a new test set. Second of all, cross validation : http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29 http://en.wikipedia.org/wiki/Cross-validation_%28statistics%.... You don't HAVE to submit to the test set to get a feel of how well your model is performing. On datasets this large, cross validation should be effective, if more time consuming method (unless your model is extremely unstable).