Congratulations to the author for both of his last 2 posts making it to the HN front page! This explains the need to drill home the train-test idea from the la

Congratulations to the author for both of his last 2 posts making it to the HN front page! This explains the need to drill home the train-test idea from the last post. I hadn't thought about this before but multiple submissions do amount to multiple peeks at your held-out test set, which is a huge ML no-no. I don't know much about LSVRC, but doesn't the way Kaggle work prevent this? AFAIR you get a "public" test-score which is used for the leaderboards, but once the deadline for submissions is up, each submission is evaluated on a held-out test set giving you a "private" score. Now that I think about it, I'm not sure how that works, I guess the accuracy they show you as your public score is only on part of the submitted rows? Regardless of how that's done, could the LSVRC organisers not do something similar?