4 ms·
My problem with a lot of these competitions, particularly the high-reward ones, is you've got people duking it out at the 4th or 5th decimal point for the top
by jbooth 13y ago
My problem with a lot of these competitions, particularly the high-reward ones, is you've got people duking it out at the 4th or 5th decimal point for the top few spots. At that point, you're not even competing for best generalizable algorithm, you're competing for 'algorithm which happens to fit the holdout set exceptionally well'. It's overfitting at a distance.
- Houshalter 13y agoThe data set used for the leaderboard is not the dataset used in the final evaluation. They also limit the number of final submissions you can have and the number of submissions you can make per day to prevent that. Often the person at the top of the leaderboard will move down a few places in the final evaluation.
- dhruvbhatia 13y agoFurther to Houshalter's point: the President & Chief Scientist of Kaggle addresses this specific issue and explains the Public/Private Leaderboard system in this webcast: http://youtu.be/GrugzF0-V3I http://youtu.be/GrugzF0-V3I