4 ms·
If I create 1000 predictive models, with random parameter settings, some of those models will outperform others. In fact, the accuracy rankings of the models gi
by pgroves 14y ago
If I create 1000 predictive models, with random parameter settings, some of those models will outperform others. In fact, the accuracy rankings of the models given a single hold-out validation set will form their own distribution. It is difficult, if not impossible, to tell if one model outranks another b/c it is actually better or just occupies a different place in that distribution due to the random properties of the hold-out set.
So, why do you believe this isn't happening when hundreds of people submit their models to be scored against a secret hold-out set? Is there any evidence that the hundreds of entries are forming a better distribution of accuracy scores than random models? Is there evidence the winners of these competitions truly have better models and don't just have a luckier placement in a random distribution of accuracy scores?
- jph00 14y agoYes, we can calculate the distribution and find the likelihood that a given team would beat another if the data is resampled. Here's a detailed example: http://www.tiberius.biz/pakdd07.html http://www.tiberius.biz/pakdd07.html The size of improvements seen in competitions vs previous best performing models (eg Allstate competition showed improvement of 270%) is well beyond that explicable by chance. Also: the test set size in most competitions is hundreds of thousands of rows - sampling error is very small in these cases.
- pgroves 14y agoI still find it hard to believe the people in the top 10 aren't all at the theoretical limit of predictive accuracy with their final order being determined by nuances in the experimental design (things like how the raw data was collected). Which would be the sort of thing statisticians live with all the time, except that Kaggle then makes it winner take all.
- jph00 14y agoThat certainly does happen - indeed our tag line is "making data science a sport"... Like in all sports, there can be a stochastic element to who ends up winning! However, in private competitions (which is where the money is!) everyone selected to participate wins something. It's more like golf - make the cut, and you'll definitely make a few dollars, but the winner will take home the biggest prize.