4 ms·

I wonder how often there is an actual, statistically-significant difference between the performance of the first and second, third, (etc) place entries. Perfor

I wonder how often there is an actual, statistically-significant difference between the performance of the first and second, third, (etc) place entries. Performance on the test set is a point estimate of the classifier's performance, but there is some variance around that point estimate due to the precise composition of the test set. For example, a classifier will perform somewhat differently if these 10,000 digits were in the test set instead of those 10,000 digits. Academic papers have gotten better about reporting confidence intervals, but I don't think I've ever seen a contest that uses them.

ACow_Adonis 10y ago
As a stats guy, that's actually been a thing that stops me participating in many of these comps. The leader board for many of the tasks actually end up being either statistically the same (which means where you finish in the top N will have a significant random factor assuming you actually know what you're doing ink the first place). The second is that it's also common for the top places to game the structure of the data or overfit to the particular data or training set. Good for competition :) But bad data science :(
- hcarvalhoalves 10y ago
  > The second is that it's also common for the top places to game the structure of the data or overfit to the particular data or training set. And somewhat often, the datasets for those competitions are f* up because having clean, unbiased, historical data is uncommon in organizations. Or they mess up while anonymizing the data. Other point about competitions is that you don't need to deal with things like concept drift, implementation complexity, resource/latency constraints for predictions and a lot of other things that people applying ML in practice have to face. More often than not, these factors dominate having a 1% or 0.1% better model - unless you're like Google or Facebook.
- mattkrause 10y ago
  Excellent--that's been my excuse too :-)

ACow_Adonis 10y ago

As a stats guy, that's actually been a thing that stops me participating in many of these comps. The leader board for many of the tasks actually end up being either statistically the same (which means where you finish in the top N will have a significant random factor assuming you actually know what you're doing ink the first place). The second is that it's also common for the top places to game the structure of the data or overfit to the particular data or training set. Good for competition :) But bad data science :(

hcarvalhoalves 10y ago
> The second is that it's also common for the top places to game the structure of the data or overfit to the particular data or training set. And somewhat often, the datasets for those competitions are f* up because having clean, unbiased, historical data is uncommon in organizations. Or they mess up while anonymizing the data. Other point about competitions is that you don't need to deal with things like concept drift, implementation complexity, resource/latency constraints for predictions and a lot of other things that people applying ML in practice have to face. More often than not, these factors dominate having a 1% or 0.1% better model - unless you're like Google or Facebook.
mattkrause 10y ago
Excellent--that's been my excuse too :-)