3 ms·

I never understood Kaggle. Most competitions don't require code to be submitted, just predictions to be made on a test set with missing labels. So, you don't ev

I never understood Kaggle. Most competitions don't require code to be submitted, just predictions to be made on a test set with missing labels. So, you don't even need to apply machine learning and I'd bet money that lots of winners don't and label by hand or outsource. I don't understand the fascination and appeal of these so-called "grandmasters".

throwawayjava 7y ago
Without very careful contest design, the best performers are obviously going to be over-fitting. Especially if the entire distribution is public. That's exactly what this team did. This is true of academic contests in general, btw, even without cheating. They stop being interesting/fun/good signals as soon as people start treating them as an independent skill set. Comparing performance on chess games for the first N games between two new players might be a good signal for some general intellectual capabilities. Comparing experienced players against one another is mostly just testing who's spent more time learning about chess.
blurps 7y ago
Winners have to submit code most of the times. There are sporadic computer vision competitions where it is possible to hand-label (such as geo-int competitions where people reverse engineered the flight paths), but vast majority it is impossible to hand-label. Nobody becomes a grandmaster without applying world-class machine learning skills. The "cheater" in this competition is both a world-class data scientist and a reverse engineering hacker. Heck, people used to write papers about how they crawled the ground truth. Now they see their name in the newspaper.
MasterScrat 7y ago
> I never understood Kaggle. Most competitions don't require code to be submitted Your point of view is outdated ;-) In recent ML competitions, participants do submit code that is run on a held-out dataset - as was the case in the PetFinder.my challenge in question here. Most competition platforms are migrating to this format, as otherwise you can just label by hand as you said. Note that this competition went even further: not only was the evaluation code run on Kaggle, the training code was also run there. This means that you couldn't even train a gigantic model then submit it: your model had to be trainable within well defined time and resource constraints, which is a great way to level the playing field. Of course, there's still some unfairness as people with more resources can try out more solutions before submitting a model to be trained on the platform. No platform has a solution for this yet!

throwawayjava 7y ago

Without very careful contest design, the best performers are obviously going to be over-fitting. Especially if the entire distribution is public. That's exactly what this team did. This is true of academic contests in general, btw, even without cheating. They stop being interesting/fun/good signals as soon as people start treating them as an independent skill set. Comparing performance on chess games for the first N games between two new players might be a good signal for some general intellectual capabilities. Comparing experienced players against one another is mostly just testing who's spent more time learning about chess.

blurps 7y ago

Winners have to submit code most of the times. There are sporadic computer vision competitions where it is possible to hand-label (such as geo-int competitions where people reverse engineered the flight paths), but vast majority it is impossible to hand-label. Nobody becomes a grandmaster without applying world-class machine learning skills. The "cheater" in this competition is both a world-class data scientist and a reverse engineering hacker. Heck, people used to write papers about how they crawled the ground truth. Now they see their name in the newspaper.

MasterScrat 7y ago

> I never understood Kaggle. Most competitions don't require code to be submitted Your point of view is outdated ;-) In recent ML competitions, participants do submit code that is run on a held-out dataset - as was the case in the PetFinder.my challenge in question here. Most competition platforms are migrating to this format, as otherwise you can just label by hand as you said. Note that this competition went even further: not only was the evaluation code run on Kaggle, the training code was also run there. This means that you couldn't even train a gigantic model then submit it: your model had to be trainable within well defined time and resource constraints, which is a great way to level the playing field. Of course, there's still some unfairness as people with more resources can try out more solutions before submitting a model to be trained on the platform. No platform has a solution for this yet!