3 ms·

This is clearly cheating. HOWEVER, the goal of these contests should be to promote the most accurate and powerful image recognition algorithms that will transf

This is clearly cheating. HOWEVER, the goal of these contests should be to promote the most accurate and powerful image recognition algorithms that will transform the world as we know it. Limiting access to training data makes it more difficult to test changes to an algorithm. These rules do not make sense to me, and I would advocate against them.

sigzero 11y ago
Yet, those are the rules as they currently stand. Baidu knew it and chose to cheat.
sweezyjeezy 11y ago
Two things - firstly you need to understand the concept of overfitting : http://en.wikipedia.org/wiki/Overfitting http://en.wikipedia.org/wiki/Overfitting . If teams were allowed to train on the full dataset, it would be possible to get a 100% score, yet still not have a model that was useful on any images that were not in the training set. Furthermore, if you allowed infinite submissions, teams could just train a million models with slightly different hyperparameters, and submit the one that did best (which may be what Baidu was trying to do here). This is a problem because now there is the possibility that you are overfitting the test data - there would be no way to tell if the accuracy generalised to other images without coming up with more labelled data, i.e. making a new test set. Second of all, cross validation : http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29 http://en.wikipedia.org/wiki/Cross-validation_%28statistics%.... You don't HAVE to submit to the test set to get a feel of how well your model is performing. On datasets this large, cross validation should be effective, if more time consuming method (unless your model is extremely unstable).
- squigs25 11y ago
  Was there a training set made available and distributed? I got the impression that there was not.
  - sweezyjeezy 11y ago
    Yeah the training set is called Imagenet, it's widely used in research.
m_ke 11y ago
They can use a subset of the training data as a validation set. Given enough attempts at the test set it's very easy to overfit your model to that test set, meaning that although the accuracy looks higher, it would generalize worse on a different subset of the data.
hiddencost 11y ago
Luckily you don't work in the field. You can get perfect accuracy on a target set while having approximately zero accuracy on any other set, if allowed to run infinite tests. This isn't cheating because it helped them be better. It's cheating because it helped them get results that look very good but won't generalize well.
- squigs25 11y ago
  In a contest like this, I would expect (and I could be wrong) that there is a final validation set not accessible until "game day". Thus, overfitting to a training data set of any kind would be detrimental and degrade model performance.
irl_zebra 11y ago
I suspect this reasoning, or something like it, is the same faulty rationale that Baidu used to justify their cheating at the competition. As others have already said better than I could, if the goal is indeed to >"promote the most accurate and powerful image recognition algorithms that will transform the world as we know it" as you say, then running hundreds of attempts at this one data set to do better at this specific competition won't foster that goal.

sigzero 11y ago

Yet, those are the rules as they currently stand. Baidu knew it and chose to cheat.

sweezyjeezy 11y ago

Two things - firstly you need to understand the concept of overfitting : http://en.wikipedia.org/wiki/Overfitting http://en.wikipedia.org/wiki/Overfitting . If teams were allowed to train on the full dataset, it would be possible to get a 100% score, yet still not have a model that was useful on any images that were not in the training set. Furthermore, if you allowed infinite submissions, teams could just train a million models with slightly different hyperparameters, and submit the one that did best (which may be what Baidu was trying to do here). This is a problem because now there is the possibility that you are overfitting the test data - there would be no way to tell if the accuracy generalised to other images without coming up with more labelled data, i.e. making a new test set. Second of all, cross validation : http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29 http://en.wikipedia.org/wiki/Cross-validation_%28statistics%.... You don't HAVE to submit to the test set to get a feel of how well your model is performing. On datasets this large, cross validation should be effective, if more time consuming method (unless your model is extremely unstable).

squigs25 11y ago
Was there a training set made available and distributed? I got the impression that there was not.
- sweezyjeezy 11y ago
  Yeah the training set is called Imagenet, it's widely used in research.

m_ke 11y ago

They can use a subset of the training data as a validation set. Given enough attempts at the test set it's very easy to overfit your model to that test set, meaning that although the accuracy looks higher, it would generalize worse on a different subset of the data.

hiddencost 11y ago

Luckily you don't work in the field. You can get perfect accuracy on a target set while having approximately zero accuracy on any other set, if allowed to run infinite tests. This isn't cheating because it helped them be better. It's cheating because it helped them get results that look very good but won't generalize well.

squigs25 11y ago
In a contest like this, I would expect (and I could be wrong) that there is a final validation set not accessible until "game day". Thus, overfitting to a training data set of any kind would be detrimental and degrade model performance.

irl_zebra 11y ago

I suspect this reasoning, or something like it, is the same faulty rationale that Baidu used to justify their cheating at the competition. As others have already said better than I could, if the goal is indeed to >"promote the most accurate and powerful image recognition algorithms that will transform the world as we know it" as you say, then running hundreds of attempts at this one data set to do better at this specific competition won't foster that goal.