3 ms·

maybe because 'bellkor's pragmatic chaos' won the progress prize and they have some special relationship with them. could be any number of reasons. this is i th

by tmsh 17y ago

maybe because 'bellkor's pragmatic chaos' won the progress prize and they have some special relationship with them. could be any number of reasons. this is i think the danger of a private corporation running a programming contest. i'm not saying netflix is unfair or corrupt. i hope not. but i think they owe it to the contestants to explain exactly why they chose the #2 entry on the leaderboard over the #1 entry. and not just (though i appreciate people's comments) because of overfitting. how is the private 'test' subset chosen? i have no affiliation with this contest or any of the teams in any way, but how would you like to work on this for years only to be told that even though you beat everyone in the quiz, in the private 'test', you lost.

ErrantX 17y ago
IIRC it was very clear: once you submitted the public test set and obtained a score that broke the 10% barrier there was a 30 day wait period for final submisions. Then the ones over 10% were run against the final test set and the best winner determined from that. Obviously releasing specific details about the final test set doesn't work because teams can fit to that :)
derefr 17y ago
They don't need to make it public beforehand, but they really should after-the-fact, to verify their calculations upon it. Then, proving it's unbiased is a simple matter of releasing the hash to the dataset before the contest starts, to prove that the final dataset they release is the same one they generated before the contest started. Now, on the front of selecting the winners before the contest has even started, and telling them about how to bias their algorithm so it will do better on the final--I can't really help you. I'd just hope that the programming teams would be honorable enough to mention if one of them were offered such a thing.
mquander 17y ago
OK, I agree that more transparency would be nice, but I don't think it's a serious concern. It's clearly in Netflix's best interest to promote a fair competition, since the whole point of the exercise (the reason they're spending a million dollars) is to produce the best possible general algorithm for their own use.
- deleted 17y ago
  [deleted]
luchak 17y ago
Did you read the contest rules and FAQ? (http://www.netflixprize.com/rules http://www.netflixprize.com/rules, http://www.netflixprize.com/faq http://www.netflixprize.com/faq -- and if you didn't, what are you doing throwing around words like "corrupt"?) It looks to me like they're being very transparent and explained things very clearly. You have a bunch of data in the test set; these are partitioned randomly into two equally-sized subsets, "quiz" and "test". You submit predictions for both subsets, but are only told how you did on "quiz". Netflix provides an MD5 checksum for the judging file that defines what the partition is; this file will be made available "at the end of the Contest". So this will be verifiable by anyone soon. Also, don't brush overfitting aside. In the (paraphrased) words of one machine learning researcher, "life is a battle against entropy. In the same way, machine learning research is a battle against overfitting." Any data whose test results you use to select your algorithm or to adjust your algorithm's parameters is not properly considered part of the test set; after your optimization that data will provide an optimistically-biased estimate of your true error. Since competitors could get regular updates on their performance on the quiz dataset, one must assume that they were attempting to optimize this performance, and so quiz set performance was not be a good estimate of their true error. You can only get a good estimate of the true error of a method by testing against data that has played no part in its development.
- deleted 17y ago
  [deleted]
- tmsh 17y ago
  > So this will be verifiable by anyone soon Yeah....what happens when the checksums don't check out?

by tmsh 17y ago

ErrantX 17y ago

IIRC it was very clear: once you submitted the public test set and obtained a score that broke the 10% barrier there was a 30 day wait period for final submisions. Then the ones over 10% were run against the final test set and the best winner determined from that. Obviously releasing specific details about the final test set doesn't work because teams can fit to that :)

derefr 17y ago

They don't need to make it public beforehand, but they really should after-the-fact, to verify their calculations upon it. Then, proving it's unbiased is a simple matter of releasing the hash to the dataset before the contest starts, to prove that the final dataset they release is the same one they generated before the contest started. Now, on the front of selecting the winners before the contest has even started, and telling them about how to bias their algorithm so it will do better on the final--I can't really help you. I'd just hope that the programming teams would be honorable enough to mention if one of them were offered such a thing.

mquander 17y ago

OK, I agree that more transparency would be nice, but I don't think it's a serious concern. It's clearly in Netflix's best interest to promote a fair competition, since the whole point of the exercise (the reason they're spending a million dollars) is to produce the best possible general algorithm for their own use.

deleted 17y ago
[deleted]

luchak 17y ago

Did you read the contest rules and FAQ? (http://www.netflixprize.com/rules http://www.netflixprize.com/rules, http://www.netflixprize.com/faq http://www.netflixprize.com/faq -- and if you didn't, what are you doing throwing around words like "corrupt"?) It looks to me like they're being very transparent and explained things very clearly. You have a bunch of data in the test set; these are partitioned randomly into two equally-sized subsets, "quiz" and "test". You submit predictions for both subsets, but are only told how you did on "quiz". Netflix provides an MD5 checksum for the judging file that defines what the partition is; this file will be made available "at the end of the Contest". So this will be verifiable by anyone soon. Also, don't brush overfitting aside. In the (paraphrased) words of one machine learning researcher, "life is a battle against entropy. In the same way, machine learning research is a battle against overfitting." Any data whose test results you use to select your algorithm or to adjust your algorithm's parameters is not properly considered part of the test set; after your optimization that data will provide an optimistically-biased estimate of your true error. Since competitors could get regular updates on their performance on the quiz dataset, one must assume that they were attempting to optimize this performance, and so quiz set performance was not be a good estimate of their true error. You can only get a good estimate of the true error of a method by testing against data that has played no part in its development.

deleted 17y ago
[deleted]
tmsh 17y ago
> So this will be verifiable by anyone soon Yeah....what happens when the checksums don't check out?