6 ms·

so why didn't the ensemble win? in the rules it says, 'At the end of this period, qualifying submissions will be judged (see Judging below) in order of the larg

by tmsh 17y ago

so why didn't the ensemble win? in the rules it says, 'At the end of this period, qualifying submissions will be judged (see Judging below) in order of the largest improvement over the qualifying RMSE on the test subset.' this isn't the icfp. i assume the person on the leaderboard is in fact the leader. the difference between the 'test' and 'quiz' sounds b.s. to me. at this rate, i know i won't even be contemplating netflix prize 2. at the very least, netflix owes it to the community to explain why they (at this point, seemingly corruptly) made the decision they did. i suppose they say they'll post the final 'test subset' scores on the leaderboard. it doesn't appear to be a very 'open' contest if they don't actually publish exactly what the test subset is and how the rankings are determined. when it's that close, everything should be shown to be exactly done within the rules. otherwise, they really risk delegitimizing the whole contest, imho.

pmjordan 17y ago
I'm not familiar with the details in any way, but it sounds like you could probably produce an algorithm that produces a perfect result for the rest if you give it a subset of the publicly available data. That doesn't mean your algorithm is good in general, that just means you essentially encoded the test data set into your algorithm. The real test is always going to be how well you do on data that you've never seen. As for why the intermittent leaderboard shows results based on the public dataset: using a specially crafted algorithm, you could leak information about hidden data through any feedback you get from the test environment. Such information could be used to create an aforementioned algorithm that does well in this specific case. see also: http://en.wikipedia.org/wiki/Overfitting http://en.wikipedia.org/wiki/Overfitting
- tmsh 17y ago
  i see. but if they don't make their private dataset public, what's to prevent netflix from fitting it to a particular entry?
  - deleted 17y ago
    [deleted]
  - deleted 17y ago
    [deleted]
  - mquander 17y ago
    Why would Netflix corrupt their own competition in this way? I don't understand what you're concerned about.
    - tmsh 17y ago
      maybe because 'bellkor's pragmatic chaos' won the progress prize and they have some special relationship with them. could be any number of reasons. this is i think the danger of a private corporation running a programming contest. i'm not saying netflix is unfair or corrupt. i hope not. but i think they owe it to the contestants to explain exactly why they chose the #2 entry on the leaderboard over the #1 entry. and not just (though i appreciate people's comments) because of overfitting. how is the private 'test' subset chosen? i have no affiliation with this contest or any of the teams in any way, but how would you like to work on this for years only to be told that even though you beat everyone in the quiz, in the private 'test', you lost.
      - ErrantX 17y ago
        IIRC it was very clear: once you submitted the public test set and obtained a score that broke the 10% barrier there was a 30 day wait period for final submisions. Then the ones over 10% were run against the final test set and the best winner determined from that. Obviously releasing specific details about the final test set doesn't work because teams can fit to that :)
      - derefr 17y ago
        They don't need to make it public beforehand, but they really should after-the-fact, to verify their calculations upon it. Then, proving it's unbiased is a simple matter of releasing the hash to the dataset before the contest starts, to prove that the final dataset they release is the same one they generated before the contest started. Now, on the front of selecting the winners before the contest has even started, and telling them about how to bias their algorithm so it will do better on the final--I can't really help you. I'd just hope that the programming teams would be honorable enough to mention if one of them were offered such a thing.
deleted 17y ago
[deleted]

by tmsh 17y ago

pmjordan 17y ago

I'm not familiar with the details in any way, but it sounds like you could probably produce an algorithm that produces a perfect result for the rest if you give it a subset of the publicly available data. That doesn't mean your algorithm is good in general, that just means you essentially encoded the test data set into your algorithm. The real test is always going to be how well you do on data that you've never seen. As for why the intermittent leaderboard shows results based on the public dataset: using a specially crafted algorithm, you could leak information about hidden data through any feedback you get from the test environment. Such information could be used to create an aforementioned algorithm that does well in this specific case. see also: http://en.wikipedia.org/wiki/Overfitting http://en.wikipedia.org/wiki/Overfitting

tmsh 17y ago
i see. but if they don't make their private dataset public, what's to prevent netflix from fitting it to a particular entry?
- deleted 17y ago
  [deleted]
- deleted 17y ago
  [deleted]
- mquander 17y ago
  Why would Netflix corrupt their own competition in this way? I don't understand what you're concerned about.
  - tmsh 17y ago
    maybe because 'bellkor's pragmatic chaos' won the progress prize and they have some special relationship with them. could be any number of reasons. this is i think the danger of a private corporation running a programming contest. i'm not saying netflix is unfair or corrupt. i hope not. but i think they owe it to the contestants to explain exactly why they chose the #2 entry on the leaderboard over the #1 entry. and not just (though i appreciate people's comments) because of overfitting. how is the private 'test' subset chosen? i have no affiliation with this contest or any of the teams in any way, but how would you like to work on this for years only to be told that even though you beat everyone in the quiz, in the private 'test', you lost.
    - ErrantX 17y ago
      IIRC it was very clear: once you submitted the public test set and obtained a score that broke the 10% barrier there was a 30 day wait period for final submisions. Then the ones over 10% were run against the final test set and the best winner determined from that. Obviously releasing specific details about the final test set doesn't work because teams can fit to that :)
    - derefr 17y ago
      They don't need to make it public beforehand, but they really should after-the-fact, to verify their calculations upon it. Then, proving it's unbiased is a simple matter of releasing the hash to the dataset before the contest starts, to prove that the final dataset they release is the same one they generated before the contest started. Now, on the front of selecting the winners before the contest has even started, and telling them about how to bias their algorithm so it will do better on the final--I can't really help you. I'd just hope that the programming teams would be honorable enough to mention if one of them were offered such a thing.

deleted 17y ago

[deleted]