5 ms·

BelKor has allegedly won Netflix Prize despite being second on the leaderboard

40 points by vang3lis 17y ago

jacquesm 17y ago
an .0001 difference would seem to me to be 'too close to call' and a reason to reopen and let both teams work for another 30 days to get to a larger than .1 difference. Just like in ping-pong, extend the game if it is that close until there is a clear winner. With a winner takes all contest like this there has to be a clear winner.
- vang3lis 17y ago
  Yes, the difference is negligible, but you obviously can't change rules of the game when it has already finished.
  - jeresig 17y ago
    And to be clear: The winner of the contest is not determined by the number that you see listed on the leaderboard. The number on the leaderboard [1] is on the Quiz dataset, the winner is determined by running against the Test dataset (which is kept private by Netflix). As stated by Yehuda on team BellKor: "our team is top contender for winning the Grand Prize, as we have a better Test score than The Ensemble." This turnaround does not surprise me. BellKor's Pragmatic Chaos took their sweet time getting to 10%+ - and in doing so they were very sure not to overfit [2] the data (making their solution a much more generic and viable solution to the dataset). It's my guess that The Ensemble rushed quickly to 10%+ and overfit their data like mad (which yielded high numbers on the public dataset, but evidently does not translate to a generic solution). I'm looking forward to seeing the final papers published by BellKor, et. al. - they're going to be a fascinating read, regardless. 1: http://www.netflixprize.com/leaderboard http://www.netflixprize.com/leaderboard 2: http://en.wikipedia.org/wiki/Overfitting http://en.wikipedia.org/wiki/Overfitting
    - jedc 17y ago
      A lot of people have been tripped up by this, and it's a VERY important distinction. I hope Netflix posts more about the final (Test) results from each when they officially announce the winner.
    - lliiffee 17y ago
      I don't think that is correct. The numbers on the leaderboard are not the scores on the training set. The netflix prize uses three datasets: the training set, the leaderboard set, and the test set. The training set is distributed to everyone. The test set is totally secret. Access to the leaderboard set is only by submitting results (once per 24 hours) and looking at the results. It is not at all trivial to "overfit" to this leaderboard set. (It could be done by, e.g. submitting results with slight tweaks to the algorithm parameters, but this would take a lot of time since you can only submit every 24 hours. Also, you would basically have to do it consciously.)
- davidw 17y ago
  Winner takes all contests seem to be zero sum games to me, in a certain sense. The pie is a fixed size, and one guy gets all of it. I think I'd rather sink my time into business or open source stuff.
  - hvs 17y ago
    Actually, the top teams have already started negotiating with other companies for the technology that they came up with. Knowledge is never zero-sum.
    - davidw 17y ago
      I wrote contest, not knowledge. The knowledge is external to the contest. Some contests, like this one, have lots of external benefits to the top guys, but "winner takes all contest" in the abstract sense seems to me to be a zero sum game, so I stand by what I said: it's generally better to sink your time into something where the "external stuff" is actually the main benefit.
      - mhb 17y ago
        The term "zero sum" does not really make sense here. Netflix obviously thinks they're getting more than what they're spending on the contest and the winning team obviously thinks they're receiving more than what they've invested. So between those two parties, there is a net gain as is the case in most voluntary transactions. At best, what you're saying is that other competitors have wasted their time, but I don't think that's true either.
        davidw 17y ago
        > Netflix ... here For the third time: I am talking about the abstract case, not 'here', and I am talking about the point of view of the competitors, not the entity running the contest. > At best, what you're saying is that other competitors have wasted their time, but I don't think that's true either. They haven't won anything, and unless there are externalities, they have wasted their time. How can they not have? They received nothing. So, what I said is true: the contest is a "zero" or "fixed" sum game, and the other competitors gain only through externalities. You can downmod that all you like, but this more or less concurs with the definition of "zero sum game" in wikipedia.
- mquander 17y ago
  Given that further work on the dataset has continued to produce rapidly diminishing returns, one would assume that prolonging the contest would be likely to result in the teams being closer, not further apart.
joshfinnie 17y ago
A contest like this one (with such a large prize at stake) should have seen this coming and instituted a different rule for their deadline. Something like "24 hours after the last submission." Therefore you wouldn't have someone sniping the win like it seems has happened.
- Herring 17y ago
  But this isn't an auction. You can't just conjure up a better result once you know you've been surpassed. Belkor etc was probably working just as hard all month & the ensemble just got lucky.

40 points by vang3lis 17y ago

an .0001 difference would seem to me to be 'too close to call' and a reason to reopen and let both teams work for another 30 days to get to a larger than .1 difference. Just like in ping-pong, extend the game if it is that close until there is a clear winner. With a winner takes all contest like this there has to be a clear winner.

vang3lis 17y ago
Yes, the difference is negligible, but you obviously can't change rules of the game when it has already finished.
- jeresig 17y ago
  And to be clear: The winner of the contest is not determined by the number that you see listed on the leaderboard. The number on the leaderboard [1] is on the Quiz dataset, the winner is determined by running against the Test dataset (which is kept private by Netflix). As stated by Yehuda on team BellKor: "our team is top contender for winning the Grand Prize, as we have a better Test score than The Ensemble." This turnaround does not surprise me. BellKor's Pragmatic Chaos took their sweet time getting to 10%+ - and in doing so they were very sure not to overfit [2] the data (making their solution a much more generic and viable solution to the dataset). It's my guess that The Ensemble rushed quickly to 10%+ and overfit their data like mad (which yielded high numbers on the public dataset, but evidently does not translate to a generic solution). I'm looking forward to seeing the final papers published by BellKor, et. al. - they're going to be a fascinating read, regardless. 1: http://www.netflixprize.com/leaderboard http://www.netflixprize.com/leaderboard 2: http://en.wikipedia.org/wiki/Overfitting http://en.wikipedia.org/wiki/Overfitting
  - jedc 17y ago
    A lot of people have been tripped up by this, and it's a VERY important distinction. I hope Netflix posts more about the final (Test) results from each when they officially announce the winner.
  - lliiffee 17y ago
    I don't think that is correct. The numbers on the leaderboard are not the scores on the training set. The netflix prize uses three datasets: the training set, the leaderboard set, and the test set. The training set is distributed to everyone. The test set is totally secret. Access to the leaderboard set is only by submitting results (once per 24 hours) and looking at the results. It is not at all trivial to "overfit" to this leaderboard set. (It could be done by, e.g. submitting results with slight tweaks to the algorithm parameters, but this would take a lot of time since you can only submit every 24 hours. Also, you would basically have to do it consciously.)
davidw 17y ago
Winner takes all contests seem to be zero sum games to me, in a certain sense. The pie is a fixed size, and one guy gets all of it. I think I'd rather sink my time into business or open source stuff.
- hvs 17y ago
  Actually, the top teams have already started negotiating with other companies for the technology that they came up with. Knowledge is never zero-sum.
  - davidw 17y ago
    I wrote contest, not knowledge. The knowledge is external to the contest. Some contests, like this one, have lots of external benefits to the top guys, but "winner takes all contest" in the abstract sense seems to me to be a zero sum game, so I stand by what I said: it's generally better to sink your time into something where the "external stuff" is actually the main benefit.
    - mhb 17y ago
      The term "zero sum" does not really make sense here. Netflix obviously thinks they're getting more than what they're spending on the contest and the winning team obviously thinks they're receiving more than what they've invested. So between those two parties, there is a net gain as is the case in most voluntary transactions. At best, what you're saying is that other competitors have wasted their time, but I don't think that's true either.
      - davidw 17y ago
        > Netflix ... here For the third time: I am talking about the abstract case, not 'here', and I am talking about the point of view of the competitors, not the entity running the contest. > At best, what you're saying is that other competitors have wasted their time, but I don't think that's true either. They haven't won anything, and unless there are externalities, they have wasted their time. How can they not have? They received nothing. So, what I said is true: the contest is a "zero" or "fixed" sum game, and the other competitors gain only through externalities. You can downmod that all you like, but this more or less concurs with the definition of "zero sum game" in wikipedia.
mquander 17y ago
Given that further work on the dataset has continued to produce rapidly diminishing returns, one would assume that prolonging the contest would be likely to result in the teams being closer, not further apart.

joshfinnie 17y ago

A contest like this one (with such a large prize at stake) should have seen this coming and instituted a different rule for their deadline. Something like "24 hours after the last submission." Therefore you wouldn't have someone sniping the win like it seems has happened.

Herring 17y ago
But this isn't an auction. You can't just conjure up a better result once you know you've been surpassed. Belkor etc was probably working just as hard all month & the ensemble just got lucky.