6 ms·
The fallacy in the Netflix algorithm
- deleted 17y ago[deleted]
- deleted 17y ago[deleted]
- frossie 17y agoLet me get this right. Dude doesn't bother to rate movies, and then complains that an algorithm based on movie ratings fails to read his mind? If you keep putting on Criterion discs on your queue and then fail to watch them or rate them, what would you like the poor software to do exactly? Netflix has an excellent method for dealing with the highbrow/lowbrow problem - it is called multiple profiles. If you want some brainless mind candy for Friday night with your mates, make a mindcandy profile on your account. Make a Criterion queue for your cinephile Sunday group. You will find the software does an excellent job of providing you with ratings tuned to your preferences.
- jpwagner 17y agoYou are way off here. The article was not a customer support complaint about his personal account. Sure he could've written this article a helluva lot better, but his point is 100% valid which you implicitly agree with by recommending he open multiple profiles. Remember that this guy is a writer not an engineer. His solution is likely bad! So throw out the idea of basing ratings on times of day or mood, etc. The problem statement is still there and valid: My preferences change over time. How can a good recommendation algorithm account for that? Brainstorm away...
- frossie 17y agoMy preferences change over time. How can a good recommendation algorithm account for that? I'm confused. You just re-rate the discrepancies - I do that routinely (some movies seem okay after you watch them, then a few days later you realise they were eminently forgettable). The Netflix UI makes this easy by showing you the movies you recently rated for a while afterwards. I'd be pretty surprised if the recommendation algorithm didn't have a time weighting in it too, counting the movies you have rated recently more than the ones you have rated in the distant past. You also seem to think that multiple profiles is a workaround - well to me, it's a really good solution to that kind of problem. When Netflix announced they were going to pull the feature I was one of the many people who complained, and was very happy when they reversed themselves. "Remember that this guy is a writer not an engineer. His solution is likely bad! [..] The problem statement is still there" But is it? I don't have any personal evidence that it is, and dude says it is but he doesn't rate movies so how can I know? Is it for you? Do you find that Netflix recommends movies you would have liked 5 years ago but no longer do so because your tastes have changed? If so then yes, we have a problem. I do have a problem with the Netflix recommendation algorithm in that I don't feel it deals well enough with klunkers - just because I enjoyed 5 Hugh Jackman movies does not mean I will enjoy Van Hesling, because Van Helsing was a really bad movie and would not have been a good movie even with the second coming of James Dean. The solution to that is pretty simple technically though.
- brg 17y agoI don't think the gp is far off. The author of the article clearly says that he places movies in his queue which he does not watch. If he provided truthful responses to Netflix (not interested, not-enough-stars-to-order-again) regarding these then the algorithm could be given a chance. But he doesn't use the algorithm, and instead chooses movies the he won't watch. And then complains that Netflix will not populate his recommendations with more movies he doesn't want to watch. Non-linear functions (such as mood over time) and non-static behavior (attitude adjustment to one title due to exposure to another) can be accounted for in the Netflix recommendation algorithm.
- kevinpet 17y agoPrecisely -- his opinion about netflix is based off misusing it. If he provided reasonable ratings, only ordered the types of movies he wants to watch, and then said "it isn't working because my moods change", then I'd pay a little more attention.
- jpwagner 17y agoOk, the author is a bit of a moron. He said stupid things and detracted from what I am guessing his point is. He also didn't read about the Netflix algorithms and how they actually work (all around terrible job of reporting indeed.) However, to circle back, what I think is his point is interesting: how do you model "mood over time" and "non-static behavior". You say they "can be accounted for" and don't talk about how. In the winner's documentation they say they treated "user preference" variation over time the same as chunked up "user bias" over time (they define these terms in the paper.) While that obviously performed well numerically, isn't it believable that this is not an ideal solution, but one done under severe time constraints? The problem is by no means solved...that's all I'm saying.
- roundsquare 17y agoIf we just take his examples though, then the problem is on its way to being solved. So he puts movies on his queue that he doesn't watch? Great, he should give them low rankings. If the algorithm is good, that problem is solved. He likes some movies at night, some on the weekends and some with his wife? Well, okay, rate the movies how you thought they were when you saw them. The algorithm will recommend some movies for each situation. Your preferences change over time? Thats a bit harder, but was also out of the scope of the challenge. They were given static data. If you want to account for changes over time, then give the ratings some weighting that decreases over time or only use the last x months of data. But, like you said, he probably doesn't know much about the algorithm.
- pmjoyce 17y ago"I haven't devoted nearly enough time to figuring out the difference between four stars and five stars" I grappled with a similar issue a few months ago when I wrote about the 5 star rating system used by Yelp. If time and mood are factors in evaluating a movie at a point in time (when the movie itself doesn't change over time) then the issue is muddled further when using a 5-star rating system for bar or restaurants where the same person can have wildly different empirical experiences in the same day. It seems to me that it's almost impossible to capture all the variables necessary to accurately capture and assess how you feel about something at a given point in time (movie/restaurant/whatever). Even if we could, would enough people want to go though the doubtless cumbersome and time consuming process of collection enough of the time to draw any meaningful conclusions?
- wglb 17y agoThoughtful article, but is "the software tries to predict how I'll feel about movies based on my past ratings" this strictly true? Might it not be a broader predictor of what you might like in a changing landscape? And most seriously, it is not about you, it is really about netflix. That is to say, it may not totally please your every whim, but if it improves netflix predictions over all of its users, it is a big win for them.
- kqr2 17y agoAs Netflix streams more movies, it might give them the ability to better predict when you might like to watch certain types of movies.
- larryfreeman 17y agoHe's not correct about the Netflix Prize algorithms. The leading ones all assume time-effects so that your rating on one day is not the same as on another day. For the explanation of the winning team's view on time-effects as of 2008, see here: http://research.att.com/~volinsky/netflix/Bellkor2008.pdf http://research.att.com/~volinsky/netflix/Bellkor2008.pdf For a simpler example of a time-based global effect, (see User x Time Effect, for example): http://algorithmsanalyzed.blogspot.com/2008/05/bellkor-algorithm-global-effects.html http://algorithmsanalyzed.blogspot.com/2008/05/bellkor-algor...
- durana 17y agoSome of the new models that came out of the Netflix Prize do take into account temporal changes. From a user perspective, it seems to be how a user's rating style can change over time and how a user's taste in movies can change with time. So the author is wrong when he says the models don't take into account temporal changes. The author suggests that the decision making process around what movie to watch in the future is different from the process of deciding what movie to watch right now. None of the models I've seen involved with the Netflix Prize try to take this into account as far as I can tell. I'm not sure how being able to make recommendations tailored to these two different decision making processes would be of value, but it would be different from the existing published models.
- ovi256 17y agoI would also recommend Yehuda Koren's (Bellkor leader, if I recall correctly) KDD 09 excellent paper, "Collaborative Filtering with Temporal Dynamics" http://research.yahoo.com/pub/2824 http://research.yahoo.com/pub/2824 which speaks exclusively of how to handle temporal dynamics.
- hc 17y agoim glad i saw this. the linked paper [ http://www.csbmb.princeton.edu/ncc/PDFs/Neural%20Economics/McClure%20et%20al%20(Science%2004).pdf http://www.csbmb.princeton.edu/ncc/PDFs/Neural%20Economics/M... ] is quite intriguing and seems relevant to a lot more than netflix
- sh1mmer 17y agoI disagree with the article. If his point has merit it might be that the data for the Netflix algorithm is the wrong or, more probably, incomplete. That doesn't invalidate it, rather it means the factors are more complex. Netflix can't know that he leaves Chaplain DVDs on his player unwatched for a week, but it does know that he returns low-brow stuff faster. That faster turn-around probably indicates a behavior. If that data wasn't exposed in the prize, then maybe Netflix will look at exposing it for the next prize, or adding it themselves.