18 ms·
When the 'Wisdom of Crowds' turns on itself: IMDB Edition
- Tichy 18y agoMaybe the majority of people would indeed get more entertainment out of "The Dark Knight" than out of "The Godfather"?
- william42 18y agoYou're right. However, if "The Dark Knight" bumping off "The Godfather" was all that was happening, that wouldn't be a problem. What's happening, though, is that the release of "The Dark Knight" made "The Shawshank Redemption" have a higher rank than "The Godfather". This means that the ranking system has lead to the rating system being used strategically. The implications of this for approval voting would be an interesting corollary. The polls could influence themselves.
- deleted 18y ago[deleted]
- wmf 18y agoIf you allow people to vote after they have seen the rankings, you have the wisdom of mobs. The only way to get the wisdom of crowds is to ask people before they have seen other people's opinions.
- breck 18y agoBut it only takes 1 spoiler to publicize the results to ruin it for most everyone else.
- Hexstream 18y agoI upmodded your post because it already had +9!
- malekov 18y agoi upmoded you because you were +7 and him because he was +14. CAN I HAS MOB NOW KTHXBYE
- cstejerean 18y agoI wonder what would happen on this site if the comments and stories were still sorted by points but the actual number was hidden (or perhaps only hidden until after you vote but I don't see why they need to be there period).
- chandler 18y ago>> (or perhaps only hidden until after you vote but I don't see why they need to be there period) That's an interesting point--why does every site need to quantify for the user the particular score of a comment? Why not indicate popularity by simply ordering the comments? Is it a trust issue?
- noonespecial 18y agoIt seems like in this case we have a temporal error. A popular movie generated buzz now and skewed a result that people pronounce loosely as of all time. Batman might be the best movie out recently but in 2 years time, it will be a dim and fading memory while the Godfather will retain its appeal as it has already proven that it can. IMDB just needs a temporal filter to keep the latest trend from skewing the universal result.
- breck 18y agoThis mob mentality, brought on by a significant event (in this case, the release of The Dark Knight), causes a total breakdown in a system that is otherwise fairly accurate, useful, and trustworthy. I wouldn't call this a "total breakdown in the system". I would call it a minor blip. I mean, ultimately the system recommends what movies to watch. If someone isn't going to watch the Godfather now that it's ranked #3 instead of #1, then the person is flawed, not the system. Otherwise, this was an interesting post. I have been astonished that TDK is #1. Didn't realize people were also downvoting other movies to keep it there.
- william42 18y agoThe flaw in the system isn't that The Godfather was bumped down. The flaw is that people are rating the Godfather down to the point at which it changed its ranking with respect to other movies. And you're right, it's not a major flaw for a movie ranking system. But let's suppose something similar to the IMDB system(which would be an approval/range voting system) is used in an election, and the first place gets President, the second choice getting Veep. With an election, you don't have real-time rankings like IMDB, but you have something similar--polls. And that's even worse, since there's a few points of inaccuracy. Strategically worded polls could change the results of an election.
- unalone 18y agoBut that change is a blip in and of itself. Godfather has been bumped down before. Happened with Lord of the Rings. It climbs back up after the initial wave.
- brokenladder 18y agoScore Voting (aka Range Votiing) and Approval Voting are actually the best voting methods, especially when you take strategic voting into account. http://rangevoting.org/StratHonMix.html http://rangevoting.org/StratHonMix.html In this case, even if the election result changed, you'd still get a candidate who was rated relatively well by the average voter, so the results would not be that much worse. Now consider that other voting methods react much worse to strategic voters, _and_ that score voting is as good with a great number of strategic voters as other systems are with 100% honest voters, and this whole argument falls flat.
- deleted 18y ago[deleted]
- attack 18y agoThe point of top-k rankings is to put items in the proper order. As far as the crowds are concerned, this is no failure. Sorry your favorite movie lost its spot.
- andreyf 18y agoThat's funny, because the book which coined the term is about exactly this - how to make Wisdom of the Crowds work.
- wmf 18y agoThe problem is that people discuss these concepts without reading the book and thus miss out on the subtleties. The long tail is another common victim. Disclaimer: I didn't read the book.
- davidw 18y agoI'm (slowly) working on the summary here: http://www.squeezedbooks.com/book/show/23/the-wisdom-of-crowds http://www.squeezedbooks.com/book/show/23/the-wisdom-of-crow... Contributions welcome!
- greenagain 18y agoIIRC, the same thing happened at the releases of Spiderman and the Royal Tenenbaums. Hype does funny things to men.
- pg 18y agoHave they ruled out the possibility that the people marketing the movie are gaming the system? Even if voting was restricted to established IMDB users, surely a reasonably well connected Hollywood marketing person could round up a few hundred of those among friends and friends of friends.
- rms 18y agohttp://www.imdb.com/title/tt0468569/ratings http://www.imdb.com/title/tt0468569/ratings is the vote breakdown.
- pg 18y agoHmm, those numbers are definitely too big to be influenced by anyone's network. Judging from the other top-ranked films, it looks as if IMDB's userbase is in the middle of changing.
- ChaitanyaSai 18y agoI don't think they need to game the system. The lukewarm or negative reviews on RottenTomatoes are being bombarded with genuine fanboy vitriol. (Many of the commenters, at least on the first page of comments, have joining dates that suggest they have been on RT for a while) http://www.rottentomatoes.com/m/the_dark_knight/?critic=creamcrop#mo http://www.rottentomatoes.com/m/the_dark_knight/?critic=crea... It is quite amusing in a sad way actually, seeing them trash the likes of David Denby and Joe Morgenstern.
- unalone 18y agoTo be fair, those reviews were both fairly terrible. Fanboyism is never justified, in my opinion, but I agree with the respondants' sentiments.
- wheels 18y agoIMDB's ranking system is one of the least sophisticated out there, which is surprising given that they're owned by Amazon these days. I assume it's still just the same couple of guys that started it left to run it in relative independence. You really don't need to game their system much; it's just bad technology.
- iamelgringo 18y agoI think that the problem is similar to the eternal September problem. I'm sure that the people who had been voting and contributing to the ratings on IMDB had been involved on the site for a while. Now, there's a big influx of people on IMDB that are voting for their favorite movie. I bet that many of the people who are voting Dark Knight up have never seen the Godfather or Shawshank. Same thing happened to my wife in film school. She went to film school because she loved the classics of the 30's - 50's. She was stunned to find out that the kids she was going to school with thought that Austin Powers was the best film ever.
- ericb 18y agoWisdom of Crowds was a good book. The sequel, Attention Span of Crowds, was significantly shorter.
- bprater 18y agoPerhaps this is an advantage to Y!Hack not having a downvote button.
- Hexstream 18y agoI'm much less worried about that happening here because of the transient nature of the site. I don't think potential manipulators would find it worthwhile.
- bprater 18y agoShould be easy to cull these votes from the system. Anyone voting Dark Knight at 10 and Godfather at 1 should get nixxed from the system. In fact, anyone voting Godfather less than 8 should be nixxed.
- delackner 18y agoThis reminds me of the recent articles about the netflix optimization challenge. Voting systems like imdb might be improved with some simple heuristics that take into account the absurdity of some people's votes. If a user makes votes that are < 2.0 for a movie that the entire community for years has rated above 9.0, then probably that user's votes should all be ignored, because they are totally uninformed.
- william42 18y agoBecause having unpopular taste is bad?
- eru 18y agoEvil.
- thwarted 18y agoWhat if you assume everything is average and you give more weight toward the finally rating for those individual ratings that are further from the norm? This means people couldn't game it by following the crowd, following the crowd gives the entire crowds rating less weight.
- unalone 18y agoSee, but it's all about opinion. Some people honestly like Dark Knight better, because it's far more gripping and easier to get into. And those people have every RIGHT to give Godfather a 1, if people that gave Godfather a 10 are giving The Dark Knight a 1 in return. The Godfather people started this days before the Dark Knight people came in.
- dangoldin 18y ago
- bryanlanders 18y agoIf you think of this system as a game and in this instance a number of users have figured out a way to act within the rules of the game to achieve an outcome inconsistent with IMDB's intended one, then IMDB can simply change the rules to get what they think is best! It wouldn't be too difficult to observe the behavior of the offending members (vote for TDK, within x hours vote down Godfather...) and simply setup criteria to disallow these actions. You could also set up an algorithm to detect screwy behavior to alert a developer or block that action for a duration. And hey, we throw error messages for repeated button clicks and such so that click-happy users or spam bots don't take down the system...can't that kind of precaution be extended if necessary? "We know you're in love with TDK, but do you really hate The Godfather...come back in a month if you still do!"
- william42 18y agoSoon you get websites with people saying "Rate The Godfather 1 before rating TDK 10."
- MikeCapone 18y agoAFter the first weekend, Dark Knight had 9.7 average with almost a hundred thousand votes. That's just crazy.
- unalone 18y agoNah. 9.7 was with only 3,000. 100,000 only hit a few days ago.
- rrf 18y agoThe problem is not the Dark Knight fans, so much as the algorithms being used to rank the movies. How can you take advantage of crowd wisdom if you don’t control for spoofing, spam and mob voting? It’s surprising that there wasn’t already in place a process for dealing with such a common phenomenon.
- Erwin 18y agoBlockbusters popular with the avid Internet users have always started at incredibly high ratings only to slowly fally to the right place. The LOTR movies were in top-10 once too. I suppose IMDB might try to account for the initial voting wave being raving fans and dampen the initial votes, but why would they? Personally I've reloaded the "Dark Knight" IMDB page half a dozen times to see if it was still at #1. The most popular topic on the IMDB message boards is "This does not deserve to be in top 250" or "Why is this not in Top 250 why some-other-movie is not?". People like to keep an eye on rankings and to comment on them. The main practical use for the ratings IMHO is to get a list of movies within each genre sorted by the rating, to see if you missed something people thought was great.
- ragaskar 18y agoexactly -- imdb has an internet and populist skew. As long as you take this into account, their ratings are fine and predictable. Gaming rankings is practically an official internet pastime, so it's not unexpected that this event is taking place, although I'm surprised it built up enough steam without (presumedly) official co-ordination to drop the Godfather a few places -- that is notable. I'm sure politicians are taking notes as we speak.
- johnyzee 18y agoI don't see why it wouldn't be possible to build into the system some of the same logic that obviously makes it clear to everyone that there is a problem with these votes. A computer might not be able to appreciate the goodness of The Godfather, but it can be taught that a movie that has consistently been voted great for decades, does not suddenly fall out of favor in a month.
- flipbrad 18y agoover time it should return to a 'fair' position, once temporary disequilibrium in the 'crowd' (induced by marketing, hype, whatever) is resolved. It's a shame people vote 'strategically', not to reflect their true sentiments - but whatever. Over time, with a diverse enough crowd, 'wisdom of the crowd' systems are effective and reasonably unbiased. doubtless IMDB will enjoy the free publicity.
- Eliezer 18y agoI've seen The Godfather. I've seen The Dark Knight. Now, maybe I'm just missing the point of that Godfather thing, but I'd say The Godfather had more dramatic tension and The Dark Knight had more philosophical depth. No, stop laughing, I'm serious. Tell me what point The Godfather makes.
- snorkel 18y agoI'm astonished. The internet hordes usually have impeccable taste in film and music. "Beverly Hills Chowahwah" is not even in the top 20?!!