4 ms·
I definitely go by the 4.5 stars == very good, <4 stars == crap heuristic, but to argue this is no good is ridiculous. It's actually very, very helpful. E.g. w
by podperson 14y ago
I definitely go by the 4.5 stars == very good, <4 stars == crap heuristic, but to argue this is no good is ridiculous. It's actually very, very helpful.
E.g. when I go to Amazon I don't buy some random product with a 4.5 star review -- I search for a specific product or a specific kind of product and then reject candidates which are lousy. How is that not INCREDIBLY useful? Similarly, who goes to a movie simply based on whether it's good or not.
In general, if you create any point rating system people who like a thing will tend to rate it towards the top of the scale, e.g. 4/5 or 9/10.
I actually did an informal experiment -- I used to run role-playing tournaments, and do exit surveys on participants. For the first few years we asked players to rate us on a 5-point scale and scored slightly over 4/5 on average. Then we switched to a 10-point scale and scored slightly over 9/10. Not scientific -- but I don't think we suddenly got better.
This finding is backed up by serious research (which is why when a psychologist creates a scale, the numerical ranges need to stay constant in follow-up studies or the results are not statistically comparable).
Netflix, which tries to give users customized ratings, actually subtracts value (in my opinion) from its scores because it tries to make ratings mean "how much will you enjoy this?" BZZZT. I pick stuff for me, my wife, my au pair, and my kids. We don't all like the same stuff, and we don't want to track ratings individually. My kids want good kid stuff. I want good me stuff. Don't try to guess what I like based on our collective tastes.
- CWuestefeld 14y agoThe problem that XKCD gets at is simply translating/scaling the results; the article is solving a different problem. Early on in the Netflix Challenge, I was able to get myself (very briefly) a leaderboard score with nothing more than analyzing every user's ratings; re-centering them by their mean, and re-scaling them according to their standard deviation. The by remembering their translations and scales, I could put a globally-predicted score back into their own language. So just some very basic statistics is sufficient to erase much of the bias toward higher numbers, as well as halo effects and the like. (I was pretty surprised that Netflix's own algorithm apparently wasn't doing anything this simple)
- CamperBob2 14y agoI was pretty surprised that Netflix's own algorithm apparently wasn't doing anything this simple Netflix does have really interesting blind spots. They claim to take ratings seriously, to the point of offering a million dollars for the best rating algorithm. Then, as the GP says, they implement the rating algorithm in a way that renders it completely worthless to any household with more than one viewer. Netflix does offer us a good demonstration of the failings of absolute technocracy, but it leaves the question of how best to rate movies wide open.
- saraid216 14y agoI never actually look at the average star rating. I read a couple 5-star reviews, read more 4-star and 2-star reviews, and decide based on that. Considering how many Amazon 1-star reviews I find that can be summed up as, "UPS sucked," averages are kinda useless.