3 ms·
It's very nice to see a mathematically motivated piece on how to sort based on users ratings. But there are two problems with the solution presented, the first
by JustSemantics 17y ago
It's very nice to see a mathematically motivated piece on how to sort based on users ratings. But there are two problems with the solution presented, the first minor, and the second more serious.
Minor: Distributions of ratings are not usually normal about the average possible rating -- they are usually skewed to the positive end, because people are pretty good at selecting what they read or view. The solution presented will tend to skew low-frequency ratings too far toward the average. It should be easy to calculate an expected average for your site, and use this instead of 1/2 for the formula given in the post.
Major: What you really need is a missing data point: the number of readers who haven't made any rating at all. It's pretty clear if you think of two examples:
Item 1
1,000,000 views
10 positive ratings
1 negative rating
Item 2
15 views
10 positive ratings
1 negative rating
Clearly, readers feel pretty blah about Item 1. You're much better off displaying Item 2, other things being equal, but the formula doesn't distinguish the two cases.
- yish 17y agoI agree, especially that the skew needs to be site dependent. For example the iPhone app store actually has the opposite problem where their rate on delete "feature", skews the average ratings way down. Reviews are heavily distributed along 1 star or 5 star ratings. Would also be curious to see a comparison of different rating systems, for example i'd like to see what the distribution is on a site thumbs up/down rating site like digg vs a 5 star system.
- JustSemantics 17y agoOne data point is the netflix rating set from their contest. The system is 1 to 5 stars. The average movie rating was 3.4 in 1998 climbing to 3.8 in 2005. Back to the original point -- if you used a formula with a default rating of 3 it would underestimate low-frequency movies. People don't view movies at random. They try to pick things they will like, which should be reflected in the statistical model.
- yetanotherlogin 17y agoThe article does actually cover your serious point. > Indeed, it may be more useful in a "top rated" list to display those items with the highest number of positive ratings per page view, download, or purchase, rather than positive ratings per rating. Many people who find something mediocre will not bother to rate it at all; the act of viewing or purchasing something and declining to rate it contains useful information about that item's quality.