4 ms·

This is a very helpful blog post from about a year ago. We implemented something similar to a Wilson score to power the default sorting for the millions of rev

This is a very helpful blog post from about a year ago. We implemented something similar to a Wilson score to power the default sorting for the millions of reviews we have at RateItAll. In general, with a system like this, the items with the most reviews are going to rise to the top. This might not be what you want. You can easily see this effect with something like our "Events of 2010" list: http://www.rateitall.com/t-2987869-events-of-2010.aspx http://www.rateitall.com/t-2987869-events-of-2010.aspx

Perceval 17y ago
>items with the most reviews are going to rise to the top Yes and no. Anything with a larger N is going to make for greater confidence in the rating. But that rating still depends on the number of positive and negative reviews. So if a product has a large number of ratings, but they are 50/50 positive and negative, then it's not going to rise to the top simply because it has the most reviews. The algorithm would presumably be very confident about placing it in the middle of the pack.
- Automatt-2 17y ago
  >So if a product has a large number of ratings, but they are 50/50 positive and negative, then it's not going to rise to the top simply because it has the most reviews. What I'm saying is that it will rise to the top because even though only 50% of the ratings are positive, it will have many many more positive ratings than the nearest next item. That 2nd item may have a much higher percentage of positive ratings, but a much lower number of total ratings.
  - DougBTX 17y ago
    It seems to work as described in the article: require 'rubygems' require 'statistics2' def ci_lower_bound(pos, n, power) if n == 0 return 0 end z = Statistics2.pnormaldist(1-power/2) phat = 1.0*pos/n (phat + z*z/(2*n) - z * Math.sqrt((phat*(1-phat)+z*z/(4*n))/n))/(1+z*z/n) end puts ci_lower_bound(60, 100, 0.10) # => 0.517809505446319 puts ci_lower_bound(500, 1000, 0.10) # => 0.474027691168875 puts ci_lower_bound(50, 100, 0.10) # => 0.418847795168265 The top ranked result has 60 positives out of 100, and beats 500 out of 1000 - almost 10 times as many positive votes.
frederickcook 17y ago
A couple of notes there: first the item at the top of the list has more than an order of magnitude more ratings than the rest. For most ordered lists, this is probably an edge case that may have to be dealt with separately. Finally, I've got to say, 26 reviews (for the #2 item) doesn't seem insignificant, and the #2 item seems to have something like a 15% higher _average_ rating. Also, on another trending rating (http://www.rateitall.com/t-3239938-2010-ncaa-tournament-teams.aspx http://www.rateitall.com/t-3239938-2010-ncaa-tournament-team...), the #2 has a significantly higher rating (17-18% it looks like), but only one less review: 4 vs. 3. I think if I were sorting either of these based on the average rating and the # of ratings, I would have done both differently. Based on this, it seems that the Wilson score probably _over-emphasizes_ sample size, especially on things like ratings on high-traffic internet sites that may have orders of magnitude swings in the number of ratings. It seems like the Amazon method of average works just fine, except for items with very few ratings possibly receiving disproportionately high ratings. Again, edge cases, which should probably just be penalized manually.

Perceval 17y ago

>items with the most reviews are going to rise to the top Yes and no. Anything with a larger N is going to make for greater confidence in the rating. But that rating still depends on the number of positive and negative reviews. So if a product has a large number of ratings, but they are 50/50 positive and negative, then it's not going to rise to the top simply because it has the most reviews. The algorithm would presumably be very confident about placing it in the middle of the pack.

Automatt-2 17y ago
>So if a product has a large number of ratings, but they are 50/50 positive and negative, then it's not going to rise to the top simply because it has the most reviews. What I'm saying is that it will rise to the top because even though only 50% of the ratings are positive, it will have many many more positive ratings than the nearest next item. That 2nd item may have a much higher percentage of positive ratings, but a much lower number of total ratings.
- DougBTX 17y ago
  It seems to work as described in the article: require 'rubygems' require 'statistics2' def ci_lower_bound(pos, n, power) if n == 0 return 0 end z = Statistics2.pnormaldist(1-power/2) phat = 1.0*pos/n (phat + z*z/(2*n) - z * Math.sqrt((phat*(1-phat)+z*z/(4*n))/n))/(1+z*z/n) end puts ci_lower_bound(60, 100, 0.10) # => 0.517809505446319 puts ci_lower_bound(500, 1000, 0.10) # => 0.474027691168875 puts ci_lower_bound(50, 100, 0.10) # => 0.418847795168265 The top ranked result has 60 positives out of 100, and beats 500 out of 1000 - almost 10 times as many positive votes.

frederickcook 17y ago

A couple of notes there: first the item at the top of the list has more than an order of magnitude more ratings than the rest. For most ordered lists, this is probably an edge case that may have to be dealt with separately. Finally, I've got to say, 26 reviews (for the #2 item) doesn't seem insignificant, and the #2 item seems to have something like a 15% higher _average_ rating. Also, on another trending rating (http://www.rateitall.com/t-3239938-2010-ncaa-tournament-teams.aspx http://www.rateitall.com/t-3239938-2010-ncaa-tournament-team...), the #2 has a significantly higher rating (17-18% it looks like), but only one less review: 4 vs. 3. I think if I were sorting either of these based on the average rating and the # of ratings, I would have done both differently. Based on this, it seems that the Wilson score probably _over-emphasizes_ sample size, especially on things like ratings on high-traffic internet sites that may have orders of magnitude swings in the number of ratings. It seems like the Amazon method of average works just fine, except for items with very few ratings possibly receiving disproportionately high ratings. Again, edge cases, which should probably just be penalized manually.