10 ms·

How Not to Sort by Average Rating (2009)

150 points by jmilloy 11y ago

bbrazil 11y ago
Previously: https://news.ycombinator.com/item?id=478632 https://news.ycombinator.com/item?id=478632
- ggreer 11y ago
  It was also discussed about 3 years ago: https://news.ycombinator.com/item?id=3792627 https://news.ycombinator.com/item?id=3792627 Some of the comments from that posting give concrete examples where the formula fails. Such as: an item with 1000 upvotes and 2000 downvotes will get ranked above one with 1 upvote and 2 downvotes. This is because the formula uses the lower bound of the Wilson interval.
stdbrouw 11y ago
Also discussed in Cameron Davidson-Pilon's Bayesian methods for Hackers in the context of Reddit ups/downs: http://nbviewer.ipython.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Chapter4_TheGreatestTheoremNeverTold/Chapter4.ipynb#Example:-How-to-order-Reddit-comments http://nbviewer.ipython.org/github/CamDavidsonPilon/Probabil... For Amazon, though, which is the example in Evan Miller's post, I don't really get why you'd first dichotomize the five-star rating into positive vs. negative and then use Wilson intervals. Just construct a run-of-the-mill 95% confidence interval for the mean of a continuous distribution and sort by the (still plausible) worst case scenario a.k.a. the lower bound of that: `mean - 1.96 * SE`, where the standard error is `SE = stddev(scores)/sqrt(n)`. Because of the central limit theorem, you can do this even if scores are not normally distributed and it'll work out too.
- vcdimension 11y ago
  For better accuracy with small samples you could use the multinomial distribution instead. The covariance matrix for the rating probabilities can be found here for example: http://www.math.wsu.edu/faculty/genz/papers/mvnsing/node8.html http://www.math.wsu.edu/faculty/genz/papers/mvnsing/node8.ht... Then the variance for the expected rating can be calculated as a weighted sum of the values in the covariance matrix. These companies really should be hiring statistics consultants instead of relying on the intuitions of their programmers.
  - stdbrouw 11y ago
    I'd prefer to just treat scores as continuous and correct using `t_ppf(.975, n-1)` instead of the normal approximation (1.96) but I suppose working from a multinomial distribution would give pretty similar results.
    - vcdimension 11y ago
      You're still relying on the central limit theorem (i.e. a reasonable amount of data) : using t instead of z just corrects for the fact that you only have sample variances instead of population variances. However, I suppose it's not unreasonable to assume that the ratings are likely to have a bell shaped distribution (which could be checked), so the normal/t approximation is probably going to be OK.
      - stdbrouw 11y ago
        Ah yes, true. Let's call it a bias/variance tradeoff ;-)
woah 11y ago
Wrong solution #1 sounds like it could work quite well for UrbanDictionary, since it would tend to reward posts that have a lot of engagement. It's probably a good solution for a lot of sites.
- Udo 11y ago
  I agree. It's a good solution for all cases where the intent is to have a negative vote exactly cancel out a positive. The method in the article combines a quality rating with a quantity rating, but it's a bit unwieldy and difficult to tune intuitively. It seems to me for a lot of purposes you might get a sufficiently similar effect by using method #1, and then multiplying the result with the sigmoid function applied to the ratio. The advantage of this would be you that the only magical numbers in the formula would be tuning factors you put in yourself. This seems more appealing to me than "((positive + 1.9208) / (positive + negative) - 1.96 SQRT((positive * negative) / (positive + negative) + 0.9604) / (positive + negative)) / (1 + 3.8416 / (positive + negative))*"
  - stdbrouw 11y ago
    YMMV, but I'd rather use an elementary and mathematically sound statistical technique than macguyvering something myself. (Though I do understand that, as some people describe in this thread, there can be different purposes to ratings and hence a need for different sorting mechanisms.) You're right that confidence intervals depend on both quality and quantity, but the reason for this is to account for uncertainty. As n goes up, the standard error goes down to practically 0, and so e.g. a 4.2/5 movie with 100 reviews is still likely to be sorted higher than a 4.1 movie with 200 reviews. Quantity only comes into play when there is very little information to go on, after that quality becomes the driving factor.
    - Udo 11y ago
      It's not about macguyvering something, it's about reflecting on how well different solutions stack up with what you actually want to do. There is no one-size-fits-all ranking formula, at least in my opinion.
spacemoelte 11y ago
I have always wondered what amazon was thinking with that way of sorting. Perhaps it's a deliberate way to spread purchases out over a span of products instead of just the two top products?
- randomtree 11y ago
  I think it's about a product discovery. If we always sort this way, new products don't have a chance. And I don't think Amazon would sort like this, it would make more sense for them to use hn/reddit way to sort items that give a chance for the new items to get to the top.
fahadalie 11y ago
The most important criteria for sorting should be the 'engagement'.
- dredmorbius 11y ago
  That's a useful element to consider, and I do favour implicit ranking inputs, but "best" requires you know your goal. What are you selecting for?
deleted 11y ago
[deleted]
Houshalter 11y ago
There is a much simpler and elegant method. Just rank posts by their probability of getting an upvote. This is just (upvotes+1)/(upvotes+downvotess+2).
- stdbrouw 11y ago
  This gives an advantage to new posts for which the probability is much more uncertain: it's easier to get 1 upvote and 0 downvotes (rank 2/3) than to get 1999 upvotes and 999 downvotes (also rank 2/3). Maybe that's what you want, but the post is exactly about those cases when this is not what you want.
  - dredmorbius 11y ago
    New posts frequently start with a disadvantage though in existing systems. Temporarily biasing them favourably increases odds of any moderation. Alternatively you could present them only to a subset of readership. I've suggested this as a solution to HN's new submissions queue problem. Increase the presentation as ratings increase.
  - Houshalter 11y ago
    You can of course use a better prior if you don't think new posts are 50% likely to receive an upvote next. Otherwise I don't see how that behavior is undesirable or against the goal of the article.
  - fludlight 11y ago
    How about fn(upvotes, downvotes)/fn(pageviews, prominence)?
- anon4327733 11y ago
  That is true for a uniform prior but not a general prior.
jrochkind1 11y ago
Interestingly, I _think_ the Reddit algorithm basically makes this mistake too -- although embedded in a more complicated algorithm that combines with 'newest first' altered by positives minus negatives. I don't think the HN algorithm is public, but wouldn't be surprised if it does the same. Perhaps the generally much smaller number of 'votes' on a HN/reddit post makes it less significant.
- gsteinb88 11y ago
  For posts, I'm not sure what the algorithm is (I think it's deliberately more complicated, and has to take into account time of posting?), but after this article [the op] was written, reddit implemented the method for comments, as explained by Randall Munroe: http://www.redditblog.com/2009/10/reddits-new-comment-sorting-system.html http://www.redditblog.com/2009/10/reddits-new-comment-sortin... You only get this ranking method if you sort the comments by 'best' though
  - jrochkind1 11y ago
    Not the default 'top' though, I think.
danialtz 11y ago
I have fundamental problem with democratic voting systems. Whatever the general view likes will tend to come on top, hence cat pictures on reddit. The most philosophiycally elegant solution I've encountered so far is the "quadratic voting" (see https://news.ycombinator.com/item?id=9477747 https://news.ycombinator.com/item?id=9477747), where every user has a limited number of credits to spend per time-period. Every vote will have a quadratic cost. Assume user has/obtains 1000 karma points a month. If the user merely likes a post or not he gives his 1 vote, which cost him 1 karma. If he strongly wants one post up he can spend a maximum of 31 vote on it. This way the minorities will have also extra influence on the voting process. The requirements is that each user have 1 account, e.g. maybe by some form of payment for 1000 karma to avoid fake voting fraud. Maybe using bitcoins if it picks up to avoid privacy problems. Do you think this will hinder the workings of sites like reddit?
- Cthulhu_ 11y ago
  > Do you think this will hinder the workings of sites like reddit? I do. The user will think "Well given that I only have X to spend, I'd better be careful what I vote for". That will lead to less votes being placed, and thus, reduced interaction / community activity. It's the same reason you don't want to cap the amount of submissions or comments someone makes. A flood control (max X per Y time period) to stop bots and spammers, sure, but no hard cap - or an inhumanly high one at least, bearing in mind that some people are indeed inhuman when it comes to voting/commenting.
  - lmm 11y ago
    > I do. The user will think "Well given that I only have X to spend, I'd better be careful what I vote for". That will lead to less votes being placed, and thus, reduced interaction / community activity. It's worth pointing out that Slashdot has run successfully for decades with users getting limited moderation points.
    - dredmorbius 11y ago
      Slashdot's moderation applies to comments only, not stories, and leaves a great deal of good discussion on the floor. And other issues: https://news.ycombinator.com/item?id=9854240 https://news.ycombinator.com/item?id=9854240
rurban 11y ago
I'm ranking movies by critics ratings. Most of them have too low numbers, and thus naive bayesian ranking by avg does not work. IMDB gets away with it, but I cannot. And you should be able to see a good preview of the expected ranking even with low numbers. So you need to check the confidence interval with Wilson, but you also need to check the quality of the reviewer. There are some in the 90%, but there are also often outliers, i.e. extremities. Mostly french btw. I updated the c and perl versions, compiled and pure perl here: https://github.com/rurban/confidence_interval https://github.com/rurban/confidence_interval
discardorama 11y ago
How well does this work when you don't have a binary (+/-) rating system, but a multi-valued one (1 - 5 stars) ?
- cgearhart 11y ago
  See the discussion elsewhere in this thread for confidence intervals in multinomial distributions. https://news.ycombinator.com/reply?id=9856607&goto=item%3Fid%3D9855784 https://news.ycombinator.com/reply?id=9856607&goto=item%3Fid...
- jmilloy 11y ago
  You could replace p/n in the formula with the average star rating and still get a reasonable ranking order.
anon4327733 11y ago
First two points are great, but why then we see this: "Given the ratings I have, there is a 95% chance that the "real" fraction of positive ratings is at least what?" What normal person thinks in terms of confidence intervals? The obvious answer is people want the product with the highest "real" rating. That is the rating the product would get if it had arbitrary many ratings. To get this you just find the mean of your posterior probability distribution. For just positive and negative reviews thats basically (positive+a)/(total+b) where a and b depend on your prior. His proposal would mean that a product with zero reviews would be rated below a product with 1 positive review. This may deal with spam and vote manipulation since things with less information are penalized more but that is a separate issue.
dredmorbius 11y ago
First off, for anyone looking at web reputation systems, read the book on the subject: Randy Farmer and Bryce Glass, Building Web Reputation Systems: Book: http://shop.oreilly.com/product/9780596159801.do http://shop.oreilly.com/product/9780596159801.do Wiki: http://buildingreputation.com/doku.php http://buildingreputation.com/doku.php Blog: http://buildingreputation.com/ http://buildingreputation.com/ I can pretty much guarantee there are elements of this you're not considering which are addressed there (though there are also elements which Farmer and Glass don't hit either). But it's an excellent foundation. Second: If you're going to have a quality classification system, you need to determine what you are ranking for. As the Cheshire Cat said, if you don't know where you're going, it doesn't much matter how you get there. Rating for popularity, sales revenue maximization, quality or truth, optimal experience, ideological purity, etc., are all different. Beyond that I've compiled some thoughts of my own from 20+ years of using (and occasionally building) reputation systems myself: "Content rating, moderation, and ranking systems: some non-brief thoughts" http://redd.it/28jfk4 http://redd.it/28jfk4 ⚫ Long version: Moderation, Quality Assessment, & Reporting are Hard ⚫ Simple vote counts or sums are largely meaningless. ⚫ Indicating levels of agreement / disagreement can be useful. ⚫ Likert scale moderation can be useful. ⚫ There's a single-metric rating that combines many of these fairly well -- yes, Evan Miller's lower-bound Wilcox score. ⚫ Rating for "popularity" vs. "truth" is very, very different. ⚫ Reporting independent statistics for popularity (n), rating (mean), and variance or controversiality (standard deviation) is more informative than a single statistic. ⚫ Indirect quality measures also matter. I should add: a LOT. ⚫ There almost certainly isn't a single "best" ranking. Fuzzing scores with randomness can help. ⚫ Not all rating actions are equally valuable. Not everyone's ratings carry the same weight. ⚫ There are things which don't work well. ⚫ Showing scores and score components can be counterproductive and leads to various perverse incentives. I'm also increasing leaning toward a multi-part system, one which rates: 1. Overall favorability. 2. Any flaggable aspects. Ultimately, "ToS" is probably the best bucket, comprising spam, harassment, illegal activity, NSFW/NSFL content (or improperly labeled same), etc. 3. A truth or validity rating. Likeley rolled up in #2. But worth mentioning separately. 4. Long-term author reputation. There's also the general problem associated with Gresham's Law, which I'm increasingly convinced is a general and quite serious challenge to market-based and popularity-based systems. Assessment of complex products, including especialy information products, is difficult, which is to say, expensive. I'm increasingly in favour of presenting newer / unrated content to subsets of the total audience, and increasing its reach as positive approval rolls in. This seems like a behavior HN's "New" page could benefit from. Decrease the exposure for any one rater, but spread ratings over more submissions, for longer. And there are other problems. Limiting individuals to a single vote (or negating the negative effects of vote gaming) is key. Watching the watchmen. Regression toward mean intelligence / content. The "evaporative cooling" effect (http://blog.bumblebeelabs.com/social-software-sundays-2-the-evaporative-cooling-effect/ http://blog.bumblebeelabs.com/social-software-sundays-2-the-...).
sova 11y ago
Awesome if you are using only "up" and "down" ...
- Houshalter 11y ago
  It should work for star ratings, and generalize to non discrete rating systems too.

150 points by jmilloy 11y ago

bbrazil 11y ago

Previously: https://news.ycombinator.com/item?id=478632 https://news.ycombinator.com/item?id=478632

ggreer 11y ago
It was also discussed about 3 years ago: https://news.ycombinator.com/item?id=3792627 https://news.ycombinator.com/item?id=3792627 Some of the comments from that posting give concrete examples where the formula fails. Such as: an item with 1000 upvotes and 2000 downvotes will get ranked above one with 1 upvote and 2 downvotes. This is because the formula uses the lower bound of the Wilson interval.

stdbrouw 11y ago

Also discussed in Cameron Davidson-Pilon's Bayesian methods for Hackers in the context of Reddit ups/downs: http://nbviewer.ipython.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Chapter4_TheGreatestTheoremNeverTold/Chapter4.ipynb#Example:-How-to-order-Reddit-comments http://nbviewer.ipython.org/github/CamDavidsonPilon/Probabil... For Amazon, though, which is the example in Evan Miller's post, I don't really get why you'd first dichotomize the five-star rating into positive vs. negative and then use Wilson intervals. Just construct a run-of-the-mill 95% confidence interval for the mean of a continuous distribution and sort by the (still plausible) worst case scenario a.k.a. the lower bound of that: `mean - 1.96 * SE`, where the standard error is `SE = stddev(scores)/sqrt(n)`. Because of the central limit theorem, you can do this even if scores are not normally distributed and it'll work out too.

vcdimension 11y ago
For better accuracy with small samples you could use the multinomial distribution instead. The covariance matrix for the rating probabilities can be found here for example: http://www.math.wsu.edu/faculty/genz/papers/mvnsing/node8.html http://www.math.wsu.edu/faculty/genz/papers/mvnsing/node8.ht... Then the variance for the expected rating can be calculated as a weighted sum of the values in the covariance matrix. These companies really should be hiring statistics consultants instead of relying on the intuitions of their programmers.
- stdbrouw 11y ago
  I'd prefer to just treat scores as continuous and correct using `t_ppf(.975, n-1)` instead of the normal approximation (1.96) but I suppose working from a multinomial distribution would give pretty similar results.
  - vcdimension 11y ago
    You're still relying on the central limit theorem (i.e. a reasonable amount of data) : using t instead of z just corrects for the fact that you only have sample variances instead of population variances. However, I suppose it's not unreasonable to assume that the ratings are likely to have a bell shaped distribution (which could be checked), so the normal/t approximation is probably going to be OK.
    - stdbrouw 11y ago
      Ah yes, true. Let's call it a bias/variance tradeoff ;-)

woah 11y ago

Wrong solution #1 sounds like it could work quite well for UrbanDictionary, since it would tend to reward posts that have a lot of engagement. It's probably a good solution for a lot of sites.

Udo 11y ago
I agree. It's a good solution for all cases where the intent is to have a negative vote exactly cancel out a positive. The method in the article combines a quality rating with a quantity rating, but it's a bit unwieldy and difficult to tune intuitively. It seems to me for a lot of purposes you might get a sufficiently similar effect by using method #1, and then multiplying the result with the sigmoid function applied to the ratio. The advantage of this would be you that the only magical numbers in the formula would be tuning factors you put in yourself. This seems more appealing to me than "((positive + 1.9208) / (positive + negative) - 1.96 SQRT((positive * negative) / (positive + negative) + 0.9604) / (positive + negative)) / (1 + 3.8416 / (positive + negative))*"
- stdbrouw 11y ago
  YMMV, but I'd rather use an elementary and mathematically sound statistical technique than macguyvering something myself. (Though I do understand that, as some people describe in this thread, there can be different purposes to ratings and hence a need for different sorting mechanisms.) You're right that confidence intervals depend on both quality and quantity, but the reason for this is to account for uncertainty. As n goes up, the standard error goes down to practically 0, and so e.g. a 4.2/5 movie with 100 reviews is still likely to be sorted higher than a 4.1 movie with 200 reviews. Quantity only comes into play when there is very little information to go on, after that quality becomes the driving factor.
  - Udo 11y ago
    It's not about macguyvering something, it's about reflecting on how well different solutions stack up with what you actually want to do. There is no one-size-fits-all ranking formula, at least in my opinion.

spacemoelte 11y ago

I have always wondered what amazon was thinking with that way of sorting. Perhaps it's a deliberate way to spread purchases out over a span of products instead of just the two top products?

randomtree 11y ago
I think it's about a product discovery. If we always sort this way, new products don't have a chance. And I don't think Amazon would sort like this, it would make more sense for them to use hn/reddit way to sort items that give a chance for the new items to get to the top.

fahadalie 11y ago

The most important criteria for sorting should be the 'engagement'.

dredmorbius 11y ago
That's a useful element to consider, and I do favour implicit ranking inputs, but "best" requires you know your goal. What are you selecting for?

deleted 11y ago

[deleted]

Houshalter 11y ago

There is a much simpler and elegant method. Just rank posts by their probability of getting an upvote. This is just (upvotes+1)/(upvotes+downvotess+2).

stdbrouw 11y ago
This gives an advantage to new posts for which the probability is much more uncertain: it's easier to get 1 upvote and 0 downvotes (rank 2/3) than to get 1999 upvotes and 999 downvotes (also rank 2/3). Maybe that's what you want, but the post is exactly about those cases when this is not what you want.
- dredmorbius 11y ago
  New posts frequently start with a disadvantage though in existing systems. Temporarily biasing them favourably increases odds of any moderation. Alternatively you could present them only to a subset of readership. I've suggested this as a solution to HN's new submissions queue problem. Increase the presentation as ratings increase.
- Houshalter 11y ago
  You can of course use a better prior if you don't think new posts are 50% likely to receive an upvote next. Otherwise I don't see how that behavior is undesirable or against the goal of the article.
- fludlight 11y ago
  How about fn(upvotes, downvotes)/fn(pageviews, prominence)?
anon4327733 11y ago
That is true for a uniform prior but not a general prior.

jrochkind1 11y ago

Interestingly, I _think_ the Reddit algorithm basically makes this mistake too -- although embedded in a more complicated algorithm that combines with 'newest first' altered by positives minus negatives. I don't think the HN algorithm is public, but wouldn't be surprised if it does the same. Perhaps the generally much smaller number of 'votes' on a HN/reddit post makes it less significant.

gsteinb88 11y ago
For posts, I'm not sure what the algorithm is (I think it's deliberately more complicated, and has to take into account time of posting?), but after this article [the op] was written, reddit implemented the method for comments, as explained by Randall Munroe: http://www.redditblog.com/2009/10/reddits-new-comment-sorting-system.html http://www.redditblog.com/2009/10/reddits-new-comment-sortin... You only get this ranking method if you sort the comments by 'best' though
- jrochkind1 11y ago
  Not the default 'top' though, I think.

danialtz 11y ago

I have fundamental problem with democratic voting systems. Whatever the general view likes will tend to come on top, hence cat pictures on reddit. The most philosophiycally elegant solution I've encountered so far is the "quadratic voting" (see https://news.ycombinator.com/item?id=9477747 https://news.ycombinator.com/item?id=9477747), where every user has a limited number of credits to spend per time-period. Every vote will have a quadratic cost. Assume user has/obtains 1000 karma points a month. If the user merely likes a post or not he gives his 1 vote, which cost him 1 karma. If he strongly wants one post up he can spend a maximum of 31 vote on it. This way the minorities will have also extra influence on the voting process. The requirements is that each user have 1 account, e.g. maybe by some form of payment for 1000 karma to avoid fake voting fraud. Maybe using bitcoins if it picks up to avoid privacy problems. Do you think this will hinder the workings of sites like reddit?

Cthulhu_ 11y ago
> Do you think this will hinder the workings of sites like reddit? I do. The user will think "Well given that I only have X to spend, I'd better be careful what I vote for". That will lead to less votes being placed, and thus, reduced interaction / community activity. It's the same reason you don't want to cap the amount of submissions or comments someone makes. A flood control (max X per Y time period) to stop bots and spammers, sure, but no hard cap - or an inhumanly high one at least, bearing in mind that some people are indeed inhuman when it comes to voting/commenting.
- lmm 11y ago
  > I do. The user will think "Well given that I only have X to spend, I'd better be careful what I vote for". That will lead to less votes being placed, and thus, reduced interaction / community activity. It's worth pointing out that Slashdot has run successfully for decades with users getting limited moderation points.
  - dredmorbius 11y ago
    Slashdot's moderation applies to comments only, not stories, and leaves a great deal of good discussion on the floor. And other issues: https://news.ycombinator.com/item?id=9854240 https://news.ycombinator.com/item?id=9854240

rurban 11y ago

I'm ranking movies by critics ratings. Most of them have too low numbers, and thus naive bayesian ranking by avg does not work. IMDB gets away with it, but I cannot. And you should be able to see a good preview of the expected ranking even with low numbers. So you need to check the confidence interval with Wilson, but you also need to check the quality of the reviewer. There are some in the 90%, but there are also often outliers, i.e. extremities. Mostly french btw. I updated the c and perl versions, compiled and pure perl here: https://github.com/rurban/confidence_interval https://github.com/rurban/confidence_interval

discardorama 11y ago

How well does this work when you don't have a binary (+/-) rating system, but a multi-valued one (1 - 5 stars) ?

cgearhart 11y ago
See the discussion elsewhere in this thread for confidence intervals in multinomial distributions. https://news.ycombinator.com/reply?id=9856607&goto=item%3Fid%3D9855784 https://news.ycombinator.com/reply?id=9856607&goto=item%3Fid...
jmilloy 11y ago
You could replace p/n in the formula with the average star rating and still get a reasonable ranking order.

anon4327733 11y ago

First two points are great, but why then we see this: "Given the ratings I have, there is a 95% chance that the "real" fraction of positive ratings is at least what?" What normal person thinks in terms of confidence intervals? The obvious answer is people want the product with the highest "real" rating. That is the rating the product would get if it had arbitrary many ratings. To get this you just find the mean of your posterior probability distribution. For just positive and negative reviews thats basically (positive+a)/(total+b) where a and b depend on your prior. His proposal would mean that a product with zero reviews would be rated below a product with 1 positive review. This may deal with spam and vote manipulation since things with less information are penalized more but that is a separate issue.

dredmorbius 11y ago

First off, for anyone looking at web reputation systems, read the book on the subject: Randy Farmer and Bryce Glass, Building Web Reputation Systems: Book: http://shop.oreilly.com/product/9780596159801.do http://shop.oreilly.com/product/9780596159801.do Wiki: http://buildingreputation.com/doku.php http://buildingreputation.com/doku.php Blog: http://buildingreputation.com/ http://buildingreputation.com/ I can pretty much guarantee there are elements of this you're not considering which are addressed there (though there are also elements which Farmer and Glass don't hit either). But it's an excellent foundation. Second: If you're going to have a quality classification system, you need to determine what you are ranking for. As the Cheshire Cat said, if you don't know where you're going, it doesn't much matter how you get there. Rating for popularity, sales revenue maximization, quality or truth, optimal experience, ideological purity, etc., are all different. Beyond that I've compiled some thoughts of my own from 20+ years of using (and occasionally building) reputation systems myself: "Content rating, moderation, and ranking systems: some non-brief thoughts" http://redd.it/28jfk4 http://redd.it/28jfk4 ⚫ Long version: Moderation, Quality Assessment, & Reporting are Hard ⚫ Simple vote counts or sums are largely meaningless. ⚫ Indicating levels of agreement / disagreement can be useful. ⚫ Likert scale moderation can be useful. ⚫ There's a single-metric rating that combines many of these fairly well -- yes, Evan Miller's lower-bound Wilcox score. ⚫ Rating for "popularity" vs. "truth" is very, very different. ⚫ Reporting independent statistics for popularity (n), rating (mean), and variance or controversiality (standard deviation) is more informative than a single statistic. ⚫ Indirect quality measures also matter. I should add: a LOT. ⚫ There almost certainly isn't a single "best" ranking. Fuzzing scores with randomness can help. ⚫ Not all rating actions are equally valuable. Not everyone's ratings carry the same weight. ⚫ There are things which don't work well. ⚫ Showing scores and score components can be counterproductive and leads to various perverse incentives. I'm also increasing leaning toward a multi-part system, one which rates: 1. Overall favorability. 2. Any flaggable aspects. Ultimately, "ToS" is probably the best bucket, comprising spam, harassment, illegal activity, NSFW/NSFL content (or improperly labeled same), etc. 3. A truth or validity rating. Likeley rolled up in #2. But worth mentioning separately. 4. Long-term author reputation. There's also the general problem associated with Gresham's Law, which I'm increasingly convinced is a general and quite serious challenge to market-based and popularity-based systems. Assessment of complex products, including especialy information products, is difficult, which is to say, expensive. I'm increasingly in favour of presenting newer / unrated content to subsets of the total audience, and increasing its reach as positive approval rolls in. This seems like a behavior HN's "New" page could benefit from. Decrease the exposure for any one rater, but spread ratings over more submissions, for longer. And there are other problems. Limiting individuals to a single vote (or negating the negative effects of vote gaming) is key. Watching the watchmen. Regression toward mean intelligence / content. The "evaporative cooling" effect (http://blog.bumblebeelabs.com/social-software-sundays-2-the-evaporative-cooling-effect/ http://blog.bumblebeelabs.com/social-software-sundays-2-the-...).

sova 11y ago

Awesome if you are using only "up" and "down" ...

Houshalter 11y ago
It should work for star ratings, and generalize to non discrete rating systems too.