10 ms·
How not to sort by average rating (2009)
- mbauman 5y agoI'd love to see an update here that: * Included a graph of the resulting ordering of the two dimensional plane and some examples * Included consideration of 5- or 10-star scales.
- ScoutOrgo 5y agoThe formula still works for scales of 5 or 10, you just have to divide by the max rating first and then multiply by it again at the end. For example a 3/5 stars turns into 0.6 positive and 0.4 negative observation. Following the formula from there will give a lower bound estimation between 0 and 1, so then you just multiple by 5 again to get it between 0 and 5.
- abetusk 5y agoThey have an article about K-star rating systems [0] which uses Bayesian approximation [1] [2] (something I know little to nothing about, I'm just regurgitating the article). There's a whole section on their website that has different statistics for programmers, including rating systems [3]. [0] https://www.evanmiller.org/ranking-items-with-star-ratings.html https://www.evanmiller.org/ranking-items-with-star-ratings.h... [1] https://en.wikipedia.org/wiki/Approximate_Bayesian_computation https://en.wikipedia.org/wiki/Approximate_Bayesian_computati... [2] https://www.evanmiller.org/bayesian-average-ratings.html https://www.evanmiller.org/bayesian-average-ratings.html [3] https://www.evanmiller.org/ https://www.evanmiller.org/ ("Mathematics of user ratings" section)
- WalterGR 5y ago(2009)
- karaterobot 5y agoIs there a better solution now?
- WalterGR 5y agoNo idea. It's customary to include the year in HN submission titles if it was published before the current year. When I made my comment, the title didn't include the year.
- driscoll42 5y agoOne alternative is SteamDB's solution: https://steamdb.info/blog/steamdb-rating/ https://steamdb.info/blog/steamdb-rating/
- anthony_r 5y agoThis is cool. But what I usually do is replace x/y with x/(y+5), and hope for the best :). The 5 can be replaced by 3 or 50, depending on what I'm dealing with. (in less important areas than sorting things by ratings to directly rank things for users; mentally bookmarked this idea for the next time I need something better, as this clearly looks better)
- mattb314 5y agoHeads up this weights all your scores towards 0. If you want to avoid this, an equally simple approach is to use (x+3)/(y+5) to weight towards 3/5, or any (x+a)/(y+b) to weight towards a/b. It turns out that this seemingly simple method has some (sorta) basis in mathematical rigor: you can model x and y as successes and total attempts from a Bernoulli random variable, a and b as the parameters in a beta prior distribution, and the final score to be the mean of the updated posterior distribution: https://en.wikipedia.org/wiki/Beta_distribution#Bayesian_inference https://en.wikipedia.org/wiki/Beta_distribution#Bayesian_inf... (I saw first this covered in Murphy's Machine Learning: A Probabilistic Perspective, which I'd recommend if you're interested in this stuff)
- rkuykendall-com 5y agoThis article inspired me so much that I based my shitty undergrad senior thesis on it. My idea was to predict the trend of the ratings by using I think a trailing weighted average, weighted to the most recent window. It managed to generate more "predictive" ratings of the following 6 months based on the Amazon dataset I used, but I doubt it would have held up to much scrutiny. I learned a ton though! Edit: Link to paper, which looks like it actually attempts to use a linear prediction algorithm. https://github.com/rkuykendall/rkuykendall.com/blob/e65147f6ddc1185e04f60082a6089c16581c0123/assets/ranking-linear-prediction/Kuykendall-Ranking_Linear_Prediction.pdf https://github.com/rkuykendall/rkuykendall.com/blob/e65147f6...
- hwbehrens 5y agoWhile I agree with the author in principle, I think there is an implicit criteria they ignore, which is the intuitive correctness from the perspective of the user. Imagine a user chooses "Sort by rating", and they subsequently observe an item with an average 4.5 ranking above a score of 5.0 because it has a higher Wilson score. Some portion of users will think "Ah, yes, this makes sense because the 4.5 rating is based on many more reviews, therefore its Wilson score is higher." and the vast, vast majority of users will think "What the heck? This site is rigging the system! How come this one is ranked higher than that one?" and erode confidence in the rankings. In fact, these kinds of black-box rankings* frequently land sites like Yelp into trouble, because it is natural to assume that the company has a finger on the scale so to speak when it is in their financial interests to do so. In particular, entries with a higher Wilson score are likely to be more expensive because their ostensibly-superior quality commands (or depends upon) their higher cost, exacerbating this effect due to perceived-higher margins. So the next logical step is to present the Wilson score directly, but this merely shifts the confusion elsewhere -- the user may find an item they're interested in buying, find it has one 5-star review, and yet its Wilson score is << 5, producing at least the same perception and possibly a worse one. Instead, providing the statistically-sound score but de-emphasizing or hiding it, such as by making it accessible in the DOM but not visible, allows for the creation of alternative sorting mechanisms via e.g. browser extensions for the statistically-minded, without sacrificing the intuition of the top-line score. * I assume that most companies would choose not to explain the statistical foundations of their ranking algorithm.
- 1024core 5y agoThis is a UX problem, which can be solved by not showing the exact rating, but showing a "rating score" which is the Wilson score.
- alecbz 5y agoOP addressed that: > So the next logical step is to present the Wilson score directly, but this merely shifts the confusion elsewhere -- the user may find an item they're interested in buying, find it has one 5-star review, and yet its Wilson score is << 5, producing at least the same perception and possibly a worse one. Though I'm not convinced how big of a deal this is. Even if you're worried about this, a further optimization may be to simply not display the score until there's enough reviews that it's unlikely anyone will manually compute the average rating.
- bradbeattie 5y agoThere are a number of approaches to this with increasing complexity: - Sum of votes divided by total votes - More advanced statistical algorithms that take into account confidence (as this article suggests) - Recommendation engines that provides a rating based on your taste profile But I'm pretty sure you could take this further depending on what data you're looking to feed in and what the end-users' expectations of the system are.
- zzzeek 5y agoif you dont have PostgreSQL it might be hard to create an index on that function. you can use a trigger that updates a fixed field on the row each time positive/negative changes, or otherwise run the calc and include it in your UPDATE statement when those numbers change.
- WithinReason 5y agoIIRC, a simple approximation of that horrendous formula is : (positive)/(positive+negative+1) It rewards items with more ratings. Basically, you initialize the number of negative ratings to 1 instead of 0.
- raldi 5y agoThis is a decent approximation. It handles all the common hazard cases: +10/-0 should rank higher than +1/-0 +10/-5 should rank higher than +10/-7 +100/-3 should rank higher than +3/-0 +10/-1 should rank higher than +900/-200
- akamoonknight 5y agoVery interesting, seems your remembering looks correct to me. x / (x+y+1) :: https://www.wolframalpha.com/input/?i=plot+x+%2F%28x+%2B+y+%2B+1%29%2C+x%3D0..10%2C+y%3D0..10 https://www.wolframalpha.com/input/?i=plot+x+%2F%28x+%2B+y+%... horrendous formula :: https://www.wolframalpha.com/input/?i=plot+%28%28x%2F%28x%2By%29%29+%2B+%28%281.96%5E2%2F2%29%2F%282*%28x%2By%29%29%29+-+%281.96%2F2%29*sqrt%28%28%28%28x%2F%28x%2By%29%29*%281-%28x%2F%28x%2By%29%29%29%29+%2B+%281.96%5E2%2F2%29%2F%284*%28x%2By%29%29%29%2F%28x%2By%29%29%29%2F%281+%2B+%281.96%5E2%2F2%29%2F%28x%2By%29%29%2C+x%3D0..10%2C+y%3D0..10 https://www.wolframalpha.com/input/?i=plot+%28%28x%2F%28x%2B... Much less prone to typos.
- anthony_r 5y agoAnd the distance between the two, specifically formula - x/(x+y+1): https://www.wolframalpha.com/input/?i=plot++-x+%2F%28x+%2B+y+%2B+1%29+%2B+%28%28x%2F%28x%2By%29%29+%2B+%28%281.96%5E2%2F2%29%2F%282*%28x%2By%29%29%29+-+%281.96%2F2%29*sqrt%28%28%28%28x%2F%28x%2By%29%29*%281-%28x%2F%28x%2By%29%29%29%29+%2B+%281.96%5E2%2F2%29%2F%284*%28x%2By%29%29%29%2F%28x%2By%29%29%29%2F%281+%2B+%281.96%5E2%2F2%29%2F%28x%2By%29%29%2C+x%3D0..10%2C+y%3D0..10 https://www.wolframalpha.com/input/?i=plot++-x+%2F%28x+%2B+y...
- gowld 5y agoThe main flaw in this formula is that when positive=0 the negative votes have no weight.
- truculent 5y agoA simpler solution: Weighted score = (positive + alpha) / (total + beta) In which alpha and beta are the mean number of positive and total votes, respectively. You may wish to estimate optimal values of alpha and beta subject to some definition of optimal, but I find the mean tends to work well enough for most situations.
- taormina 5y agoThis is a blast from the past. It's also surprisingly simple to implement his "correct" sort. Seriously, this link should make the rounds every year or so here.
- ChrisArchitect 5y agoAnything new here? Some previous discussions: 4 years ago https://news.ycombinator.com/item?id=15131611 https://news.ycombinator.com/item?id=15131611 6 years ago https://news.ycombinator.com/item?id=9855784 https://news.ycombinator.com/item?id=9855784 10 years ago https://news.ycombinator.com/item?id=3792627 https://news.ycombinator.com/item?id=3792627 13 years ago https://news.ycombinator.com/item?id=478632 https://news.ycombinator.com/item?id=478632 Reminder: you can enjoy the article without upvoting it
- oehpr 5y agoMaybe what we need here is an extension where you can filter out articles? It adds a click event to each link for the article, and then after a day has passed, will start filtering that link out from HN results? I give it a gap of a day because maybe you'd want to return and leave a comment. I might try my hand at a greasemonkey script if you're interested. Though, personally, I have no great issue seeing high quality posts again occasionally.
- chias 5y agoThe new thing is another cohort of people getting to be today's lucky 10,000.
- svnpenn 5y ago4 years? I think that's fine for a repost.
- dang 5y agoThanks! Macroexpanded: How Not to Sort by Average Rating (2009) - https://news.ycombinator.com/item?id=15131611 https://news.ycombinator.com/item?id=15131611 - Aug 2017 (156 comments) How Not to Sort by Average Rating (2009) - https://news.ycombinator.com/item?id=9855784 https://news.ycombinator.com/item?id=9855784 - July 2015 (59 comments) How Not To Sort By Average Rating - https://news.ycombinator.com/item?id=3792627 https://news.ycombinator.com/item?id=3792627 - April 2012 (153 comments) How Not To Sort By Average Rating - https://news.ycombinator.com/item?id=1218951 https://news.ycombinator.com/item?id=1218951 - March 2010 (31 comments) How Not To Sort By Average Rating - https://news.ycombinator.com/item?id=478632 https://news.ycombinator.com/item?id=478632 - Feb 2009 (56 comments)
- chias 5y agoI've been using this at work for the last year or so to great success. For example, we have an internal phishing simulation/assessment program, and want to track metrics like improvement and general uncertainty. Since implementing this about a year ago, we've been able to make great improvements such as: * for a given person, identify the wilson lower bound that they would not get phished if they were targeted * for the employee population as a whole, determine the 95% uncertainty on whether a sample employee would get phished if targeted It lets us make much more intelligent inferences about things, much more accurate risk assessments, and also lets us improve the program pretty significantly (e.g. your probability of being targeted being weighted by a combination of your wilson lower bound and your wilson uncertainty). There are SO MANY opportunities to improve things by using this method. Obviously it isn't applicable everywhere, but I'd suggest you look at any metrics you have that use an average and just take a moment to ask yourself if a Wilson bound would be more appropriate, or might enable you to make marked improvements.
- user5994461 5y agoSounds like people who don't read their emails would get the best score because they don't get phished.
- chias 5y agoPretty much, yep :) They're also less likely to get phished in general. Though this property may be suboptimal for other reasons.
- deleted 5y ago[deleted]
- jedberg 5y agoFun fact, this article inspired the sysadmin at XKCD to submit a patch to open source reddit to implement this sort on comments. It lives still today as the "best" sort. The blog post that explained it: https://web.archive.org/web/20091210120206/http://blog.reddit.com/2009/10/reddits-new-comment-sorting-system.html https://web.archive.org/web/20091210120206/http://blog.reddi...
- DangerousPie 5y agoOne of my sites has been using a ranking algorithm based on this article for over 10 years now. Nobody ever complained, so it must be pretty good.
- Waterluvian 5y agoYou can’t rate 0 stars so the entire range is shifted by 1 star. This makes any star rating system fatally flawed to begin with. Humans will see 3 stars and not perceive that as 50%.
- feoren 5y agoIs that really a fatal flaw? It's humans reading the ratings, and humans doing the ratings, so our human-factors might balance out a bit. I don't think people come in expecting the rating system to be perfectly linear because we have a mental model of how other humans rate things -- 1 star and 5 stars are very common, even when there's obviously ways the thing could be worse/better. So even though 3 stars sounds like more than 50%, most people would consider 3.0 stars a very poor rating.
- Waterluvian 5y agoI think you make a good point. But I don’t think it completely defeats the bias. Especially given that the star system that existed before the Web had 0 and half stars. It seems like it’s purely a result of widget design deficiency: how do you turn a null into a 0 with a star widget? (You could add an extra button but naturally designers will poo poo that)
- Macha 5y agoPercentage systems aren't immune to this, various pieces of games media were often accused of a 70-100% rating scale. Anything below 70 was perceived as a terrible game, and they didn't want to harm their relationship with publishers. So 70 became the "You might like it if there are some specifics that appeal to you" and 80 was a pretty average game.
- kazinator 5y agoThis still has the problem that some item with 12 votes will be ranked higher than some item with 12,000 votes. Oh, and also has the problem that some item with 12 votes will be ranked lower than some item with 12,000 votes. I think you simply need separate search categories for this. Say I want to look for underrated or undiscovered gems: "Give me the best ranked items that have 500 votes or less." It is misleading to throw a 12 vote item together into the same list as a 12,000,000 vote item, and present them as being ranked relative to each other.
- ximm 5y agoExactly. At a previous job we asked applicants about sort that had to satisfy some criteria like "more votes + same ration ranks higher" and "same amount of total votes, better ratio ranks higher" and (positive - negative) was the simplest solution that satisfied them all. That is not to say that (positive - negative) is perfect, but just dismissing it because it doesn't satisfy your randomly chosen criteria is just lazy.
- 1970-01-01 5y agoMy anecdotally accurate advice (AAA) is to always read 2-star reviews before purchase.
- actually_a_dog 5y agoWhy 2 star? I get the whole "forget about the 5 star reviews, because they're not going to tell you any of the downsides of the product," and "forget the 1 star reviews, because they're often unrelated complaints about shipping or delivery, and generally don't tell you much about the product." But, why not 3 star reviews? I generally pay the most attention to 3 star reviews, because they tend to be pretty balanced and actually tell you the plusses and minuses of the product. It seems like 2 star reviews would be somewhat like that, but leaning toward the negative/critical side. Is the negative/critical feedback what you're after?
- 1970-01-01 5y agoBecause therein I find the best explanations for product failures. 3-star reviews tend to contain less failures and more "this could have been much better if they ___" . Again, it's anecdotal. I have no data to back my words.
- gowld 5y ago"3 stars" means "meh, it's fine. I don't want to commit to rating but I'm not a sucker who gives 5 to everything" "2 stars" means "I really don't like it, but I can control my emotions and explain myself".
- deleted 5y ago[deleted]
- voldemort1968 5y agoSimilarly, the problem of calculating "Trending" https://smosa.com/adam/code-and-technology https://smosa.com/adam/code-and-technology
- deleted 5y ago[deleted]
- uoaei 5y agoSmall nitpick: > What we want to ask is: Given the ratings I have, there is a 95% chance that the “real” fraction of positive ratings is at least what? > Score = Lower bound of Wilson score confidence interval for a Bernoulli parameter There is actually a 97.5% chance that the "real" fraction of positive ratings is at least X, because the Wilson interval is a confidence interval which excludes a tail at each extremum. I personally would prefer to use a Bayesian derivation and just compute a one-tailed statistic, rather than applying a CLT approximation out-of-the-box with no tailoring as is typical in frequentist statistics. Regardless though, the principles are much the same and the sorted result probably would match very closely, so if this computation is significantly faster than stick with that.
- mindvirus 5y agoThis is really neat, I'd never heard of the Wilson score. Some other thoughts - anyone know what state of the art is in ratings? - especially for technology products, time on market seems like a big thing. How should the iPhone 8 be rated today? It was a great product in its day, but if someone is looking at it now there are a lot more trade offs. - I wonder how we should consider the users in this. It seems like certain users are more harsh or fair than others, and have different rating patterns. Should we try to identify "good" reviewers? - Similar to the above, I wonder if for example Uber drivers get harsher reviews if traffic or weather is bad. So should we normalize for local trends in ratings?
- Misdicorl 5y ago>Should we try to identify "good" reviewers? To this day I don't understand how the original Netflix algorithm for predicting user desires isn't co-opted for everything. I don't mind people rating taco bell 5 stars. I just don't want any of their reviews influencing the restaurant ratings I see. Same with basically every product category. We have this insanely intricate and rich data set that could truly help people find things that are great for them. But it's all locked away and only used for making a buck with nobody giving a shit about making people's lives better.
- wodenokoto 5y agoAs the other kinda mentions, average ratings is the first steps towards a full recommendation system. So that’s the state of the art, in some sense.
- noduerme 5y agoOblique to this but still relevant. There's a retail chain I write the software for, including the website and online store. They send out occasional customer surveys, which I also built, and which are constantly reviewed by corporate management. So, they wanted to raise their Google review rating, which was hovering between 4.6 to 4.8 stars, varying by location. The CEO called me with this idea that at the end of taking a survey, people who give us five stars in all ten survey categories should be shown a link to directly review the store on Google. Me, as an engineer - I was like, okay let's sit down and review metrics for each survey question, weight them and figure out how likely someone is to give us 5, rather than 4 stars in aggregate, and use that to decide whether to give them the link. He said no. The only people who should get the link are people who gave 5 stars on the survey in every single category. This seemed counterintuitive to me and I ran the numbers and found that only a small fraction of people did. But I accepted the dictat and did it his way. End result: About a 30% increase in 5-star google ratings across the board. His intuition was right, that the people most motivated to take the survey and complete it with straight-A's would be the people who would take the time to write great comments. You can and probably should infer from this that if you actually need to balance "average" ratings, the outliers on the top and bottom of the scale exert an inordinate amount of pull and tend to skew results toward the extremes.