5 ms·

Why the 5 Star Rating System Isn't Suited to Local Reviews

18 points by pmjoyce 17y ago

Yelp (or Qype/Citysearch/TrustedPlaces/Tipped etc.) are not deliberately misleading their users, but fundamentally they’re all broken, the foundations on which they’re built are unsound.

nimbix 17y ago
I like Amazon's 5 star rating system because it also tells you how many people gave the product which rating. There's a big difference if 20 people gave a product 3 stars or if 10 gave it 5 stars and 10 gave it 1 star. Amazon's system makes this difference visible, but most others don't.
- Genghis_Cohen 17y ago
  In fact, the best books often have all 1s or 5s. Boring books have all 3s. The variance may be more interesting than the average
- zain 17y ago
  Is everyone looking at a different Yelp than I am? I see the same ratings distribution as Amazon on Yelp, as well as a "trend" graph that tells you how the ratings changed over time. Here's a screenshot: http://img.skitch.com/20090504-n2jsr44y1py38pbd5wgcqs5uyn.png http://img.skitch.com/20090504-n2jsr44y1py38pbd5wgcqs5uyn.pn...
  - nimbix 17y ago
    The article made me think that they use a one-dimensional star count (could be because of that image), so my reply was formed around that idea. I have to admit I never checked what kind of rating Yelp actually uses.
  - pmjoyce 17y ago
    I still think this lacks any degree of real clarity, although I admit that it looks like it does. At best I might use the trend graphs as an starting point from which to interrogate the qualitative data (comments). One major problem is the need to click through to the detail page to see these charts, this doesn't lend itself to easy decision making on the fly. However, the real issue is that even when you do click through, it doesn't show probably another important metric by which to judge: volume of votes over time. If you were to add that into the equation and suddenly the user has 3 charts per venue to interrogate to establish the reliability of the initial score out of 5. I think that's a reasonably big ask for someone who just wants to know where's to get a decent Kung Pao chicken around here.
pmjoyce 17y ago
Right, but that's not the whole story. The 5 star system is probably a lot better suited to the sorts of things that Amazon sell (music, DVDs, electronics etc.). These ratings are not as time sensitive as a bar or restaurant ratings for example which could change entirely in the space of a very short period of time.
thwarted 17y ago
While a simple scale rating system is just that, simple, anything more complex quickly exceeds the amount of effort both rater and searchers are willing to put in to understanding it. "Rate this restaurant on a scale of 1-10" is much simpler to understand than a massive questionnaire (even ratings on something like 3 or 4 axes may be too many/too much effort). It also has the advantage of being ambiguous enough that ANY opinion can fit into it -- you would be hard pressed to find the right kind of questions that fit all the different things that can be rated (What if you liked a place but didn't experience any of the specific things being asked about? Would you rate a movie theater on service if you never used the concession stand?). A site that didn't allow a reviewer to put in prose to explain their rating would be useless; I believe all the sites the OP talked about do. The rating scale is just meant to be a summary, each one is a summary of the thing being reviewed, and in aggregate it's a summary of the community's feel for the thing being reviewed. When flipping channels looking at movies that are on, the star rating system is largely worthless because 1) it was most likely done by some paid movie buff who has inherently different motivations and likes/dislikes than I do and 2) there is no explanation as to the reasoning why a rating was given. It's all subjective measurements, and it can't be objective. The whole point of rating systems is opinion. It's just like asking a friend if a place is worth going. Often times, the use case is "find something good with a minimal amount of hassle". This comes down to things like "places people liked that are within a mile of where I am right now". I have five minutes to make a decision on this, the single data point of a rating scale helps me make the decision quickly and (perceptually) accurately (it may not actually be accurate, but I feel like I'm making a good decision). These sites are also designed for repeat users and users being contributors. You learn the way other people have rated things on the site over time and are more able to decide for yourself what a "1 star" means vs a "5 star" means given the context. I think a larger problem is getting people to want to expend the effort to provide a review of a place that they aren't extremely excited about (either positively or negatively). I know I don't bother to review places I had a so-so experience at, but if I had excellent service or really bad service, I make it a point to rate them. In some way, this skews the results, but is most likely isn't that big of a deal. I don't know what the OP has in store for his next posting, but if it's truly revolutionary, he should be starting his own competing review service.
- 17y ago
adamc 17y ago
Nothing about his analysis is limited to local reviews -- the mean is a limited measure regardless, and the limitation of reviewers reviewing at different times is a function of the number of reviews per unit time, not locality. (They're correlated, obviously, but far enough out in the "long tail", even international reviews are going to be thin.)
Caged 17y ago
I've always thought 5 star rating systems were fundamentally flawed because they're based on the assumption that the same experience experienced by two different people will get the same star rating. For example, what might be a four star experience for me could be a two star experience for you. Maybe it's just me, but I'd prefer a probability system such as up and down votes. If I see a restaurant has 100 "likes" and 10 "dislikes" it's probable that I'll also enjoy the place (given the fact I already know they're serving food and drinks I would normally enjoy).
- chris11 17y ago
  I think that would be relatively easy to estimate from Amazon's rating since they break down the votes. It's a safe assumption that 1 or 2 star votes would be equivalent to a down vote, and 4 or 5 star votes would garner an up vote.
derefr 17y ago
There's nothing wrong with asking users to rate things out of five. However, just seeing an out-of-five score is, indeed, pointless when you talk about products that can vary per-person and over time. I'd suggest that the score be rendered as a Sparkline of aggregate-score-over-time, with a surrounding colored field, the width of which is the deviation for that aggregate sample point. Thus, you could see whether a five-star restaurant used to be a three-star at a glance, and see how many people disagree with the current rating with a simple visual geometric comparison.
apinstein 17y ago
This article seems kinda troll-y to me. Why? Because even Yelp understands the "The man who drowned in the river whose average depth was 6 inches" problem, which is exactly why they offer both a histogram view of ratings, and an average-over-time graph. I frequently use both of those tools. Now don't get me wrong there are all kinds of problems with 1-5 rating schemes, and there are probably better schemes out there. But what Yelp does is as good as any I've seen towards combatting the problem, which is why I enjoy using it. As someone that took 4 semesters of business statistics in college, my personal favorite alternative scheme is paired-comparison, or even better yet, the related (and newer) MaxDiff algorithm [http://en.wikipedia.org/wiki/MaxDiff http://en.wikipedia.org/wiki/MaxDiff]. The only problem with these schemes are that they require much more user input which frankly is a lot to ask for on a site like Yelp.
- pmjoyce 17y ago
  I agree advanced statistical analysis is not suited to these types of sites as the requirement on the user is such that the response rate would be virtually nil. But the point is that while the 5-star system initially looks and feels like it's offering some real data with which to evaluate a bar/restaurant, the reality is that the data need advanced interpretation to come even close to being meaningful not to mind useful. There's probably not a perfect system out there but I'm pretty sure that other models would be more "honest" at least.
  - thwarted 17y ago
    The 5-star rating system is exactly the same as asking someone "would you use/watch/go/buy-from this product/place/movie/business again?". You ask a few people and you consolidate their answers into a aggregate perception and then make your own decision based on that. Then you go there (or not) and make your own judgement and add your input to the pool. This is exactly useful for the intended purpose and audience.
    - pmjoyce 17y ago
      I disagree strongly with your statement that the 5 star system is "exactly the same as asking someone "would you use/watch/go/buy-from this product/place/movie/business again?". " In fact, I think this is exactly where the complication arises from. I agree that that is probably a close approximation of the question one would like to have answered, but asking people to give a rating out of 5 stars certainly doesn't ask it. Not only that but it actually muddies the water somewhat when aggregate data is spat out as an answer.
edw519 17y ago
That's why you read the comments. The best rating systems I've seen looked something like this: # of stars (1-5): __ If not 5, the #1 thing that would get more stars: _________ or If not 5, the top 3 things that would get more stars: _______________ _______________ _______________
deleted 17y ago
[deleted]
jambalaya 17y ago
What about the Wilson score confidence interval mentioned here? http://www.evanmiller.org/how-not-to-sort-by-average-rating.html http://www.evanmiller.org/how-not-to-sort-by-average-rating....
- benburkert 17y ago
  I recently implemented a ranking system which combined the two: http://pastie.org/467334 http://pastie.org/467334 Not sure how sound my math is, but it seems to produce the desired results.
jonno99 17y ago
this type of problem is very common when surveying people. A 6 or 4 star system would be much better. People then have to decide it it was above or below "average" because there is no middle star.

18 points by pmjoyce 17y ago

Yelp (or Qype/Citysearch/TrustedPlaces/Tipped etc.) are not deliberately misleading their users, but fundamentally they’re all broken, the foundations on which they’re built are unsound.

nimbix 17y ago

I like Amazon's 5 star rating system because it also tells you how many people gave the product which rating. There's a big difference if 20 people gave a product 3 stars or if 10 gave it 5 stars and 10 gave it 1 star. Amazon's system makes this difference visible, but most others don't.

Genghis_Cohen 17y ago
In fact, the best books often have all 1s or 5s. Boring books have all 3s. The variance may be more interesting than the average
zain 17y ago
Is everyone looking at a different Yelp than I am? I see the same ratings distribution as Amazon on Yelp, as well as a "trend" graph that tells you how the ratings changed over time. Here's a screenshot: http://img.skitch.com/20090504-n2jsr44y1py38pbd5wgcqs5uyn.png http://img.skitch.com/20090504-n2jsr44y1py38pbd5wgcqs5uyn.pn...
- nimbix 17y ago
  The article made me think that they use a one-dimensional star count (could be because of that image), so my reply was formed around that idea. I have to admit I never checked what kind of rating Yelp actually uses.
- pmjoyce 17y ago
  I still think this lacks any degree of real clarity, although I admit that it looks like it does. At best I might use the trend graphs as an starting point from which to interrogate the qualitative data (comments). One major problem is the need to click through to the detail page to see these charts, this doesn't lend itself to easy decision making on the fly. However, the real issue is that even when you do click through, it doesn't show probably another important metric by which to judge: volume of votes over time. If you were to add that into the equation and suddenly the user has 3 charts per venue to interrogate to establish the reliability of the initial score out of 5. I think that's a reasonably big ask for someone who just wants to know where's to get a decent Kung Pao chicken around here.

pmjoyce 17y ago

Right, but that's not the whole story. The 5 star system is probably a lot better suited to the sorts of things that Amazon sell (music, DVDs, electronics etc.). These ratings are not as time sensitive as a bar or restaurant ratings for example which could change entirely in the space of a very short period of time.

thwarted 17y ago

While a simple scale rating system is just that, simple, anything more complex quickly exceeds the amount of effort both rater and searchers are willing to put in to understanding it. "Rate this restaurant on a scale of 1-10" is much simpler to understand than a massive questionnaire (even ratings on something like 3 or 4 axes may be too many/too much effort). It also has the advantage of being ambiguous enough that ANY opinion can fit into it -- you would be hard pressed to find the right kind of questions that fit all the different things that can be rated (What if you liked a place but didn't experience any of the specific things being asked about? Would you rate a movie theater on service if you never used the concession stand?). A site that didn't allow a reviewer to put in prose to explain their rating would be useless; I believe all the sites the OP talked about do. The rating scale is just meant to be a summary, each one is a summary of the thing being reviewed, and in aggregate it's a summary of the community's feel for the thing being reviewed. When flipping channels looking at movies that are on, the star rating system is largely worthless because 1) it was most likely done by some paid movie buff who has inherently different motivations and likes/dislikes than I do and 2) there is no explanation as to the reasoning why a rating was given. It's all subjective measurements, and it can't be objective. The whole point of rating systems is opinion. It's just like asking a friend if a place is worth going. Often times, the use case is "find something good with a minimal amount of hassle". This comes down to things like "places people liked that are within a mile of where I am right now". I have five minutes to make a decision on this, the single data point of a rating scale helps me make the decision quickly and (perceptually) accurately (it may not actually be accurate, but I feel like I'm making a good decision). These sites are also designed for repeat users and users being contributors. You learn the way other people have rated things on the site over time and are more able to decide for yourself what a "1 star" means vs a "5 star" means given the context. I think a larger problem is getting people to want to expend the effort to provide a review of a place that they aren't extremely excited about (either positively or negatively). I know I don't bother to review places I had a so-so experience at, but if I had excellent service or really bad service, I make it a point to rate them. In some way, this skews the results, but is most likely isn't that big of a deal. I don't know what the OP has in store for his next posting, but if it's truly revolutionary, he should be starting his own competing review service.

17y ago

adamc 17y ago

Nothing about his analysis is limited to local reviews -- the mean is a limited measure regardless, and the limitation of reviewers reviewing at different times is a function of the number of reviews per unit time, not locality. (They're correlated, obviously, but far enough out in the "long tail", even international reviews are going to be thin.)

Caged 17y ago

I've always thought 5 star rating systems were fundamentally flawed because they're based on the assumption that the same experience experienced by two different people will get the same star rating. For example, what might be a four star experience for me could be a two star experience for you. Maybe it's just me, but I'd prefer a probability system such as up and down votes. If I see a restaurant has 100 "likes" and 10 "dislikes" it's probable that I'll also enjoy the place (given the fact I already know they're serving food and drinks I would normally enjoy).

chris11 17y ago
I think that would be relatively easy to estimate from Amazon's rating since they break down the votes. It's a safe assumption that 1 or 2 star votes would be equivalent to a down vote, and 4 or 5 star votes would garner an up vote.

derefr 17y ago

There's nothing wrong with asking users to rate things out of five. However, just seeing an out-of-five score is, indeed, pointless when you talk about products that can vary per-person and over time. I'd suggest that the score be rendered as a Sparkline of aggregate-score-over-time, with a surrounding colored field, the width of which is the deviation for that aggregate sample point. Thus, you could see whether a five-star restaurant used to be a three-star at a glance, and see how many people disagree with the current rating with a simple visual geometric comparison.

apinstein 17y ago

This article seems kinda troll-y to me. Why? Because even Yelp understands the "The man who drowned in the river whose average depth was 6 inches" problem, which is exactly why they offer both a histogram view of ratings, and an average-over-time graph. I frequently use both of those tools. Now don't get me wrong there are all kinds of problems with 1-5 rating schemes, and there are probably better schemes out there. But what Yelp does is as good as any I've seen towards combatting the problem, which is why I enjoy using it. As someone that took 4 semesters of business statistics in college, my personal favorite alternative scheme is paired-comparison, or even better yet, the related (and newer) MaxDiff algorithm [http://en.wikipedia.org/wiki/MaxDiff http://en.wikipedia.org/wiki/MaxDiff]. The only problem with these schemes are that they require much more user input which frankly is a lot to ask for on a site like Yelp.

pmjoyce 17y ago
I agree advanced statistical analysis is not suited to these types of sites as the requirement on the user is such that the response rate would be virtually nil. But the point is that while the 5-star system initially looks and feels like it's offering some real data with which to evaluate a bar/restaurant, the reality is that the data need advanced interpretation to come even close to being meaningful not to mind useful. There's probably not a perfect system out there but I'm pretty sure that other models would be more "honest" at least.
- thwarted 17y ago
  The 5-star rating system is exactly the same as asking someone "would you use/watch/go/buy-from this product/place/movie/business again?". You ask a few people and you consolidate their answers into a aggregate perception and then make your own decision based on that. Then you go there (or not) and make your own judgement and add your input to the pool. This is exactly useful for the intended purpose and audience.
  - pmjoyce 17y ago
    I disagree strongly with your statement that the 5 star system is "exactly the same as asking someone "would you use/watch/go/buy-from this product/place/movie/business again?". " In fact, I think this is exactly where the complication arises from. I agree that that is probably a close approximation of the question one would like to have answered, but asking people to give a rating out of 5 stars certainly doesn't ask it. Not only that but it actually muddies the water somewhat when aggregate data is spat out as an answer.

edw519 17y ago

That's why you read the comments. The best rating systems I've seen looked something like this: # of stars (1-5): __ If not 5, the #1 thing that would get more stars: _________ or If not 5, the top 3 things that would get more stars: _______________ _______________ _______________

deleted 17y ago

[deleted]

jambalaya 17y ago

What about the Wilson score confidence interval mentioned here? http://www.evanmiller.org/how-not-to-sort-by-average-rating.html http://www.evanmiller.org/how-not-to-sort-by-average-rating....

benburkert 17y ago
I recently implemented a ranking system which combined the two: http://pastie.org/467334 http://pastie.org/467334 Not sure how sound my math is, but it seems to produce the desired results.

jonno99 17y ago

this type of problem is very common when surveying people. A 6 or 4 star system would be much better. People then have to decide it it was above or below "average" because there is no middle star.