5 ms·

Maybe it's the "star rating" system that's broken. When reviewing rated product/service offerings, I go straight to the 1-star reviews. Skimming those, I diffe

Maybe it's the "star rating" system that's broken. When reviewing rated product/service offerings, I go straight to the 1-star reviews. Skimming those, I differentiate the "valid complaints" from flukes (a small failure rate is understandable), unrealistic expectations (it's a vintage hotel, not a new Hilton), tangential problems (Amazon didn't smash the box, the shipper did), hysteria (political opposition isn't a reason), humor (Family Circus isn't peer-reviewed), etc. We need a way to express these, especially a "for what it is, it's good (whether it met my needs/interests or not)".

peroo 12y ago
As bad as the star system is, I'm not sure there's anything better. I think Amazon has the right idea with the "Was this review helpful to you?" buttons, though that may not work quite as well for Yelp if you consider the possibility of review-burying by owners.
- minimaxir 12y ago
  Unfortunately, the "helpful" rating on Amazon's reviews are heavily biased: http://minimaxir.com/img/amazon/amzn-basic-helpful.png http://minimaxir.com/img/amazon/amzn-basic-helpful.png
- nitrogen 12y ago
  We could have stars for multiple axes, plus a "choose n-of-m" keyword selection, but I wonder if that would be too complicated for users. Example: Service: 1-5 Cleanliness: 1-5 Concierge: 1-5 Modernity: 1-5 .... Choose three words to describe the hotel stay: _ modern _ rustic _ rude _ quaint _ clean ...
  - ghaff 12y ago
    There's a lot of evidence that minimizing dimensions--even to the point of getting rid of obviously relevant ones--generally works best. Though Zagat does have food, decor, service, and cost so it can work especially when you're focused on a single thing. It gets harder when you're rating different things that span different dimensions. Amazon, for example, obviously just has a star system and DVD reviews are this sorta annoying mix of reviews of the movie and reviews of the specific DVD product.
- aestra 12y ago
  >I think Amazon has the right idea with the "Was this review helpful to you?" buttons Doubtful. "Was this review helpful to you?" is actually the "I disagree with this review" button in practice. Amazon reviews are so crappy. There is tons of reviews that say JUST "I haven't used this product/read this book yet." plus any number of stars.
  - DanBC 12y ago
    > Amazon reviews are so crappy. There is tons of reviews that say JUST "I haven't used this product/read this book yet." plus any number of stars. While it's extra work I love those reviews. "I haven't read this book but $THING IS WRONG, thus one " mean I can ignore some of the low ratings. While reviews and ratings are weird and probably need some tweaking at some point[1] they are not nearly as bad as the weirdly broken search. I've got to the point where I would pay money to have better Amazon (also ebay) search*.[2] [1]along the lines suggested in this thread - I might dislike a movie but love the genre; I might love a movie but generally dislike that genre; etc. So some method of saying why you like something so much would perhaps be handy. [2] I go to Amazon.co.uk and I search for [microwave oven]. I chose a department - kitchen & home. I then sort by price, low to high. I am flooded with totally irrelevant items. (Egg cup; aluminum foil; salt&pepper shakers; children's cutlery sets; etc). You're not supposed to use search to find microwave ovens, you're supposed to drill down the tree of department, items, specific items.
    - aestra 12y ago
      Found a review of such a nature http://www.amazon.com/review/R35GSCGMDIAI9G/ref=cm_cr_pr_perm?ie=UTF8&ASIN=0743264363 http://www.amazon.com/review/R35GSCGMDIAI9G/ref=cm_cr_pr_per... >Have not read >At this time I am caught up with so many pressing activites that is not one of my priorities. Will get to it probably this summer One star review. I agree completely about the searches. I'm starting to avoid Amazon cause I can't find what I want on there. >"I haven't read this book but $THING IS WRONG, thus one " mean I can ignore some of the low ratings. There are equal number of higher ratings and mid ratings of this nature.
dec0dedab0de 12y ago
I think you're talking about the old slashdot moderation system. +5 insightful
ben1040 12y ago
>Maybe it's the "star rating" system that's broken. While less so for product reviews, I feel like restaurant/hotel reviews are more like movie reviews. Just like Battlefield Earth, there are some bad restaurants that universally will get panned. And there are some just wonderful establishments everyone will love, just like Shawshank Redemption. But for everything in between, it's a matter of taste, which is why Netflix tries to infer your tastes and predict a star rating for _you_ rather than gives you an aggregate rating. Someone might give a dive bar a one-star review because it's smoky, they only accept cash, and they only sell Bud Light. However that same person might give a 5-star review to a craft beer bar that refuses to sell mass market lagers. On the flipside, someone might give the dive 5 stars because the drinks are cheap and it's one of the last places in town where they can go out and still light up. And they might give the fancy bar one star for being too pretentious. Neither opinion is wrong, people just like different things. Glomming this all into an average star rating just seems to give me useless information.
- fragsworth 12y ago
  > for everything in between, it's a matter of taste, which is why Netflix tries to infer your tastes and predict a star rating for _you_ rather than gives you an aggregate rating. This works only to an extent. There are many movie qualities that can be very important to you that you can't represent with a 5 star rating system. Because of this, the main way Netflix clusters your preferences is by the things they know, like genre, actors, producers/directors, etc., and they assume your high rating is (in part) for these things. This problem is much more prevalent in music suggestions (Pandora, Spotify, etc.) where your preferences are determined almost entirely by genre and artist. The problem is even more prevalent in ratings for restaurants. If you give 5 stars to an excellent Mexican restaurant, Yelp and Google Maps will suggest to you other Mexican restaurants. So as a savvy user, you are in a strange position of having to realize how they are clustering things, and give 5 stars to things you don't really like, because you want to be given recommendations for high-rated things in those genres. And you can't give 1 star to a movie in a genre that you like, otherwise the system will think you hate that genre. So I would agree with the previous comment, that the "star rating" system is totally broken.
- DanBC 12y ago
madaxe_again 12y ago
It's not just the star ratings that are broken, it's the entire hospitality system. Having had the opportunity to take a good, long stare at the underbelly of how hotel rooms are arbitraged, marketed, booked, and how the ensuing reviews and PR is cultured, the whole damn thing is a cartel controlled by surprisingly few entities and individuals. Bookings all funnel through systems like TRX and concur, all of whom take a slice (the actual hotelier, if there is one, tends to get about as much as a recording artist in terms of percentages - and they often end up PAYING booking engines and arbiters to fill capacity), and reviews are similarly curated and controlled. There are numerous businesses whose entire function is controlling ratings for hotels and hotel groups - and they use every dirty trick in the box. We worked on a startup which did hotel bookings in such a fashion that it circumvented this ratings cartel - and promptly had several major groups refuse to work with them, until they played the game. It's dirty, monopolistic, and so full of anti-trust it's not funny - but lobbying.
- DanBC 12y ago
  That sounds like an excellent indie documentary. If you have the stomach for it you could Kickstart it. Or at least a series of blog posts about how the free market for hotels is broken.
dredmorbius 12y ago
Any ratings or classification system has some very fundamental challenges. There's no way to collapse a multidimensional metric to a single value without losing information. That's what a "star system", numeric review, or others, do. Quick, what are food grades, and what are the grading criteria? (See: http://articles.latimes.com/1987-11-13/business/fi-13882_1_meat http://articles.latimes.com/1987-11-13/business/fi-13882_1_m... and https://www.princeton.edu/~ota/disk3/1977/7707/770706.PDF https://www.princeton.edu/~ota/disk3/1977/7707/770706.PDF) Different people have different assessment criteria. That seems to be the problem the hotel in question claims to be facing. "What's best?" inherently presumes "For what?". A vintage hotel shouldn't be rated on the same criteria as a five-star downtown hotel or business hotel. Or for its healthcare, auto-repair, clothing, bookstore, or hardware characteristics. The challenge is when an entity tries to claim a special status or twist that's at strong odds to most general expectations. Thin data. Only a small number of people typically leave reviews, and you're likely to get feedback only for exceptionally positive, or exceptionally negative, experiences. Or those who are motivated for whatever reasons (competition trying to shut you down, paid positive reviewers, etc.) to tell a specific story. This produces a highly non-randomized sampling selection. Unqualified reviewers. This is a problem pretty much any collectively rated system has. People who are not capable of making a valid assessment. Among other services, reddit and HN are suffering from this. The significance of ratings. Meaningless as they are, small changes in ratings can be hugely significant for a business. A major problem of the market is that information on quality is thin, non-abundant, and often expensive to come by. Ratings systems, even imperfect ones, address the ratings cost issue, even if the data conveyed are low-quality (think of racial discrimination or similar behaviors). So despite the problems, people care. One way around these issues is to come up with a "based on your profile you'd likely prefer X". This is what Amazon tries to accomplish with its product recommendations, though the appropriateness of these is often pretty thin. The problem space is a hard one.

peroo 12y ago

As bad as the star system is, I'm not sure there's anything better. I think Amazon has the right idea with the "Was this review helpful to you?" buttons, though that may not work quite as well for Yelp if you consider the possibility of review-burying by owners.

minimaxir 12y ago
Unfortunately, the "helpful" rating on Amazon's reviews are heavily biased: http://minimaxir.com/img/amazon/amzn-basic-helpful.png http://minimaxir.com/img/amazon/amzn-basic-helpful.png
nitrogen 12y ago
We could have stars for multiple axes, plus a "choose n-of-m" keyword selection, but I wonder if that would be too complicated for users. Example: Service: 1-5 Cleanliness: 1-5 Concierge: 1-5 Modernity: 1-5 .... Choose three words to describe the hotel stay: _ modern _ rustic _ rude _ quaint _ clean ...
- ghaff 12y ago
  There's a lot of evidence that minimizing dimensions--even to the point of getting rid of obviously relevant ones--generally works best. Though Zagat does have food, decor, service, and cost so it can work especially when you're focused on a single thing. It gets harder when you're rating different things that span different dimensions. Amazon, for example, obviously just has a star system and DVD reviews are this sorta annoying mix of reviews of the movie and reviews of the specific DVD product.
aestra 12y ago
>I think Amazon has the right idea with the "Was this review helpful to you?" buttons Doubtful. "Was this review helpful to you?" is actually the "I disagree with this review" button in practice. Amazon reviews are so crappy. There is tons of reviews that say JUST "I haven't used this product/read this book yet." plus any number of stars.
- DanBC 12y ago
  > Amazon reviews are so crappy. There is tons of reviews that say JUST "I haven't used this product/read this book yet." plus any number of stars. While it's extra work I love those reviews. "I haven't read this book but $THING IS WRONG, thus one " mean I can ignore some of the low ratings. While reviews and ratings are weird and probably need some tweaking at some point[1] they are not nearly as bad as the weirdly broken search. I've got to the point where I would pay money to have better Amazon (also ebay) search*.[2] [1]along the lines suggested in this thread - I might dislike a movie but love the genre; I might love a movie but generally dislike that genre; etc. So some method of saying why you like something so much would perhaps be handy. [2] I go to Amazon.co.uk and I search for [microwave oven]. I chose a department - kitchen & home. I then sort by price, low to high. I am flooded with totally irrelevant items. (Egg cup; aluminum foil; salt&pepper shakers; children's cutlery sets; etc). You're not supposed to use search to find microwave ovens, you're supposed to drill down the tree of department, items, specific items.
  - aestra 12y ago
    Found a review of such a nature http://www.amazon.com/review/R35GSCGMDIAI9G/ref=cm_cr_pr_perm?ie=UTF8&ASIN=0743264363 http://www.amazon.com/review/R35GSCGMDIAI9G/ref=cm_cr_pr_per... >Have not read >At this time I am caught up with so many pressing activites that is not one of my priorities. Will get to it probably this summer One star review. I agree completely about the searches. I'm starting to avoid Amazon cause I can't find what I want on there. >"I haven't read this book but $THING IS WRONG, thus one " mean I can ignore some of the low ratings. There are equal number of higher ratings and mid ratings of this nature.

dec0dedab0de 12y ago

I think you're talking about the old slashdot moderation system. +5 insightful

ben1040 12y ago

>Maybe it's the "star rating" system that's broken. While less so for product reviews, I feel like restaurant/hotel reviews are more like movie reviews. Just like Battlefield Earth, there are some bad restaurants that universally will get panned. And there are some just wonderful establishments everyone will love, just like Shawshank Redemption. But for everything in between, it's a matter of taste, which is why Netflix tries to infer your tastes and predict a star rating for _you_ rather than gives you an aggregate rating. Someone might give a dive bar a one-star review because it's smoky, they only accept cash, and they only sell Bud Light. However that same person might give a 5-star review to a craft beer bar that refuses to sell mass market lagers. On the flipside, someone might give the dive 5 stars because the drinks are cheap and it's one of the last places in town where they can go out and still light up. And they might give the fancy bar one star for being too pretentious. Neither opinion is wrong, people just like different things. Glomming this all into an average star rating just seems to give me useless information.

fragsworth 12y ago
> for everything in between, it's a matter of taste, which is why Netflix tries to infer your tastes and predict a star rating for _you_ rather than gives you an aggregate rating. This works only to an extent. There are many movie qualities that can be very important to you that you can't represent with a 5 star rating system. Because of this, the main way Netflix clusters your preferences is by the things they know, like genre, actors, producers/directors, etc., and they assume your high rating is (in part) for these things. This problem is much more prevalent in music suggestions (Pandora, Spotify, etc.) where your preferences are determined almost entirely by genre and artist. The problem is even more prevalent in ratings for restaurants. If you give 5 stars to an excellent Mexican restaurant, Yelp and Google Maps will suggest to you other Mexican restaurants. So as a savvy user, you are in a strange position of having to realize how they are clustering things, and give 5 stars to things you don't really like, because you want to be given recommendations for high-rated things in those genres. And you can't give 1 star to a movie in a genre that you like, otherwise the system will think you hate that genre. So I would agree with the previous comment, that the "star rating" system is totally broken.
DanBC 12y ago

madaxe_again 12y ago

It's not just the star ratings that are broken, it's the entire hospitality system. Having had the opportunity to take a good, long stare at the underbelly of how hotel rooms are arbitraged, marketed, booked, and how the ensuing reviews and PR is cultured, the whole damn thing is a cartel controlled by surprisingly few entities and individuals. Bookings all funnel through systems like TRX and concur, all of whom take a slice (the actual hotelier, if there is one, tends to get about as much as a recording artist in terms of percentages - and they often end up PAYING booking engines and arbiters to fill capacity), and reviews are similarly curated and controlled. There are numerous businesses whose entire function is controlling ratings for hotels and hotel groups - and they use every dirty trick in the box. We worked on a startup which did hotel bookings in such a fashion that it circumvented this ratings cartel - and promptly had several major groups refuse to work with them, until they played the game. It's dirty, monopolistic, and so full of anti-trust it's not funny - but lobbying.

DanBC 12y ago
That sounds like an excellent indie documentary. If you have the stomach for it you could Kickstart it. Or at least a series of blog posts about how the free market for hotels is broken.

dredmorbius 12y ago

Any ratings or classification system has some very fundamental challenges. There's no way to collapse a multidimensional metric to a single value without losing information. That's what a "star system", numeric review, or others, do. Quick, what are food grades, and what are the grading criteria? (See: http://articles.latimes.com/1987-11-13/business/fi-13882_1_meat http://articles.latimes.com/1987-11-13/business/fi-13882_1_m... and https://www.princeton.edu/~ota/disk3/1977/7707/770706.PDF https://www.princeton.edu/~ota/disk3/1977/7707/770706.PDF) Different people have different assessment criteria. That seems to be the problem the hotel in question claims to be facing. "What's best?" inherently presumes "For what?". A vintage hotel shouldn't be rated on the same criteria as a five-star downtown hotel or business hotel. Or for its healthcare, auto-repair, clothing, bookstore, or hardware characteristics. The challenge is when an entity tries to claim a special status or twist that's at strong odds to most general expectations. Thin data. Only a small number of people typically leave reviews, and you're likely to get feedback only for exceptionally positive, or exceptionally negative, experiences. Or those who are motivated for whatever reasons (competition trying to shut you down, paid positive reviewers, etc.) to tell a specific story. This produces a highly non-randomized sampling selection. Unqualified reviewers. This is a problem pretty much any collectively rated system has. People who are not capable of making a valid assessment. Among other services, reddit and HN are suffering from this. The significance of ratings. Meaningless as they are, small changes in ratings can be hugely significant for a business. A major problem of the market is that information on quality is thin, non-abundant, and often expensive to come by. Ratings systems, even imperfect ones, address the ratings cost issue, even if the data conveyed are low-quality (think of racial discrimination or similar behaviors). So despite the problems, people care. One way around these issues is to come up with a "based on your profile you'd likely prefer X". This is what Amazon tries to accomplish with its product recommendations, though the appropriateness of these is often pretty thin. The problem space is a hard one.