7 ms·

The whole system of review curation that infests online markets is a problem waiting to be solved. Right now we basically have to parse reviews on the 4.4-5.0 s

The whole system of review curation that infests online markets is a problem waiting to be solved. Right now we basically have to parse reviews on the 4.4-5.0 scale because anything outside that is meaningless. It's amazing to go read a novel rated 3 stars (because it operates outside this loop) and find it transcendently good.

ghaff 10y ago
It's a difficult problem. Survey response scales have probably been a matter of research and discussion before the web even existed. There are particular instances (like in this case) where there are a lot of incentives for just about everyone to give 5 stars, 10 stars, or a positive and call it a day. But, in general, there are issues with self-selection, bias toward the high end of the scale, ratings meaning different things for different types and classes of product, etc.
antisthenes 10y ago
Reviews are a problem, but not for novels. Perception of books and concepts covered within are entirely subjective. However, when it comes to service or product quality, you can be almost entirely objective in its value judgment.
- dragonwriter 10y ago
  > However, when it comes to service or product quality, you can be almost entirely objective in its value judgment. No, all assessments of value are subjective (and for most services involve subjective weighting of multiple different subjective quality factors.) And the mapping from an internal value judgement to a star rating is something which is a separate subjective evaluation, and one on which research has shown substantial cultural variation along ethnic and other lines.
  - antisthenes 10y ago
    While that may be true for other people, I have no trouble giving a purchased object a rating for its objective value (but I do excessive research on buying options beforehand). Even if the judgment is relative to other products, it can still be objective. The rating may be inaccurate because you can only measure in full stars, but it'll still be objective and the inaccuracy is the fault of the rating system, not me. Objective (adjective): > (of a person or their judgment) not influenced by personal feelings or opinions in considering and representing facts.
- ghaff 10y ago
  >Reviews are a problem, but not for novels. Hmm. I'm willing to accept that the average reader is a more sophisticated and thoughtful consumer than for some other product categories. And there may be less overt tribalism and fanboyism in book reviews. But, in my experience, there's still a lot of people who will almost unthinkingly give 5 stars to a favorite author or genre. (And I know some of those people who are pretty uncritical about SF books or whatever.)
  - robryan 10y ago
    Goodreads reviews are still pretty inflated. Anything around or under 3.5 or so probably has some big flaws.
    - antisthenes 10y ago
      I have found no correlation between the Goodreads score and my own perception of the book. Again, novels and products are entirely different in that you're hopefully shopping for a product by its specs. You can't 'research' novels beforehand (unless you count as reading other novels by the same author as research) and they don't have any specs to go by.
nradov 10y ago
Instead of asking customers to give numeric star ratings, perhaps a better option would be a short list of Yes / No / Unsure questions. Were there clean sheets on the bed? Were there any cockroaches? Did the appliances work? Etc. Such questions are more objective and less prone to inflation.
- golergka 10y ago
  They also require more time, and most people don't bother to leave reviews anyway.
  - warrenm 10y ago
    But if you're going to review, a few extra seconds shouldn't matter It's how the "official" reviews from Marriott/Hilton/etc all work
  - Taek 10y ago
    1. Require each person to answer one specific question per listing. Yes/no/almost 2. Ask each person a different objective question, intentionally filling in the gaps. 3. After a few people, you have an objective, relatively thorough view of the house, but you've not burdened any of your users with more than one question per stay.
- ansgri 10y ago
  Booking.com does it, btw. And I strongly prefer it.
  - prawn 10y ago
    Agree - it really helps you identify which parts of the experience are problematic. If the location is 8.7 where everything else is 9.5+, you know you'll be walking a distance to get anywhere useful. If the location is 9.5 while cleanliness is 7.8, you can consider the trade-off to be in the thick of things.
  - mosselman 10y ago
    Surprised there is only one mention of Booking.com here. It is an indispensable tool for booking my vacations. The reviews are very useful and a lot more accurate than things like yelp. I have glanced at other websites, but nothing gives me the reassurance of the legitimacy of the reviews that booking.com does.
suresk 10y ago
Is anyone trying to solve this right now? It seems like a really interesting problem that would be really useful to solve. It isn't just online, and it isn't just a technology problem. I recently bought a car and the manufacturer sent me a survey about the process. The salesman made sure, numerous times, to let me know that I'd be getting said survey and that anything less than a 10 meant he failed and that there would be negative consequences for him. Since I was mostly happy with the process, I gave him all 10s, even though one or two areas I would have rated a little lower. He got his good review and it didn't really bother me, but the car company lost out on some potentially useful feedback. Some of the problems - like different people having a tendency to rate on different scales, ratings coming from different factors, etc - seem pretty amenable to a ML approach to make reviews a lot more useful, though. I wonder if anyone outside of a handful of companies (Amazon, Yelp, AirBnB, Google, etc) has enough data to make it useful, though?
- SerpentJoe 10y ago
  Many services have this issue, that perfection is the minimum requirement. I don't understand why, given this, we don't just go to a thumbs up / thumbs down rating system. We have one option that means "fine" and four (or nine) that mean "terrible, and a risk to your business". I know I personally gave plenty of three star reviews on Lyft before I learned better, meaning them as "no complaints". Who knows what happened to those drivers as a result. Why even invite the misunderstanding?
  - GuiA 10y ago
    This could be a cultural design thing. A lot of these apps (Airbnb, Lyft, etc) are from the US, where getting full marks in school is a frequent occurence if you study reasonably well. In France for instance, it's very rare for a student to go above 15/20 or so at the college level (I graduated in the top 10% of my class with a 12/20 average; the top ranked student, whom I was friends with, had a 14/20 average). But when I graded homework as a graduate student for an american university, the professor told me I was too harsh, and after adjusting my criteria to match what she wanted, I found myself giving many 100/100.
fiatjaf 10y ago
Has anyone ever tried 1. normalize reviews to make a 4 star from someone who always gives 5 stars worth less than a 3 star from someone who always gives 2 stars; and 2. make reviews secret?
- wnissen 10y ago
  I've wondered this myself. Someone with a dozen 5-star reviews is essentially providing no information and should be removed from the equation.
lordnacho 10y ago
How about making you rank the places you've stayed at? You could do it by subset if there's a lot.
- usrusr 10y ago
  Or normalize the ranges between min/avg/max of the ratings given by each individual user to min/middle/max of the scale before merging the number into the total. If you do it secretly like shadowbanning and introduce it softly by slowly blending between conventional score and normalized, it would drive a whole generation of SEO/social/share-economy experts delightfully mad. Think of all the free publicity! Jokes aside, it might help to regard low ratings as what they are: a way to flag bad behavior, dead serious stuff. We should stop pretending that it is a game, then maybe solutions will be more likely to appear.
tafda 10y ago
This doesn't seem to be as big of a problem in Japan. On the popular restaurant review site Tabelog, a 3.6 (out of 5) is a very good ranking; there's more 3-star Michelin restaurants in Tokyo than restaurants above 4.5. https://tabelog.com/en https://tabelog.com/en
nocoder 10y ago
Fundamental challenge with star rating system is your perception of what counts as 5 star & my perception of what counts as 5 can be very different. It basically comes down to what your personal standards of something 5 star are. I am generally a very adjusting person, so minor glitches might not lead me to move from 5 to 2 star but someone could be more particular than me in which case his 5 star will move to 1 star at slightest issue.

ghaff 10y ago

It's a difficult problem. Survey response scales have probably been a matter of research and discussion before the web even existed. There are particular instances (like in this case) where there are a lot of incentives for just about everyone to give 5 stars, 10 stars, or a positive and call it a day. But, in general, there are issues with self-selection, bias toward the high end of the scale, ratings meaning different things for different types and classes of product, etc.

antisthenes 10y ago

Reviews are a problem, but not for novels. Perception of books and concepts covered within are entirely subjective. However, when it comes to service or product quality, you can be almost entirely objective in its value judgment.

dragonwriter 10y ago
> However, when it comes to service or product quality, you can be almost entirely objective in its value judgment. No, all assessments of value are subjective (and for most services involve subjective weighting of multiple different subjective quality factors.) And the mapping from an internal value judgement to a star rating is something which is a separate subjective evaluation, and one on which research has shown substantial cultural variation along ethnic and other lines.
- antisthenes 10y ago
  While that may be true for other people, I have no trouble giving a purchased object a rating for its objective value (but I do excessive research on buying options beforehand). Even if the judgment is relative to other products, it can still be objective. The rating may be inaccurate because you can only measure in full stars, but it'll still be objective and the inaccuracy is the fault of the rating system, not me. Objective (adjective): > (of a person or their judgment) not influenced by personal feelings or opinions in considering and representing facts.
ghaff 10y ago
>Reviews are a problem, but not for novels. Hmm. I'm willing to accept that the average reader is a more sophisticated and thoughtful consumer than for some other product categories. And there may be less overt tribalism and fanboyism in book reviews. But, in my experience, there's still a lot of people who will almost unthinkingly give 5 stars to a favorite author or genre. (And I know some of those people who are pretty uncritical about SF books or whatever.)
- robryan 10y ago
  Goodreads reviews are still pretty inflated. Anything around or under 3.5 or so probably has some big flaws.
  - antisthenes 10y ago
    I have found no correlation between the Goodreads score and my own perception of the book. Again, novels and products are entirely different in that you're hopefully shopping for a product by its specs. You can't 'research' novels beforehand (unless you count as reading other novels by the same author as research) and they don't have any specs to go by.

nradov 10y ago

Instead of asking customers to give numeric star ratings, perhaps a better option would be a short list of Yes / No / Unsure questions. Were there clean sheets on the bed? Were there any cockroaches? Did the appliances work? Etc. Such questions are more objective and less prone to inflation.

golergka 10y ago
They also require more time, and most people don't bother to leave reviews anyway.
- warrenm 10y ago
  But if you're going to review, a few extra seconds shouldn't matter It's how the "official" reviews from Marriott/Hilton/etc all work
- Taek 10y ago
  1. Require each person to answer one specific question per listing. Yes/no/almost 2. Ask each person a different objective question, intentionally filling in the gaps. 3. After a few people, you have an objective, relatively thorough view of the house, but you've not burdened any of your users with more than one question per stay.
ansgri 10y ago
Booking.com does it, btw. And I strongly prefer it.
- prawn 10y ago
  Agree - it really helps you identify which parts of the experience are problematic. If the location is 8.7 where everything else is 9.5+, you know you'll be walking a distance to get anywhere useful. If the location is 9.5 while cleanliness is 7.8, you can consider the trade-off to be in the thick of things.
- mosselman 10y ago
  Surprised there is only one mention of Booking.com here. It is an indispensable tool for booking my vacations. The reviews are very useful and a lot more accurate than things like yelp. I have glanced at other websites, but nothing gives me the reassurance of the legitimacy of the reviews that booking.com does.

suresk 10y ago

Is anyone trying to solve this right now? It seems like a really interesting problem that would be really useful to solve. It isn't just online, and it isn't just a technology problem. I recently bought a car and the manufacturer sent me a survey about the process. The salesman made sure, numerous times, to let me know that I'd be getting said survey and that anything less than a 10 meant he failed and that there would be negative consequences for him. Since I was mostly happy with the process, I gave him all 10s, even though one or two areas I would have rated a little lower. He got his good review and it didn't really bother me, but the car company lost out on some potentially useful feedback. Some of the problems - like different people having a tendency to rate on different scales, ratings coming from different factors, etc - seem pretty amenable to a ML approach to make reviews a lot more useful, though. I wonder if anyone outside of a handful of companies (Amazon, Yelp, AirBnB, Google, etc) has enough data to make it useful, though?

SerpentJoe 10y ago
Many services have this issue, that perfection is the minimum requirement. I don't understand why, given this, we don't just go to a thumbs up / thumbs down rating system. We have one option that means "fine" and four (or nine) that mean "terrible, and a risk to your business". I know I personally gave plenty of three star reviews on Lyft before I learned better, meaning them as "no complaints". Who knows what happened to those drivers as a result. Why even invite the misunderstanding?
- GuiA 10y ago
  This could be a cultural design thing. A lot of these apps (Airbnb, Lyft, etc) are from the US, where getting full marks in school is a frequent occurence if you study reasonably well. In France for instance, it's very rare for a student to go above 15/20 or so at the college level (I graduated in the top 10% of my class with a 12/20 average; the top ranked student, whom I was friends with, had a 14/20 average). But when I graded homework as a graduate student for an american university, the professor told me I was too harsh, and after adjusting my criteria to match what she wanted, I found myself giving many 100/100.

fiatjaf 10y ago

Has anyone ever tried 1. normalize reviews to make a 4 star from someone who always gives 5 stars worth less than a 3 star from someone who always gives 2 stars; and 2. make reviews secret?

wnissen 10y ago
I've wondered this myself. Someone with a dozen 5-star reviews is essentially providing no information and should be removed from the equation.

lordnacho 10y ago

How about making you rank the places you've stayed at? You could do it by subset if there's a lot.

usrusr 10y ago
Or normalize the ranges between min/avg/max of the ratings given by each individual user to min/middle/max of the scale before merging the number into the total. If you do it secretly like shadowbanning and introduce it softly by slowly blending between conventional score and normalized, it would drive a whole generation of SEO/social/share-economy experts delightfully mad. Think of all the free publicity! Jokes aside, it might help to regard low ratings as what they are: a way to flag bad behavior, dead serious stuff. We should stop pretending that it is a game, then maybe solutions will be more likely to appear.

tafda 10y ago

This doesn't seem to be as big of a problem in Japan. On the popular restaurant review site Tabelog, a 3.6 (out of 5) is a very good ranking; there's more 3-star Michelin restaurants in Tokyo than restaurants above 4.5. https://tabelog.com/en https://tabelog.com/en

nocoder 10y ago

Fundamental challenge with star rating system is your perception of what counts as 5 star & my perception of what counts as 5 can be very different. It basically comes down to what your personal standards of something 5 star are. I am generally a very adjusting person, so minor glitches might not lead me to move from 5 to 2 star but someone could be more particular than me in which case his 5 star will move to 1 star at slightest issue.