4 ms·

Ratings should always be on a curve for the rater, and on a 10 point scale, with no official meaning attached to the number of stars. If you rate everything 10

Ratings should always be on a curve for the rater, and on a 10 point scale, with no official meaning attached to the number of stars. If you rate everything 10 stars, you just gave everything 5 stars. If you want your 10 stars to mean 10 stars, you need to rate a bell curve of other stuff too. (you might say "but if I only select the best places, and the ratings are always confirmed, that would be downvoting them." No, it wouldn't, that's exactly the type of situation that Gaussians know how to handle. Whatever is the average rating and whatever is the excellent rating will emerge from the collective.) if you wish to attack my plan, I'll only listen if you offer a defense of the status quo, or offer a better improvement plan.

Eisenstein 3y ago
> if you wish to attack my plan, I'll only listen if you offer a defense of the status quo, or offer a better improvement plan. Sorry but you don't get to set the terms of your own critique. If you refuse to listen then that impugns your character, but it doesn't make the critique invalid. You method is flawed because some people only leave reviews when they good service or only when they get bad service. Everyone has a different reason why they do things, and what motivates them to do things. This will only work if you force people to leave reviews, and in that case your chances of getting a legitimate rating is much smaller. You system would be better suited to de-rate the importance of people who leave a lot of reviews that are consistently skewed towards one end, but apply such a system to everyone is not going to make the results more fair.
neilv 3y ago
> If you want your 10 stars to mean 10 stars, you need to rate a bell curve of other stuff too. If you want your positive rating of something to have the most impact (for whatever reason), then you need to give negative ratings to other things (even if they deserve better)?
jmye 3y ago
My issue is that we’ve been trained to see things on a broken scale - 10 is good, 9 is good, 8 is probably ok, 7 is questionable and everything else is really bad. I get it’s chicken/egg, but I feel like you can’t adjust ratings without retraining people on what ratings mean. I guess that means I think we should probably do away with the numbers and think about a more qualitative approach? Not a strong belief.
- fsckboy 3y ago
  there needs to be a way to indicate something exceptionally good, and everything can't be that. if 5 is the center of the new scale where the current grade inflation puts everything, people will not need to be retrained. Then, let them vote. You don't need to understand grading on a curve to be graded on a curve, and the grade you get is the right one.
candiddevmike 3y ago
Counterpoint: steam reviews with their binary choice result in a better aggregate experience and remove the need for bell curves. Binary choice leaves less room for interpretation and fuzzy ness, you either like something or you don't.
- labster 3y ago
  Steam has other problems with user submitted genre tags. Make a DLC that people thought under-delivered and you be may surprised to learn that you shipped a hentai game. Though that is definitely on the user’s side.
- fwmcbumcrumble 3y ago
  Positive negative and meh is my preferred. Some things aren’t great, but were cheap enough to justify it. Maybe make it a really tiny box in the middle that you need to zoom in to click to encourage mostly positive or negative reviews.
  - subroutine 3y ago
    How about... - thumb down - thumb up - two thumbs up
- subroutine 3y ago
  I've heard the arguments for switching to binary choice (netflix also did a big writeup about it), but I'd still prefer the ability to give a bit more nuance in my rating. It feels weird giving a movie that is okay and a movie that is excellent the same rating. Rotten Tomatoes goes further to emphasize the "love it or hate it" dichotomy - they display a ripe red tomato for a movie with a rating of 60 and a splattered green tomato for a movie with a score of 59.
  - joshspankit 3y ago
    You might prefer binary choice on more granular criteria: Like the pacing of the overall plot, dislike the performance of the antagonist
mplewis 3y ago
Stuff like this doesn’t naturally fall on a bell curve. Most hotels are great. Most restaurants are pretty good. There is no such thing as an objective 5.
- fsckboy 3y ago
  in a normal distribution, you expect the mode to be at the mean to be at the median. If that's where most hotels are, and most restaurants, welcome to what 5 objectively means.
  - Eisenstein 3y ago
    You expect that people will rate most things to be 5s based on averages, since if you aggregate your best experiences as 10s and your worst as 0s, then most everything is a 5, right? But human brains don't work like that. We don't compare a dinner at a restaurant to literally the best dining experience we ever had or a trip to a theme park with the birth our child, so you can't realistically expect people to operate in that way when filling out a review form.
    - fsckboy 3y ago
      yes, and that's why figuring out the distribution of your opinions and normalizing them to convolve them with the rest of the raters will extract useful information from your ratings.
      - Eisenstein 3y ago
        According to your logic we should be failing half of all school classes because half the kids shouldn't be above average. Not many people are going to think that a system which weights everything to be distributed that way is useful as a metric for evaluating human experiences.
yowzadave 3y ago
Maybe this could help with rating inflation by legitimate reviewers, but I don’t think it would help (and could possibly hurt) with the problem of fraudulent reviews. It would just mean that fake reviews would counterbalance their fake positive reviews with fake negative reviews of other products, making all reviews, in both directions, totally unreliable.
deleted 3y ago
[deleted]
miki123211 3y ago
a five-star system, but with stars instead of labels. "Absolutely awful service" / "dissatisfied" / "basically what I expected" / "positively surprised" / "incredibly excellent" This way, most customers would click "basically what I expected", making 3 star the default. Adjusting all reviews based on the general mean and standard deviation after the fact makes sense, though.

Eisenstein 3y ago

> if you wish to attack my plan, I'll only listen if you offer a defense of the status quo, or offer a better improvement plan. Sorry but you don't get to set the terms of your own critique. If you refuse to listen then that impugns your character, but it doesn't make the critique invalid. You method is flawed because some people only leave reviews when they good service or only when they get bad service. Everyone has a different reason why they do things, and what motivates them to do things. This will only work if you force people to leave reviews, and in that case your chances of getting a legitimate rating is much smaller. You system would be better suited to de-rate the importance of people who leave a lot of reviews that are consistently skewed towards one end, but apply such a system to everyone is not going to make the results more fair.

neilv 3y ago

> If you want your 10 stars to mean 10 stars, you need to rate a bell curve of other stuff too. If you want your positive rating of something to have the most impact (for whatever reason), then you need to give negative ratings to other things (even if they deserve better)?

jmye 3y ago

My issue is that we’ve been trained to see things on a broken scale - 10 is good, 9 is good, 8 is probably ok, 7 is questionable and everything else is really bad. I get it’s chicken/egg, but I feel like you can’t adjust ratings without retraining people on what ratings mean. I guess that means I think we should probably do away with the numbers and think about a more qualitative approach? Not a strong belief.

fsckboy 3y ago
there needs to be a way to indicate something exceptionally good, and everything can't be that. if 5 is the center of the new scale where the current grade inflation puts everything, people will not need to be retrained. Then, let them vote. You don't need to understand grading on a curve to be graded on a curve, and the grade you get is the right one.

candiddevmike 3y ago

Counterpoint: steam reviews with their binary choice result in a better aggregate experience and remove the need for bell curves. Binary choice leaves less room for interpretation and fuzzy ness, you either like something or you don't.

labster 3y ago
Steam has other problems with user submitted genre tags. Make a DLC that people thought under-delivered and you be may surprised to learn that you shipped a hentai game. Though that is definitely on the user’s side.
fwmcbumcrumble 3y ago
Positive negative and meh is my preferred. Some things aren’t great, but were cheap enough to justify it. Maybe make it a really tiny box in the middle that you need to zoom in to click to encourage mostly positive or negative reviews.
- subroutine 3y ago
  How about... - thumb down - thumb up - two thumbs up
subroutine 3y ago
I've heard the arguments for switching to binary choice (netflix also did a big writeup about it), but I'd still prefer the ability to give a bit more nuance in my rating. It feels weird giving a movie that is okay and a movie that is excellent the same rating. Rotten Tomatoes goes further to emphasize the "love it or hate it" dichotomy - they display a ripe red tomato for a movie with a rating of 60 and a splattered green tomato for a movie with a score of 59.
- joshspankit 3y ago
  You might prefer binary choice on more granular criteria: Like the pacing of the overall plot, dislike the performance of the antagonist

mplewis 3y ago

Stuff like this doesn’t naturally fall on a bell curve. Most hotels are great. Most restaurants are pretty good. There is no such thing as an objective 5.

fsckboy 3y ago
in a normal distribution, you expect the mode to be at the mean to be at the median. If that's where most hotels are, and most restaurants, welcome to what 5 objectively means.
- Eisenstein 3y ago
  You expect that people will rate most things to be 5s based on averages, since if you aggregate your best experiences as 10s and your worst as 0s, then most everything is a 5, right? But human brains don't work like that. We don't compare a dinner at a restaurant to literally the best dining experience we ever had or a trip to a theme park with the birth our child, so you can't realistically expect people to operate in that way when filling out a review form.
  - fsckboy 3y ago
    yes, and that's why figuring out the distribution of your opinions and normalizing them to convolve them with the rest of the raters will extract useful information from your ratings.
    - Eisenstein 3y ago
      According to your logic we should be failing half of all school classes because half the kids shouldn't be above average. Not many people are going to think that a system which weights everything to be distributed that way is useful as a metric for evaluating human experiences.

yowzadave 3y ago

Maybe this could help with rating inflation by legitimate reviewers, but I don’t think it would help (and could possibly hurt) with the problem of fraudulent reviews. It would just mean that fake reviews would counterbalance their fake positive reviews with fake negative reviews of other products, making all reviews, in both directions, totally unreliable.

deleted 3y ago

[deleted]

miki123211 3y ago

a five-star system, but with stars instead of labels. "Absolutely awful service" / "dissatisfied" / "basically what I expected" / "positively surprised" / "incredibly excellent" This way, most customers would click "basically what I expected", making 3 star the default. Adjusting all reviews based on the general mean and standard deviation after the fact makes sense, though.