5 ms·

This has always bothered me, both on Amazon and the iTunes app store. No lie, when I published my first app, one of the comments was "I'd rate this zero stars i

This has always bothered me, both on Amazon and the iTunes app store. No lie, when I published my first app, one of the comments was "I'd rate this zero stars if I could." and my first thought was actually "wait, why can't you?" Overall, I think YouTube was on the ball by switching to a "Thumbs up, Thumbs down" rating system. "Out of five star" rating systems are always sabotaged by the fact that most people think in absolutes: Either something is the bee's knees or it's fried shit on a biscuit. Given that most people vote this way, up/down vote systems are better at giving an honest impression of how people feel about something.

brownbat 13y ago
I wonder if a five star rating system could secretly count only the 2 and 4 star reviews without telling anyone, sort of filter out anyone passionate enough to try to game things...
- mistercow 13y ago
  That might work, but I have doubts. Interestingly, it's the opposite approach the the one taken by pre-employment personality tests, in which each question has no partial credit and exactly one right answer, which is either "strongly agree" or "strongly disagree".
  - brownbat 13y ago
    Oh right! I think Barbara Ehrenreich mentioned that in Nickel and Dimed. At Walmart, one of the hiring surveys included some question pitting personal considerations or ethics against loyalty to management, and the only correct answer was absolute loyalty to management. When she debated the point, she was not offered a job.
    - thaumasiotes 13y ago
      In college, I got a part-time job for a stretch of some months grading exercises at the local Kumon center. The job consisted primarily of comparing a filled-out exercise booklet to a printed list of answers and marking incorrect answers. (The other part, the good part, was talking to the kids and trying to help them with problems they didn't get.) My boss was never happy with the speed at which I worked. Eventually she told me I was too smart for the job and should aim higher. In a vaguely similar way, I don't think Wal-mart is looking for their lowest-level employees to make a lot of independent ethical decisions on the job. If you're going to be hiring unskilled labor, you might not want them to exercise a lot of discretion, on the theory that they're at least as likely as not to screw it up.
- hga 13y ago
  I think the human mind is best at that. Even if there are too many reviews for you to completely read, you can sample them, especially the ones others have rated for usefulness, and with some practice recognize the ones that are passionate with sufficient support. Specificity is one think I really look for. Those who say essentially nothing more than "It's wonderful" or "It sucks" are noise. Those that say, "It's great, but I'm subtracting one star because the control console is a pain because of X, W and Y" are golden ... especially when people add comments on workarounds.... It's kind of funny, in that this "impersonal" mail order outfit very often manages to provide a better shopping experience than brick and mortar stores for those of us willing to buy things sight unseen. Heck, it's not uncommon for me to check Amazon.com reviews before buying something from one of those.
- Amadou 13y ago
  IMDB (which is a 10-star system) has a 'secret algorithm' that fiddles with the raw data to come up with a computed score. It's been suggested that some of the factors involved include reduced weight for 1 and 10 star ratings (as well as the age of the user's account - votes from new accounts don't get as much weight, etc).
- jjoonathan 13y ago
  I'd guess that moderation (as measured by 2-4 as opposed to 1 or 5 stars) is a worse predictor of an opinion's relevance than even the most simplistic automated classification of textual or behavioral feedback.
hga 13y ago
While you both have a point, I'm not sure it makes a difference in practice. OK, granted, I've been an Amazon.com customer since 1997 or so, but to me a 3.5 star rating is like writing BEWARE! in flaming letters across the sky. I never buy something that poorly rated before thoroughly investigating what's behind it. E.g. Kindle version's content is messed up I discount since I don't play in that ecosystem except for freebies. While there are plenty of mostly bi-modal distributions (5 or 1 stars), for lots of things there are carefully chosen ratings in-between, often telling you why it lost a star or 2. Even ones edited some time after with additional commentary and often an adjusted score. So I'd say it gets the specific job done, which is quite different than YouTube's where the cost of a wrong choice is very very small. I find the reviews invaluable, to the point where there's a lot of stuff I won't buy unless it has enough quality reviews for me to make an informed judgement.
- fjarlq 13y ago
  http://xkcd.com/1098/ http://xkcd.com/1098/
mcpherrinm 13y ago
I find the 5 star system on Amazon useful. I don't always look at the average rating, but rather the distribution graph of ratings. For example, one common shape looks like this: 5 stars: ||||||||||| 4 stars: |||||||| 3 stars: |||| 2 stars: || 1 star: ||||||| There's way more 1 star than 2 or 3. That usually indicates the product shows up DOA for a lot of people, or is otherwise probabilistically totally unsuitable for some group of people. If the number of star ratings follows a more normal curve like the following, then I'm more confident in the quality of the product, even if both have an average 4 star rating: 5 stars: ||||||| 4 stars: |||||||| 3 stars: |||| 2 stars: || 1 star: |
- mashmac2 13y ago
  This would make a fascinating study of reliability - can you determine statistically if a product will be reliable based on rating distribution. You could study Amazon, App Store ratings, etc. Cool insight!
  - sanskritabelt 13y ago
    It's been done, and I've seen it as a worked example in introduction to Bayesian stats pages, maybe some of which were posted on HN.
  - arh68 13y ago
    I wouldn't take any rating metric seriously unless it accounted for how long they've owned it. Plus there's the difference between "I've owned 12 lawnmowers and this one's pretty okay, only broke down after 1.5yrs, 3 stars" and "this is my first lawnmower, used it once, it's amazing, never going back to cutting my lawn with scissors, 5 stars". Plus niche sites sometimes have better reviews for the same products (newegg). All that said, there is information to be harvested in some of these reviews.
    - ludzone 13y ago
      Good point about niche sites and the length of ownership. I've noticed that a number of starts on products like top rated power supplies goes down after a year an a half or so. While you might be getting a newly released power supply with 500 reviews averaging closer to 5 stars, after 2 years, you will notice that same product with 1600 reviews averaging at below 4 stars, because owners probably came back and updated their reviews. It would be interesting to see a trend line of change in sentiment about a products over time, using existing data.
MichaelGG 13y ago
Netflix does a great job by showing hint text of "Hated it" for 1 star. It certainly helps calibrate people, rather than having individuals come up with their own meaning of what 1-5 means.
ekianjo 13y ago
up/down systens are stupid since they are so binary. It only remotely works to give you an actual picture of what people think about it. Having a five scale score is very logical, since it's symetric (and therefore not biased like the 6 scores rating you suggest if you could rate without stars), and you give a degree of satisfaction and not just up or down. It's way superior to tell you if the product is exceptional or just "meh". Of course you have to look at the ratings distribution and not just the average. The average tells you almost nothing.

brownbat 13y ago

I wonder if a five star rating system could secretly count only the 2 and 4 star reviews without telling anyone, sort of filter out anyone passionate enough to try to game things...

mistercow 13y ago
That might work, but I have doubts. Interestingly, it's the opposite approach the the one taken by pre-employment personality tests, in which each question has no partial credit and exactly one right answer, which is either "strongly agree" or "strongly disagree".
- brownbat 13y ago
  Oh right! I think Barbara Ehrenreich mentioned that in Nickel and Dimed. At Walmart, one of the hiring surveys included some question pitting personal considerations or ethics against loyalty to management, and the only correct answer was absolute loyalty to management. When she debated the point, she was not offered a job.
  - thaumasiotes 13y ago
    In college, I got a part-time job for a stretch of some months grading exercises at the local Kumon center. The job consisted primarily of comparing a filled-out exercise booklet to a printed list of answers and marking incorrect answers. (The other part, the good part, was talking to the kids and trying to help them with problems they didn't get.) My boss was never happy with the speed at which I worked. Eventually she told me I was too smart for the job and should aim higher. In a vaguely similar way, I don't think Wal-mart is looking for their lowest-level employees to make a lot of independent ethical decisions on the job. If you're going to be hiring unskilled labor, you might not want them to exercise a lot of discretion, on the theory that they're at least as likely as not to screw it up.
hga 13y ago
I think the human mind is best at that. Even if there are too many reviews for you to completely read, you can sample them, especially the ones others have rated for usefulness, and with some practice recognize the ones that are passionate with sufficient support. Specificity is one think I really look for. Those who say essentially nothing more than "It's wonderful" or "It sucks" are noise. Those that say, "It's great, but I'm subtracting one star because the control console is a pain because of X, W and Y" are golden ... especially when people add comments on workarounds.... It's kind of funny, in that this "impersonal" mail order outfit very often manages to provide a better shopping experience than brick and mortar stores for those of us willing to buy things sight unseen. Heck, it's not uncommon for me to check Amazon.com reviews before buying something from one of those.
Amadou 13y ago
IMDB (which is a 10-star system) has a 'secret algorithm' that fiddles with the raw data to come up with a computed score. It's been suggested that some of the factors involved include reduced weight for 1 and 10 star ratings (as well as the age of the user's account - votes from new accounts don't get as much weight, etc).
jjoonathan 13y ago
I'd guess that moderation (as measured by 2-4 as opposed to 1 or 5 stars) is a worse predictor of an opinion's relevance than even the most simplistic automated classification of textual or behavioral feedback.

hga 13y ago

While you both have a point, I'm not sure it makes a difference in practice. OK, granted, I've been an Amazon.com customer since 1997 or so, but to me a 3.5 star rating is like writing BEWARE! in flaming letters across the sky. I never buy something that poorly rated before thoroughly investigating what's behind it. E.g. Kindle version's content is messed up I discount since I don't play in that ecosystem except for freebies. While there are plenty of mostly bi-modal distributions (5 or 1 stars), for lots of things there are carefully chosen ratings in-between, often telling you why it lost a star or 2. Even ones edited some time after with additional commentary and often an adjusted score. So I'd say it gets the specific job done, which is quite different than YouTube's where the cost of a wrong choice is very very small. I find the reviews invaluable, to the point where there's a lot of stuff I won't buy unless it has enough quality reviews for me to make an informed judgement.

fjarlq 13y ago
http://xkcd.com/1098/ http://xkcd.com/1098/

mcpherrinm 13y ago

I find the 5 star system on Amazon useful. I don't always look at the average rating, but rather the distribution graph of ratings. For example, one common shape looks like this: 5 stars: ||||||||||| 4 stars: |||||||| 3 stars: |||| 2 stars: || 1 star: ||||||| There's way more 1 star than 2 or 3. That usually indicates the product shows up DOA for a lot of people, or is otherwise probabilistically totally unsuitable for some group of people. If the number of star ratings follows a more normal curve like the following, then I'm more confident in the quality of the product, even if both have an average 4 star rating: 5 stars: ||||||| 4 stars: |||||||| 3 stars: |||| 2 stars: || 1 star: |

mashmac2 13y ago
This would make a fascinating study of reliability - can you determine statistically if a product will be reliable based on rating distribution. You could study Amazon, App Store ratings, etc. Cool insight!
- sanskritabelt 13y ago
  It's been done, and I've seen it as a worked example in introduction to Bayesian stats pages, maybe some of which were posted on HN.
- arh68 13y ago
  I wouldn't take any rating metric seriously unless it accounted for how long they've owned it. Plus there's the difference between "I've owned 12 lawnmowers and this one's pretty okay, only broke down after 1.5yrs, 3 stars" and "this is my first lawnmower, used it once, it's amazing, never going back to cutting my lawn with scissors, 5 stars". Plus niche sites sometimes have better reviews for the same products (newegg). All that said, there is information to be harvested in some of these reviews.
  - ludzone 13y ago
    Good point about niche sites and the length of ownership. I've noticed that a number of starts on products like top rated power supplies goes down after a year an a half or so. While you might be getting a newly released power supply with 500 reviews averaging closer to 5 stars, after 2 years, you will notice that same product with 1600 reviews averaging at below 4 stars, because owners probably came back and updated their reviews. It would be interesting to see a trend line of change in sentiment about a products over time, using existing data.

MichaelGG 13y ago

Netflix does a great job by showing hint text of "Hated it" for 1 star. It certainly helps calibrate people, rather than having individuals come up with their own meaning of what 1-5 means.

ekianjo 13y ago

up/down systens are stupid since they are so binary. It only remotely works to give you an actual picture of what people think about it. Having a five scale score is very logical, since it's symetric (and therefore not biased like the 6 scores rating you suggest if you could rate without stars), and you give a degree of satisfaction and not just up or down. It's way superior to tell you if the product is exceptional or just "meh". Of course you have to look at the ratings distribution and not just the average. The average tells you almost nothing.