5 ms·

Even Netflix finally moved over to up/down and they were famous for squeezing every drop out of their previous star based reviews [1]. In theory stars work bett

Even Netflix finally moved over to up/down and they were famous for squeezing every drop out of their previous star based reviews [1]. In theory stars work better, but the issue seems to be everyone has a different ranking system. For example, Uber seems to think anything but a 5/5 is a failure. I know this so I skew to accommodate, but in my personal ranking system I've only had a couple 5 star rides (someone really going above and beyond). Up/down with an optional qualifier afterwards (e.g. "why were you unhappy?" after a thumbs down) seems to remove a lot of confusion. [1] https://en.wikipedia.org/wiki/Netflix_Prize https://en.wikipedia.org/wiki/Netflix_Prize

yoz-y 9y ago
Maybe the problem is with stars and wording. Currently most of systems are worded (e.g. amazon) in a way that 3 stars is the base and people would add stars if their expectations were exceeded and remove them if they were not met. However at all places that I have seen it is like you say 5 stars is for expectations being met and it only goes downhill from that. I think that a wording and iconography in 4 steps could be useful. -2 = something really bad happened, -1 = below expectations, 0 = happy customer, 1 = exceeded expectations. Forcing people to write a detail on any rating other than 0 would make most ratings 0. Angry people usually like to write comments anyways.
- ghaff 9y ago
  I'm not sure an expectations-based rating system is the norm though. To use an example I gave elsewhere. I order a cable from Amazon. It works. Therefore it met my expectations of a working cable. Yet, I think most people would interpret a 3-star rating as my being lukewarm on my purchase. I'm not. But what the heck do I expect a cable to do other than being a fair price and to work? With respect to movies. Some movies get really built up and I go in expecting great things (e.g. Fury Road). I come out thinking they were just OK. So maybe 3 stars. But definitely not -1 or 2 stars. My personal expectations aren't necessarily a good baseline.
  - yoz-y 9y ago
    For a single data point they are useless. But I thing a rating in aggregate could be informative. Of course, the rating is self defeating as too many people who were positively surprised will raise the expectations. For movies I would love to have reviews in style: Somebody writes up their expectations before going to the movie, and then rates the movie according to that. I find most "press and critics" reviews useless as if somebody is not a fan of story-less action movies, then why the hell is he reviewing them.
- enobrev 9y ago
  This is how I see it as well. It would also likely get rid of comments / ratings like "I would give this 0 stars if I could". I think the numbering you mention is just as essential (especially 0). The way I generally rate things falls in line with this idea: 5: Excellent 4: Pretty Good 3: Average; Unsurprising. Not Impressed, Not Disappointed 2: Kinda Sucks 1: Run Away This definitely doesn't seem to be how everyone else is using the 5-star scale.
dredmorbius 9y ago
Normalising a given rater's stars, or asssigning costs to higher / lower ratings, is another option. Essentially you have a "ratings budget" you can spend, up or down, on your assessments. Stack Exchange has this to an extent, where negative ratings cost the rater points -- you have to really want to assign a negative.
- deleted 9y ago
  [deleted]
stinkytaco 9y ago
Everyone does have a different ranking system, but Netflix was good at predicting what I would give a movie, so my ranking system was the only one that mattered. Their special sauce in the background made it pretty accurate. The problem with an up/down for me is that it doesn't capture an ambivalent reaction. That means transactions will tend toward the mode, imo. You will do enough to get a thumbs up, that's all.
- kalleboo 9y ago
  I often wish it was an up/down/love
ghaff 9y ago
This was (is? I haven't really used them much in a while) an issue with eBay's reputational system as well. Anything other than a Positive and "A++++++++++++ seller" was interpreted to mean they shipped you a box full of bricks rather than they took a week to ship things.
dennisgorelik 9y ago
5-star ratings probably carry more information than binary thumbs up/down, but every 5-star vote is more complex to collect, so Netflix was probably getting less 5-star votes than they are getting up/down votes. Overall number of votes matters too.

yoz-y 9y ago

Maybe the problem is with stars and wording. Currently most of systems are worded (e.g. amazon) in a way that 3 stars is the base and people would add stars if their expectations were exceeded and remove them if they were not met. However at all places that I have seen it is like you say 5 stars is for expectations being met and it only goes downhill from that. I think that a wording and iconography in 4 steps could be useful. -2 = something really bad happened, -1 = below expectations, 0 = happy customer, 1 = exceeded expectations. Forcing people to write a detail on any rating other than 0 would make most ratings 0. Angry people usually like to write comments anyways.

ghaff 9y ago
I'm not sure an expectations-based rating system is the norm though. To use an example I gave elsewhere. I order a cable from Amazon. It works. Therefore it met my expectations of a working cable. Yet, I think most people would interpret a 3-star rating as my being lukewarm on my purchase. I'm not. But what the heck do I expect a cable to do other than being a fair price and to work? With respect to movies. Some movies get really built up and I go in expecting great things (e.g. Fury Road). I come out thinking they were just OK. So maybe 3 stars. But definitely not -1 or 2 stars. My personal expectations aren't necessarily a good baseline.
- yoz-y 9y ago
  For a single data point they are useless. But I thing a rating in aggregate could be informative. Of course, the rating is self defeating as too many people who were positively surprised will raise the expectations. For movies I would love to have reviews in style: Somebody writes up their expectations before going to the movie, and then rates the movie according to that. I find most "press and critics" reviews useless as if somebody is not a fan of story-less action movies, then why the hell is he reviewing them.
enobrev 9y ago
This is how I see it as well. It would also likely get rid of comments / ratings like "I would give this 0 stars if I could". I think the numbering you mention is just as essential (especially 0). The way I generally rate things falls in line with this idea: 5: Excellent 4: Pretty Good 3: Average; Unsurprising. Not Impressed, Not Disappointed 2: Kinda Sucks 1: Run Away This definitely doesn't seem to be how everyone else is using the 5-star scale.

dredmorbius 9y ago

Normalising a given rater's stars, or asssigning costs to higher / lower ratings, is another option. Essentially you have a "ratings budget" you can spend, up or down, on your assessments. Stack Exchange has this to an extent, where negative ratings cost the rater points -- you have to really want to assign a negative.

deleted 9y ago
[deleted]

stinkytaco 9y ago

Everyone does have a different ranking system, but Netflix was good at predicting what I would give a movie, so my ranking system was the only one that mattered. Their special sauce in the background made it pretty accurate. The problem with an up/down for me is that it doesn't capture an ambivalent reaction. That means transactions will tend toward the mode, imo. You will do enough to get a thumbs up, that's all.

kalleboo 9y ago
I often wish it was an up/down/love

ghaff 9y ago

This was (is? I haven't really used them much in a while) an issue with eBay's reputational system as well. Anything other than a Positive and "A++++++++++++ seller" was interpreted to mean they shipped you a box full of bricks rather than they took a week to ship things.

dennisgorelik 9y ago

5-star ratings probably carry more information than binary thumbs up/down, but every 5-star vote is more complex to collect, so Netflix was probably getting less 5-star votes than they are getting up/down votes. Overall number of votes matters too.