5 ms·

One possible partial solution is to express the quality of the source of a one-star review. A one-star review from someone who has posted many, many 4 and 5-st

by beloch 12y ago

One possible partial solution is to express the quality of the source of a one-star review. A one-star review from someone who has posted many, many 4 and 5-star reviews should carry some weight, while a one-time reviewer or someone with a history of giving many 1-star reviews should not. Perhaps color coding such reviews, or preferentially ranking reviews from the reliable at the top. This might lead to some legitimate 1-star reviews being ignored and it is at odds with the anonymity most people prefer on the internet, but it's better than presenting all 1-star reviews equally. Companies that make employment decisions based on these reviews might also choose to weight reviews based on the reliability of the source as well.

dragonwriter 12y ago
> A one-star review from someone who has posted many, many 4 and 5-star reviews should carry some weight, while a one-time reviewer or someone with a history of giving many 1-star reviews should not. Shouldn't the reverse be true as well? A 5-star review from someone who gives lots of a 1- and 2-star reviews should mean a lot more than a 5-star review from someone with little history or a history of giving many 5-star reviews. Really, a big problem here is cultural norms vary and subjective ordinal ratings don't mean the same thing to different raters -- ISTR studies showing that White Americans tend to rate things near the top of a scale if they are acceptable, toward the middle but still above the center even when they are unacceptable, and below the center only rarely -- whereas different nationalities and ethnicities had measurably different patterns of ratings compared to their actual descriptions of their responses to products. If that's true, a one-sided selective discounting of reviewers that give lots of low-star reviews may be effectively saying "I want reviews from White Americans to count the most."
- __david__ 12y ago
  That got me thinking. If you averaged the users' total reviews and then centered their score on a slightly larger scale then you could accomplish something like that: Adjusted Value 1 2 3 4 5 6 7 8 9 Ave +------------------ 1 | 1 2 3 4 5 2 | 1 2 3 4 5 3 | 1 2 3 4 5 4 | 1 2 3 4 5 5 | 1 2 3 4 5 So if I rate a bunch of things and my average review score is 4, then things I rate as a "1" would have an adjusted value of "2", and things I rate as a "5" have an adjusted value of "6". That would "find everyone's center" and maybe score things more evenly.
- deciplex 12y ago
  Is that because they're using the same scale as for grading school performance? So, three stars out of five is seen as 60% which is only barely passing, when in fact it should be for average service. Weighting reviews based on the variance in the history of that reviewer seems like a great idea to me, because it gives preference to reviews that probably carry more information in them. A review of five stars from someone who always hands out five star reviews, is meaningless and doesn't inform anyone else of anything. Likewise for one star reviews. In fact, intuitively I think maybe even a five star review from someone who usually hands out one star reviews, or vice versa, isn't very informative either? Reason being that these people clearly don't put a lot of thought into their reviews in the first place.

by beloch 12y ago

dragonwriter 12y ago

> A one-star review from someone who has posted many, many 4 and 5-star reviews should carry some weight, while a one-time reviewer or someone with a history of giving many 1-star reviews should not. Shouldn't the reverse be true as well? A 5-star review from someone who gives lots of a 1- and 2-star reviews should mean a lot more than a 5-star review from someone with little history or a history of giving many 5-star reviews. Really, a big problem here is cultural norms vary and subjective ordinal ratings don't mean the same thing to different raters -- ISTR studies showing that White Americans tend to rate things near the top of a scale if they are acceptable, toward the middle but still above the center even when they are unacceptable, and below the center only rarely -- whereas different nationalities and ethnicities had measurably different patterns of ratings compared to their actual descriptions of their responses to products. If that's true, a one-sided selective discounting of reviewers that give lots of low-star reviews may be effectively saying "I want reviews from White Americans to count the most."

__david__ 12y ago
That got me thinking. If you averaged the users' total reviews and then centered their score on a slightly larger scale then you could accomplish something like that: Adjusted Value 1 2 3 4 5 6 7 8 9 Ave +------------------ 1 | 1 2 3 4 5 2 | 1 2 3 4 5 3 | 1 2 3 4 5 4 | 1 2 3 4 5 5 | 1 2 3 4 5 So if I rate a bunch of things and my average review score is 4, then things I rate as a "1" would have an adjusted value of "2", and things I rate as a "5" have an adjusted value of "6". That would "find everyone's center" and maybe score things more evenly.
deciplex 12y ago
Is that because they're using the same scale as for grading school performance? So, three stars out of five is seen as 60% which is only barely passing, when in fact it should be for average service. Weighting reviews based on the variance in the history of that reviewer seems like a great idea to me, because it gives preference to reviews that probably carry more information in them. A review of five stars from someone who always hands out five star reviews, is meaningless and doesn't inform anyone else of anything. Likewise for one star reviews. In fact, intuitively I think maybe even a five star review from someone who usually hands out one star reviews, or vice versa, isn't very informative either? Reason being that these people clearly don't put a lot of thought into their reviews in the first place.