3 ms·

This seems like something that can be corrected via normalization, at the per-user level. Find the average rating (and variation) for each individual user. Use

by whack 9y ago

This seems like something that can be corrected via normalization, at the per-user level. Find the average rating (and variation) for each individual user. Use these 2 metrics, in order to transform all user-supplied-ratings into normalized-ratings. For example, if someone always gives a 5 rating, then treat all their 5 ratings as 3s, and any sub-5 ratings as 1s. Conversely, if someone mostly gives 2s, then any of their 3s/4s should be treated as 5s.

Steuard 9y ago
I once had similar thoughts about, say, movie rating systems like IMDB. But you can pretty quickly convince yourself that straightforward ways of implementing this would open themselves up to horrific levels of system gaming very easily: a user who gives everyone a 2 but occasionally gives their best friend a 5 will wind up having that "5" normalized to some astronomically great rating (which even by itself could heavily inflate the friend's average). And every fix for that sort of thing that I came up with opened the door to yet another exploitable flaw or failure to accurately reflect user preferences. It's a hard problem. And especially in cases like Uber, I don't think there's a technical fix: as long as the company puts the cutoff for continued employment above the second-highest rating, there's a high perceived moral cost to giving a rating less than the maximum.
- whack 9y ago
  > a user who gives everyone a 2 but occasionally gives their best friend a 5 will wind up having that "5" normalized to some astronomically great rating You can just cap the max-value at a 5, to avoid someone getting a "astronomically great rating". Rating normalization isn't meant to prevent rating-fraud, nor does it make the problem significantly better/worse than it currently is. But it will solve the problem of rating inflation. Not to mention that once users know that their ratings are being normalized anyway, they won't feel the need to give everyone a 5-star rating.
uremog 9y ago
With Uber though, they specifically instruct that a normal, decent ride should be considered a 5. Does this still work when most users rate most rides a 5?
- whack 9y ago
  Once you know the user's average/variation, you can normalize it in any way you want. If you think that the "average user experience" merits a 5, you can do that as well. The downside of course, is that the above-average drivers don't get any recognition at all for going the extra mile.

by whack 9y ago

Steuard 9y ago

I once had similar thoughts about, say, movie rating systems like IMDB. But you can pretty quickly convince yourself that straightforward ways of implementing this would open themselves up to horrific levels of system gaming very easily: a user who gives everyone a 2 but occasionally gives their best friend a 5 will wind up having that "5" normalized to some astronomically great rating (which even by itself could heavily inflate the friend's average). And every fix for that sort of thing that I came up with opened the door to yet another exploitable flaw or failure to accurately reflect user preferences. It's a hard problem. And especially in cases like Uber, I don't think there's a technical fix: as long as the company puts the cutoff for continued employment above the second-highest rating, there's a high perceived moral cost to giving a rating less than the maximum.

whack 9y ago
> a user who gives everyone a 2 but occasionally gives their best friend a 5 will wind up having that "5" normalized to some astronomically great rating You can just cap the max-value at a 5, to avoid someone getting a "astronomically great rating". Rating normalization isn't meant to prevent rating-fraud, nor does it make the problem significantly better/worse than it currently is. But it will solve the problem of rating inflation. Not to mention that once users know that their ratings are being normalized anyway, they won't feel the need to give everyone a 5-star rating.

uremog 9y ago

With Uber though, they specifically instruct that a normal, decent ride should be considered a 5. Does this still work when most users rate most rides a 5?

whack 9y ago
Once you know the user's average/variation, you can normalize it in any way you want. If you think that the "average user experience" merits a 5, you can do that as well. The downside of course, is that the above-average drivers don't get any recognition at all for going the extra mile.