3 ms·
Here is how star rating actually works: For a truly excellent product, the distribution should approximate a power law graph: XXXXXXXXXXXXXXXX XXXXXX
by stcredzero 13y ago
Here is how star rating actually works:
For a truly excellent product, the distribution should approximate a power law graph:
XXXXXXXXXXXXXXXX
XXXXXX
XX
X
X
If you see something like these:
XXXXXXX
X
X
X
X
X
X
X
X
XXXXX
XXXXXXXXXXXXXXXX
XXXXXX
XXX
X
XXXXXXXX
Then ask yourself if the sample size makes this plausible, in which case the star rating isn't worth much, because there isn't enough data. Ask yourself if it's possible that someone has been gaming the system for or against.
The uppercase gamma-shaped graph is either someone gaming the system or it's a power law graph hidden by aliasing. (In which case, the thing is of godlike quality.)
We should do away with star ratings, and instead just present an aggregate "rating." Genuine customer ratings are valuable, but are almost immediately gamed by cheaters. If I were Apple or Amazon, I would build stealth captchas and "honeypot" traps for ratings manipulation, and figure those into the rating. Be sure to separate detection from consequences by a random time interval of at least a few days. This makes the "development cost" for the cheaters much higher.
The problem with star ratings is obviously what PG refers to as "schlepping". Is anyone tackling this already?
EDIT: I would surmise that AirBNB is somewhat tackling this problem already.