3 ms·

I'm far from an expert on statistics, but wouldn't the fact that star ratings tend to fall on a bimodal distribution just mean that the ratings that don't fit t

I'm far from an expert on statistics, but wouldn't the fact that star ratings tend to fall on a bimodal distribution just mean that the ratings that don't fit the bimodal distribution have a higher signal/noise ratio than the ones that do? I'd think you could use a 2-4 rating as an indicator of a thoughtful reviewer. Additionally, it seems like you could also use the fact that a particular reviewer doesn't fit the overall bimodal distribution as a sign that this reviewer's reviews are similarly more thought out.

Indy_Dh 10y ago
The fact that a reviewer uses 2 and 4 ratings might not be too meanngful. One reviewer might use 2s and 4s the way another uses 1s and 5s. There may be reviewers who do not generally fit the bimodal distribution, but if they are the minority, why design your whole rating system around them? Better to optimize your system for the way most of your users behave.
- sfifs 10y ago
  yes scaled ratings do tend to have subjectivity which in practice we normalize by databasing surveys over time for categories x markets. The reason online ratings tend to be bimodal is because the respondents are self selecting sample rather than randomized. Since product ratings are not typically compulsory, the extreme likers/dis-likers tend to fill the survey question. When implemented as compulsory, it drives poor user experience and having to fill a survey to get something anyway introduces a bias (just fill something to get it over with). So no easy answer. This is also the reason traditional survey firms haven't really gone out of business even with the wealth of online data from Facebook, Twitter and product reviews available, though better/easier access to actual behavioral signals are certainly replacing surveys in some areas.
StefanKarpinski 10y ago
I've built recommender systems at Etsy, among other places, and you get much better results if you make the data yes/no first. Shockingly MUCH better results. You'd think it would help to take into account, say how _many_ items of a shop a user likes, rather than just the fact that they liked some item, but it doesn't – it actively hurts. The only meaningful piece of information, it turns out is does this person like this thing or not.
- ghaff 10y ago
  That doesn't really surprise me. A lot of people imagine that highly granular, multi-dimensional rating systems must be best and they're often not. Of course, it depends a bit on what you're trying to distinguish but then it may not really matter to distinguish among the 75% that people are genuinely happy with.

Indy_Dh 10y ago

The fact that a reviewer uses 2 and 4 ratings might not be too meanngful. One reviewer might use 2s and 4s the way another uses 1s and 5s. There may be reviewers who do not generally fit the bimodal distribution, but if they are the minority, why design your whole rating system around them? Better to optimize your system for the way most of your users behave.

sfifs 10y ago
yes scaled ratings do tend to have subjectivity which in practice we normalize by databasing surveys over time for categories x markets. The reason online ratings tend to be bimodal is because the respondents are self selecting sample rather than randomized. Since product ratings are not typically compulsory, the extreme likers/dis-likers tend to fill the survey question. When implemented as compulsory, it drives poor user experience and having to fill a survey to get something anyway introduces a bias (just fill something to get it over with). So no easy answer. This is also the reason traditional survey firms haven't really gone out of business even with the wealth of online data from Facebook, Twitter and product reviews available, though better/easier access to actual behavioral signals are certainly replacing surveys in some areas.

StefanKarpinski 10y ago

I've built recommender systems at Etsy, among other places, and you get much better results if you make the data yes/no first. Shockingly MUCH better results. You'd think it would help to take into account, say how _many_ items of a shop a user likes, rather than just the fact that they liked some item, but it doesn't – it actively hurts. The only meaningful piece of information, it turns out is does this person like this thing or not.

ghaff 10y ago
That doesn't really surprise me. A lot of people imagine that highly granular, multi-dimensional rating systems must be best and they're often not. Of course, it depends a bit on what you're trying to distinguish but then it may not really matter to distinguish among the 75% that people are genuinely happy with.