4 ms·
We've analysed product ratings in the past and found that the online ratings tend to have primarily bimodal distributions. People who rate online welf select an
by sfifs 10y ago
We've analysed product ratings in the past and found that the online ratings tend to have primarily bimodal distributions. People who rate online welf select and either love something and give very good ratings or they dislike and give poor ratings. Unlike surveys where people have to answer questions, only people with extreme reactions tend to answer these online rating questions and you don't get any meaningful distributions. So the only meaningful way to model say drivers from comments vs. ratings is to treat the ratings as binary and do a logistic type model.
Ultimately the key reasons to have these ratings in the first place is not to let people express their feelings, it is to get data for the recommendation modeling to predict better. Perhaps showing a simple like/don't like increases the number of people rating something which gives you more training data and thus better business outcome from your model without compromising on midel quality since you had to anyway model it bi-modally before.
- fiddlerwoaroof 10y agoI'm far from an expert on statistics, but wouldn't the fact that star ratings tend to fall on a bimodal distribution just mean that the ratings that don't fit the bimodal distribution have a higher signal/noise ratio than the ones that do? I'd think you could use a 2-4 rating as an indicator of a thoughtful reviewer. Additionally, it seems like you could also use the fact that a particular reviewer doesn't fit the overall bimodal distribution as a sign that this reviewer's reviews are similarly more thought out.
- Indy_Dh 10y agoThe fact that a reviewer uses 2 and 4 ratings might not be too meanngful. One reviewer might use 2s and 4s the way another uses 1s and 5s. There may be reviewers who do not generally fit the bimodal distribution, but if they are the minority, why design your whole rating system around them? Better to optimize your system for the way most of your users behave.
- sfifs 10y agoyes scaled ratings do tend to have subjectivity which in practice we normalize by databasing surveys over time for categories x markets. The reason online ratings tend to be bimodal is because the respondents are self selecting sample rather than randomized. Since product ratings are not typically compulsory, the extreme likers/dis-likers tend to fill the survey question. When implemented as compulsory, it drives poor user experience and having to fill a survey to get something anyway introduces a bias (just fill something to get it over with). So no easy answer. This is also the reason traditional survey firms haven't really gone out of business even with the wealth of online data from Facebook, Twitter and product reviews available, though better/easier access to actual behavioral signals are certainly replacing surveys in some areas.
- StefanKarpinski 10y agoI've built recommender systems at Etsy, among other places, and you get much better results if you make the data yes/no first. Shockingly MUCH better results. You'd think it would help to take into account, say how _many_ items of a shop a user likes, rather than just the fact that they liked some item, but it doesn't – it actively hurts. The only meaningful piece of information, it turns out is does this person like this thing or not.
- ghaff 10y agoThat doesn't really surprise me. A lot of people imagine that highly granular, multi-dimensional rating systems must be best and they're often not. Of course, it depends a bit on what you're trying to distinguish but then it may not really matter to distinguish among the 75% that people are genuinely happy with.
- wj 10y agoI have no doubt you are right about the people who rate online (you definitely see it in restaurant ratings and comment sections) but when I rate movies on Netflix I very, very rarely ever rate something a 1 or a 5. I would imagine that 2/3 of what I rate has been a three (so average content for me) and that it takes something special to go up or down from there. Just curious if I am an outlier with how I do my ratings.
- vvvv 10y agoInteresting. I pretty much only rate 1s and 5s, hoping that this will influence what content is recommended to me.
- tbirrell 10y agoI'm the same way. 1 - Terrible. Good Lord, who thought making this was a good idea? 2 - Poor. Would not watch again 3 - Fair. Neither for or against. Unlikely to watch again, but I don't grudge the time spent. 4 - Good. I enjoyed it. I would probably watch again. 5 - Exceptional. Probably in my top 10 now, and foresee enjoying watching for years to come. 3 and 4 are used a lot. 2 is used occasionally. 1 and 5 are rare.
- fryguy 10y agoI prefer the Goodreads rating system which compresses 1 and 2: 1 - Did not like it 2 - It was ok 3 - Liked it 4 - Really liked it 5 - It was amazing The reason being is that most books won't end up being 1 or 2, and there really isn't much difference between them. But it sucks giving 2 stars to something that wasn't bad. Perhaps emoji arranged in a non-star-like pattern would be the best.
- ajmurmann 10y agoI've heard this a lot. When I look at reviews anywhere where there is a star rating and text review the only useful reviews tend to be 2-4 star reviews. 1 and 5 star reviews seem to have a tendency to be outliers or people who are really upset about something that most people won't care about. For example 1 star restaurant reviews that waited for someone to take their order and left because it took too long. I wonder if that could be applied to weighing the ratings accordingly. I have no data to test this unfortunately.