4 ms·

I really appreciated the post. Unfortunately we're using 5 stars, and need to do it a bit less wrong. The main thorniness of 5 stars is that you have to answe

by YokoZar 15y ago

I really appreciated the post. Unfortunately we're using 5 stars, and need to do it a bit less wrong. The main thorniness of 5 stars is that you have to answer the question of what the difference in star ratings actually mean. Is going from one star to two stars the same as four to five? Probably not, based on the way users rate, which means an algorithm like the arithmetic mean that treats them the same is wrong. Personally, I think it's very reasonable to treat rating stars the way we should treat grades: as ordinal data, where we know that a higher rating is better but assume nothing beyond that. The difference between an F and a D is not the same as an A and a B, and the same is likely true of 1-2 stars and 4-5 stars. I have made an attempt at applying this idea to the Ubuntu App Store's rating algorithm. I'm very much interested in comments. https://bugs.launchpad.net/ubuntu/+source/software-center/+bug/894468 https://bugs.launchpad.net/ubuntu/+source/software-center/+b...

Someone 15y ago
"we know that a higher rating is better." Typically, you do not know that, especially if you are comparing across reviewers. Some will only score zero and 5 stars, other will have 10% two stars, 80% three stars, and 10% four stars, yet others will have 10% three stars, 80% four stars, and 10% five stars. If you have sufficient data (rare), it may be possible to (somewhat) correct for that. IIRC, this was something that helped in winning the Netflix challenge. For example, http://www.netflixprize.com/assets/GrandPrize2009_BPC_PragmaticTheory.pdf http://www.netflixprize.com/assets/GrandPrize2009_BPC_Pragma... models the assignment of stars as two parts: - modeling the user's appreciation of the movie - modeling how the user translates his appreciation to a star rating "where we know that a higher rating is better but assume nothing beyond that" Problem with that is that you throw away information with that assumption. You do know that 2 star scores are very unlikely to be about very good items.

by YokoZar 15y ago

Someone 15y ago

"we know that a higher rating is better." Typically, you do not know that, especially if you are comparing across reviewers. Some will only score zero and 5 stars, other will have 10% two stars, 80% three stars, and 10% four stars, yet others will have 10% three stars, 80% four stars, and 10% five stars. If you have sufficient data (rare), it may be possible to (somewhat) correct for that. IIRC, this was something that helped in winning the Netflix challenge. For example, http://www.netflixprize.com/assets/GrandPrize2009_BPC_PragmaticTheory.pdf http://www.netflixprize.com/assets/GrandPrize2009_BPC_Pragma... models the assignment of stars as two parts: - modeling the user's appreciation of the movie - modeling how the user translates his appreciation to a star rating "where we know that a higher rating is better but assume nothing beyond that" Problem with that is that you throw away information with that assumption. You do know that 2 star scores are very unlikely to be about very good items.