3 ms·

There are two issues with 5 star ranking. One the more thought required to give feedback the less likely users give feedback. More interactions is very helpful

There are two issues with 5 star ranking. One the more thought required to give feedback the less likely users give feedback. More interactions is very helpful for ml so minimizing the decisions needed by a user helps a lot. Great ui/ux for engagement aims for signals to require as little action from the user as possible. As a side effect short content is very nice as finish/playtime become strong signals that require a user to do almost nothing. The second issue is a lot of people use it like 1/5 system. While yes some users use more of the intermediates and you can adjust on per user basis for that feature it’s just not as useful as you’d expect. Of course any decision like this I recommend a company ab test and I’m just summarizing past experience in this area. I had a similar discussion at a prior job on having creators add content tags to there submission. The main avoidance was while good tags would be nice for recommendation, if tags are required less content would be submitted and that hurts the platform.

cardanome 5y ago
> One the more thought required to give feedback the less likely users give feedback. That is great! This means the people giving feedback would tend to be power users that are able to giver higher quality feedback. > As a side effect short content is very nice as finish/playtime become strong signals that require a user to do almost nothing. Perfect! It is clear by now that users vastly prefer short content that gets to the point quickly. So basically 5 stars ranking would vastly increase the quality of the content but that is bad, how? The only issue I see is (political) vote brigading. Maybe set additional hurdles like minimum account age and the like for voting. Strong content moderation could help. I guess there is no silver bullet for that issue.
- Mehdi2277 5y ago
  Your first point isn’t true because most of the 5 star usage boils down to 1/5 stars and not 2-4. There are some users that use whole range and the info does have value. The main problem though is while that system may be better for certain users, it will be worse for the entire user base as you want a system that works well for both casual and power users. Really you want a system that works better for casual users as most users on typical system like YouTube/spotify/tiktok/etc would be casual users. And while having model distinguish between user types is possible, ui wise you aren’t going to want to have both thumps up/down as an option and 5 stars. A toggle is possible but even for users that would use full ratings, a toggle is fairly unlikely to be used. Even for power users it may not be net benefit for them. Recommendations are not based solely on that user’s ratings but all ratings. If power users give better signal but total quantity of signals decreases then it’s likely that recommendation quality drops for both casual/power users. Making up guess numbers a 5 star system is likely to get you 3x less data points and many of those data points are of similar quality anyway (users that only rate 1/5 stars). Worse signal that has higher volume often leads to better model.
  - cardanome 5y ago
    My argument was based on the premise that votes from power users are more valuable as they better at judging the quality of the content. While the quality is partly subjective, it is not entirely so. Meaning power users, having more experience, know better what casual users should watch and would like then the casual users themselves. It is in the interest of the casual users for them to not be able to vote/their vote to count less. I later suggested ways to make the hurdles for voting higher, which would fix the problems you mentioned. Required account age, maybe even how much content you need to have watched. Honestly, we could just filter out the people that only do 1/5 star votes. (With niche content of course there would be the issue of having no/just a few votes but voting would not be the only metric anyway. I am very heretical towards common held UX believes and think trying to design for the lowest common denominator just results in all around awful experiences for everyone. We should rather strive to empower users.
    - Mehdi2277 5y ago
      I don’t think counting there vote less is useful. Low quality signal that is plentiful is very often better than sparse higher quality signal. My past experience is working on recommendation systems at tiktok/snap. Having a lot of weak signal from casual users is extremely helpful for power users. Recommendations are often based on large amount of basic engagement data. I think a large aspect to TikTok’s system being considered strong at recommendation is because Ux is designed for very frequent simple signals. I can also say working at companies like that generally any system changes require an ab test and long running back test. I’ve generally seen companies move from 5 star to simpler like/dislike system and those changes would have been ab tested and found that user experience is overall better. I don’t know for specific systems whether overall better was across all users or some users.
    - watwut 5y ago
      > My argument was based on the premise that votes from power users are more valuable as they better at judging the quality of the content. That is quite unlikely. What you call power user is someone distinguished by willingness to rank. It does not make that person better representation of all users or be more knowledgeable off quality or more experienced. Just more judgy. It means you will be recommended based on tastes of special minority and likely alienate the rest of users in the process.
tempestn 5y ago
I think these issues are more significant for public rankings. When I'm ranking songs, it's to curate my own listening. So if I only want to use 1 and 5 stars, fine—no worse off than a thumbs up/down system. But for those of us who do want to be more precise, it just gives more flexibility.
- Mehdi2277 5y ago
  The systems I worked on were mainly for user curated recommendations and not public rankings. Your rating is only small part of your recommendation. A large amount of your recommendation is finding other users with similar ratings on similar content to you. If 1/5 star system leads to less ratings from other users, then system's ability to find similar users/similar ratings become worse and can negatively impact your recommendations.

cardanome 5y ago

> One the more thought required to give feedback the less likely users give feedback. That is great! This means the people giving feedback would tend to be power users that are able to giver higher quality feedback. > As a side effect short content is very nice as finish/playtime become strong signals that require a user to do almost nothing. Perfect! It is clear by now that users vastly prefer short content that gets to the point quickly. So basically 5 stars ranking would vastly increase the quality of the content but that is bad, how? The only issue I see is (political) vote brigading. Maybe set additional hurdles like minimum account age and the like for voting. Strong content moderation could help. I guess there is no silver bullet for that issue.

Mehdi2277 5y ago
Your first point isn’t true because most of the 5 star usage boils down to 1/5 stars and not 2-4. There are some users that use whole range and the info does have value. The main problem though is while that system may be better for certain users, it will be worse for the entire user base as you want a system that works well for both casual and power users. Really you want a system that works better for casual users as most users on typical system like YouTube/spotify/tiktok/etc would be casual users. And while having model distinguish between user types is possible, ui wise you aren’t going to want to have both thumps up/down as an option and 5 stars. A toggle is possible but even for users that would use full ratings, a toggle is fairly unlikely to be used. Even for power users it may not be net benefit for them. Recommendations are not based solely on that user’s ratings but all ratings. If power users give better signal but total quantity of signals decreases then it’s likely that recommendation quality drops for both casual/power users. Making up guess numbers a 5 star system is likely to get you 3x less data points and many of those data points are of similar quality anyway (users that only rate 1/5 stars). Worse signal that has higher volume often leads to better model.
- cardanome 5y ago
  My argument was based on the premise that votes from power users are more valuable as they better at judging the quality of the content. While the quality is partly subjective, it is not entirely so. Meaning power users, having more experience, know better what casual users should watch and would like then the casual users themselves. It is in the interest of the casual users for them to not be able to vote/their vote to count less. I later suggested ways to make the hurdles for voting higher, which would fix the problems you mentioned. Required account age, maybe even how much content you need to have watched. Honestly, we could just filter out the people that only do 1/5 star votes. (With niche content of course there would be the issue of having no/just a few votes but voting would not be the only metric anyway. I am very heretical towards common held UX believes and think trying to design for the lowest common denominator just results in all around awful experiences for everyone. We should rather strive to empower users.
  - Mehdi2277 5y ago
    I don’t think counting there vote less is useful. Low quality signal that is plentiful is very often better than sparse higher quality signal. My past experience is working on recommendation systems at tiktok/snap. Having a lot of weak signal from casual users is extremely helpful for power users. Recommendations are often based on large amount of basic engagement data. I think a large aspect to TikTok’s system being considered strong at recommendation is because Ux is designed for very frequent simple signals. I can also say working at companies like that generally any system changes require an ab test and long running back test. I’ve generally seen companies move from 5 star to simpler like/dislike system and those changes would have been ab tested and found that user experience is overall better. I don’t know for specific systems whether overall better was across all users or some users.
  - watwut 5y ago
    > My argument was based on the premise that votes from power users are more valuable as they better at judging the quality of the content. That is quite unlikely. What you call power user is someone distinguished by willingness to rank. It does not make that person better representation of all users or be more knowledgeable off quality or more experienced. Just more judgy. It means you will be recommended based on tastes of special minority and likely alienate the rest of users in the process.

tempestn 5y ago

I think these issues are more significant for public rankings. When I'm ranking songs, it's to curate my own listening. So if I only want to use 1 and 5 stars, fine—no worse off than a thumbs up/down system. But for those of us who do want to be more precise, it just gives more flexibility.

Mehdi2277 5y ago
The systems I worked on were mainly for user curated recommendations and not public rankings. Your rating is only small part of your recommendation. A large amount of your recommendation is finding other users with similar ratings on similar content to you. If 1/5 star system leads to less ratings from other users, then system's ability to find similar users/similar ratings become worse and can negatively impact your recommendations.