4 ms·

I really don't see why, as the article claims, the fact that opinions are subjective and everyone has a different one makes the 1-5 star rating any more useless

by brazzy 17y ago

I really don't see why, as the article claims, the fact that opinions are subjective and everyone has a different one makes the 1-5 star rating any more useless than a thumbs up/thumbs down/do nothing or a favorite/not favorite rating. Those are subjective and different for everyone as well, and the idea is that it averages out over lots of people. No, the problem is that people can't be bothered to spend time to think about just how much they like something, compared to everything else, and that it can become uncomfortable to try and do that because you realize that your relative liking may not be neither constant nor consistent, and trying to make it so could be a lot of work (I like A a lot, but B even more... hey, C is really cool! But some things about A are better than C... now what?).

Timothee 17y ago
Actually, I remember reading a post around the Netflix prize that was basically saying that it wasn't important to know what people exactly thought 2, 3, 4... stars meant for them, because overall, it was still enough to obtain the data and rating relevant to your point-of-view.
- jhancock 17y ago
  Would be interesting to see Netflix data on this. Does is look the same as YouTube? If not, why?
potatolicious 17y ago
"I really don't see why, as the article claims, the fact that opinions are subjective and everyone has a different one makes the 1-5 star rating any more useless than a thumbs up/thumbs down" Because you're implementing a system that on paper has a lot more resolution than what you're really getting. Imagine buying a 1080p HDTV and then showing solid colors on it only - not only a waste of engineering effort but also building subsequent systems around the validity of your 5-star ratings will also be fundamentally broken. Also, based on their data there's a very concrete reason why the 1-5 star rating is worse than the thumbs up/thumbs down. With the thumbs up/down system you have a single dimension of data ("likedness"), whereas with the 1-5 star rating system they're only getting data from people who like the video (and almost none from people who disliked it, look at the distribution). This makes the data practically useless for determining the quality and user preference for a video. Consequently ranking algorithms just won't work on the star system - the difference between video #1 and #100,000 can be an average rating of 4.8 and 4.92. "Those are subjective and different for everyone as well" So are movie ratings - but it's still a very useful metric to a lot of people. With a large enough sample size you get the lowest common denominator preference measure - which may be what YouTube wants. "the problem is that people can't be bothered" I object to the labeling of basic user behaviour as laziness or some type of stupidity. Users will behave how they behave - assigning value judgments to this behaviour just makes you a prick, and disconnects you from your users (who also happen to be your customers, yay!). If your users aren't using your system in the way you intended, you need to fix it. Trying to pawn off your responsibility in the equation as "lazy users can't be bothered" simply is a cop-out.
- brazzy 17y ago
  Because you're implementing a system that on paper has a lot more resolution than what you're really getting. Imagine buying a 1080p HDTV and then showing solid colors on it only - not only a waste of engineering effort The difference is that the HDTV costs extra, while the rating resolution does not. but also building subsequent systems around the validity of your 5-star ratings will also be fundamentally broken. It depends on how you interpret and use the data. Also, based on their data there's a very concrete reason why the 1-5 star rating is worse than the thumbs up/thumbs down. With the thumbs up/down system you have a single dimension of data ("likedness"), whereas with the 1-5 star rating system they're only getting data from people who like the video (and almost none from people who disliked it, look at the distribution). This makes the data practically useless for determining the quality and user preference for a video. Again: With only people who like voting 5 stars and a few people who dislike voting 1 star you have the exact same information as with a "favoriting only" voting system, or as a thumbs up/down one with few people voting down (which is very likely). So how is it any more useless? The only real problem I see is that people probably interpret the ratings differently - to some, 1 star is "the worst possible vote", to others it is "one notch better than no vote". Consequently ranking algorithms just won't work on the star system - the difference between video #1 and #100,000 can be an average rating of 4.8 and 4.92. Which is not necessarily a problem, if you weigh the average by the number of votes. Sure, you don't want to rate a video with one 5-star rating higher than one with 5000 5-star ratings and one 1-star rating. But you don't have to. So are movie ratings - but it's still a very useful metric to a lot of people. With a large enough sample size you get the lowest common denominator preference measure - which may be what YouTube wants. Um, yeah. That's exactly what I said. I object to the labeling of basic user behaviour as laziness or some type of stupidity. You're reading something into my comment that I did not say. Actually yeah, it's lazy... and people have every right to be - they go to YouTube to be entertained, not because they're being paid for it. People do something because there are rewards. With user-generated content, the rewards are generally immaterial: attention, status, self-expression. The only reward for thoughtful ratings is self-expression, but very limited because nobody can see your vote. Even a single-word comment is more rewarding than that. So if you want people to spend the time and effort that thoughtful fine-grained ratings require, you'd have to add artificial rewards - which is almost impossible because you can't measure how thoughtful a rating is. The alternative it to make rating easier, so that more people will do it at the current reward level. And "easier" here means "requiring less thought".

by brazzy 17y ago

Timothee 17y ago

Actually, I remember reading a post around the Netflix prize that was basically saying that it wasn't important to know what people exactly thought 2, 3, 4... stars meant for them, because overall, it was still enough to obtain the data and rating relevant to your point-of-view.

jhancock 17y ago
Would be interesting to see Netflix data on this. Does is look the same as YouTube? If not, why?

potatolicious 17y ago

"I really don't see why, as the article claims, the fact that opinions are subjective and everyone has a different one makes the 1-5 star rating any more useless than a thumbs up/thumbs down" Because you're implementing a system that on paper has a lot more resolution than what you're really getting. Imagine buying a 1080p HDTV and then showing solid colors on it only - not only a waste of engineering effort but also building subsequent systems around the validity of your 5-star ratings will also be fundamentally broken. Also, based on their data there's a very concrete reason why the 1-5 star rating is worse than the thumbs up/thumbs down. With the thumbs up/down system you have a single dimension of data ("likedness"), whereas with the 1-5 star rating system they're only getting data from people who like the video (and almost none from people who disliked it, look at the distribution). This makes the data practically useless for determining the quality and user preference for a video. Consequently ranking algorithms just won't work on the star system - the difference between video #1 and #100,000 can be an average rating of 4.8 and 4.92. "Those are subjective and different for everyone as well" So are movie ratings - but it's still a very useful metric to a lot of people. With a large enough sample size you get the lowest common denominator preference measure - which may be what YouTube wants. "the problem is that people can't be bothered" I object to the labeling of basic user behaviour as laziness or some type of stupidity. Users will behave how they behave - assigning value judgments to this behaviour just makes you a prick, and disconnects you from your users (who also happen to be your customers, yay!). If your users aren't using your system in the way you intended, you need to fix it. Trying to pawn off your responsibility in the equation as "lazy users can't be bothered" simply is a cop-out.

brazzy 17y ago
Because you're implementing a system that on paper has a lot more resolution than what you're really getting. Imagine buying a 1080p HDTV and then showing solid colors on it only - not only a waste of engineering effort The difference is that the HDTV costs extra, while the rating resolution does not. but also building subsequent systems around the validity of your 5-star ratings will also be fundamentally broken. It depends on how you interpret and use the data. Also, based on their data there's a very concrete reason why the 1-5 star rating is worse than the thumbs up/thumbs down. With the thumbs up/down system you have a single dimension of data ("likedness"), whereas with the 1-5 star rating system they're only getting data from people who like the video (and almost none from people who disliked it, look at the distribution). This makes the data practically useless for determining the quality and user preference for a video. Again: With only people who like voting 5 stars and a few people who dislike voting 1 star you have the exact same information as with a "favoriting only" voting system, or as a thumbs up/down one with few people voting down (which is very likely). So how is it any more useless? The only real problem I see is that people probably interpret the ratings differently - to some, 1 star is "the worst possible vote", to others it is "one notch better than no vote". Consequently ranking algorithms just won't work on the star system - the difference between video #1 and #100,000 can be an average rating of 4.8 and 4.92. Which is not necessarily a problem, if you weigh the average by the number of votes. Sure, you don't want to rate a video with one 5-star rating higher than one with 5000 5-star ratings and one 1-star rating. But you don't have to. So are movie ratings - but it's still a very useful metric to a lot of people. With a large enough sample size you get the lowest common denominator preference measure - which may be what YouTube wants. Um, yeah. That's exactly what I said. I object to the labeling of basic user behaviour as laziness or some type of stupidity. You're reading something into my comment that I did not say. Actually yeah, it's lazy... and people have every right to be - they go to YouTube to be entertained, not because they're being paid for it. People do something because there are rewards. With user-generated content, the rewards are generally immaterial: attention, status, self-expression. The only reward for thoughtful ratings is self-expression, but very limited because nobody can see your vote. Even a single-word comment is more rewarding than that. So if you want people to spend the time and effort that thoughtful fine-grained ratings require, you'd have to add artificial rewards - which is almost impossible because you can't measure how thoughtful a rating is. The alternative it to make rating easier, so that more people will do it at the current reward level. And "easier" here means "requiring less thought".