5 ms·

I feel like star ratings, if just given in general, are pretty useless for anything that has multiple qualities. For example, maybe I like the cinematography of

by yladiz 10y ago

I feel like star ratings, if just given in general, are pretty useless for anything that has multiple qualities. For example, maybe I like the cinematography of a movie but I think the acting is shit, or maybe I think the taste of the beer is nice but it's too acidic or sour and wouldn't work with specific foods because it's overpowering. Airbnb does this pretty much spot on, in that they ask for 1-5 stars for five categories, rather than just "did you enjoy this Airbnb?" I also think that Untappd does this wrong, because in general beers can't be described in just a 1-5 star rating without deeper explanation. I think that moving from a pure star rating to a thumbs up and down rating is better overall, if only because it makes me, as a watcher, not have to think as much and therefore give a rating where I might not have before. If I want to go more in depth, I can explain more too.

masklinn 10y ago
Incidentally, Steam ratings are also simple up/down, with total and sliding window 30 days aggregate, and an Amazon-style helpful/unhelpful/funny meta-rating.
- Houshalter 10y ago
  Steam ratings have their own problems. Like the creators changing something minor and getting thousands of protest negative reviews from upset people. Which doesn't reflect the quality of the game at all. But does discourage new people from trying the game. My favorite example is Fallout 4. There are tons of negative reviews from people who put 600+ hours into the game. How the hell do you put 25 days of your life into a game you don't like? They claim it "got boring". Not many games will not be boring after 600 hours...
curun1r 10y ago
> I feel like star ratings, if just given in general, are pretty useless for anything that has multiple qualities. This is only true when those star ratings are taken out of context. Individual reviewers can have completely coherent star ratings that are very useful for making determinations. The golden age of Netflix discoverability, for me, was back when they added social networking features. I connected my account with many friends and family and was really able to get a sense for everyone's taste in movies. When it came time to discover new movies/tv to watch, I'd open up the ratings history for someone who's tastes I felt aligned with the mood that I was in. Some of my friends would only rate high quality content with 5 stars, others would typically rate mass market, mindless entertainment with 5 stars and others tended to include a lot more 5-star reviews within a certain genre. Over time, I got a feel for that and could use it to my advantage. But when you remove that individual context and only aggregate 5 star reviews across a massive population who all view the 5-star scale in their own way, you get that uselessness that you talked about. Some people may reserve 5-star ratings for truly extraordinary content, but Netflix doesn't weight those ratings any more highly than the 5-star ratings of someone who rates nearly everything as 5 stars. This makes the ratings that Netflix displays meaningless. But that's only because Netflix's current methodology makes them meaningless, not because they're inherently meaningless.
- bcbrown 10y ago
  > Netflix doesn't weight those ratings any more highly than the 5-star ratings of someone who rates nearly everything as 5 stars. Is that fact or speculation? It's a common recommendation technique to find the mean rating for a person, and use that to normalize. Normalized score = (actual_score - mean_score + 2.5) for a 5-point scale, for example. I have no insight into whether or not Netflix does something similar, but I'd be surprised if they don't.
  - kartickv 10y ago
    Doesn't normalisation assume that a person who rates everything five-star has low standards? It may just be that that person rates something only if they're really (un)happy. Imagine two people watched 100 movies, and assessed them the same way, say that 20 are five-star, 20 are four-star, 20 are three-star, and so on. The first person rates all 100 movies, but the second one has a habit of writing a review only if he considers a movie excellent. Then your algorithm will wrongly discount the second person's ratings. Is this true in the real world? Have you found different people to have different habits, like rating everything vs rating only good movies vs rating only bad movies?
- kartickv 10y ago
  First, few people will put in effort into understanding their friends' psychology and habits to use Netflix effectively. We shouldn't optimise for such users. Second, I don't know whether individual ratings should be normalised as you say, but the final rating that's displayed should be normalised. For example, the app stores should have 20% of apps rated five stars, 20% four stars, and so on. This will make the ratings more objective and easy to understand. If the average rating is 4, then an an app rated 4 is merely average, which is counter-intuitive. (Perhaps normalised within a top-level category, like Productivity and Games)
kolbe 10y ago
I like to use the French Laundry example for how meaningless the star ratings are. Was it the best meal of my life? Yes. Was having the best meal of my life worth $1000? Not even close. So, did I have a 5-star meal, or did I have a one-star money waste? And are my preferences at all meaningful to a random person who has no knowledge of my finances or desire for good tasting food?
rdiddly 10y ago
Certainly the data you get will be of better quality that way, I just don't know that you'll get a huge quantity of it. Seems like for better or worse, Netflix is going for a higher quantity of ratings. (As well as the possible ulterior motive mentioned elsewhere in this thread, of making lame movies look not-so-bad - like helping out the dullards in class by grading it pass/fail.) An exquisitely detailed system like Airbnb's costs the user more time & hassle. Which means, assuming you make it voluntary, that a lot of them skip it. Meanwhile a simple/easy system like this, turns a scalpel into a sledgehammer somewhat, but you get a lot of them. So there's a balance to be struck for your site's particular userbase.
- yladiz 10y ago
  I agree, that's a good point. I think it's a different cost/value proposition with Airbnb vs Netflix, since you paid for a, sometimes expensive, service and then when the service is rendered you can review it, whereas with Netflix, while you are paying for the service you're not paying directly for a movie/show. So, with Airbnb, since you invested a bit of money you feel like you should take the time and review, but with Netflix, if it's not too simple you may just skip the review.
scott_s 10y ago
I think that binary ratings are the purest form of rating. But you have to ask the right question. "Did you enjoy this Airbnb?" is not the right question, because you end up wanting to give a nuanced answer. But "Would you recommend this Airbnb to a friend?" is, I think, the right binary question because it forces you to distill all of your likes and dislikes into a single, unambiguous decision. The nuance comes when you aggregate many people's binary decisions. A value near 1 tells you many people felt the good way outweighed the bad; a value near 0 tells you the opposite; and a value near 0.5 tells you that people are split. I think that's far more useful than trying to ask many individuals to provide you with a granular score.
kartickv 10y ago
That may be true from your point of view as a rater, but from mine as someone deciding whether to watch a movie, I want you to condense your thoughts down to a yes or no by assigning an appropriate weight to each factor. Was the acting so bad that it ruined the movie despite the good cinematography? For example, I found Mirror Mask enjoyable and would recommend it. They dreamed up and brought to life a fantastical world full of magic and wonder. The visuals are excellent. The story is mediocre, so you should watch it with the right expectations, but watch it nevertheless. The ability to distill something complex and multi-dimensional to a clear conclusion ("enjoyable and I recommend it" in the above paragraph) is a sign of clarity of thinking and expression. If you can't do that, I'll go by other people's ratings. Remember that the goal isn't to accurately capture one person's opinion, but to help potential viewers decide whether to watch it.

by yladiz 10y ago

masklinn 10y ago

Incidentally, Steam ratings are also simple up/down, with total and sliding window 30 days aggregate, and an Amazon-style helpful/unhelpful/funny meta-rating.

Houshalter 10y ago
Steam ratings have their own problems. Like the creators changing something minor and getting thousands of protest negative reviews from upset people. Which doesn't reflect the quality of the game at all. But does discourage new people from trying the game. My favorite example is Fallout 4. There are tons of negative reviews from people who put 600+ hours into the game. How the hell do you put 25 days of your life into a game you don't like? They claim it "got boring". Not many games will not be boring after 600 hours...

curun1r 10y ago

> I feel like star ratings, if just given in general, are pretty useless for anything that has multiple qualities. This is only true when those star ratings are taken out of context. Individual reviewers can have completely coherent star ratings that are very useful for making determinations. The golden age of Netflix discoverability, for me, was back when they added social networking features. I connected my account with many friends and family and was really able to get a sense for everyone's taste in movies. When it came time to discover new movies/tv to watch, I'd open up the ratings history for someone who's tastes I felt aligned with the mood that I was in. Some of my friends would only rate high quality content with 5 stars, others would typically rate mass market, mindless entertainment with 5 stars and others tended to include a lot more 5-star reviews within a certain genre. Over time, I got a feel for that and could use it to my advantage. But when you remove that individual context and only aggregate 5 star reviews across a massive population who all view the 5-star scale in their own way, you get that uselessness that you talked about. Some people may reserve 5-star ratings for truly extraordinary content, but Netflix doesn't weight those ratings any more highly than the 5-star ratings of someone who rates nearly everything as 5 stars. This makes the ratings that Netflix displays meaningless. But that's only because Netflix's current methodology makes them meaningless, not because they're inherently meaningless.

bcbrown 10y ago
> Netflix doesn't weight those ratings any more highly than the 5-star ratings of someone who rates nearly everything as 5 stars. Is that fact or speculation? It's a common recommendation technique to find the mean rating for a person, and use that to normalize. Normalized score = (actual_score - mean_score + 2.5) for a 5-point scale, for example. I have no insight into whether or not Netflix does something similar, but I'd be surprised if they don't.
- kartickv 10y ago
  Doesn't normalisation assume that a person who rates everything five-star has low standards? It may just be that that person rates something only if they're really (un)happy. Imagine two people watched 100 movies, and assessed them the same way, say that 20 are five-star, 20 are four-star, 20 are three-star, and so on. The first person rates all 100 movies, but the second one has a habit of writing a review only if he considers a movie excellent. Then your algorithm will wrongly discount the second person's ratings. Is this true in the real world? Have you found different people to have different habits, like rating everything vs rating only good movies vs rating only bad movies?
kartickv 10y ago
First, few people will put in effort into understanding their friends' psychology and habits to use Netflix effectively. We shouldn't optimise for such users. Second, I don't know whether individual ratings should be normalised as you say, but the final rating that's displayed should be normalised. For example, the app stores should have 20% of apps rated five stars, 20% four stars, and so on. This will make the ratings more objective and easy to understand. If the average rating is 4, then an an app rated 4 is merely average, which is counter-intuitive. (Perhaps normalised within a top-level category, like Productivity and Games)

kolbe 10y ago

I like to use the French Laundry example for how meaningless the star ratings are. Was it the best meal of my life? Yes. Was having the best meal of my life worth $1000? Not even close. So, did I have a 5-star meal, or did I have a one-star money waste? And are my preferences at all meaningful to a random person who has no knowledge of my finances or desire for good tasting food?

rdiddly 10y ago

Certainly the data you get will be of better quality that way, I just don't know that you'll get a huge quantity of it. Seems like for better or worse, Netflix is going for a higher quantity of ratings. (As well as the possible ulterior motive mentioned elsewhere in this thread, of making lame movies look not-so-bad - like helping out the dullards in class by grading it pass/fail.) An exquisitely detailed system like Airbnb's costs the user more time & hassle. Which means, assuming you make it voluntary, that a lot of them skip it. Meanwhile a simple/easy system like this, turns a scalpel into a sledgehammer somewhat, but you get a lot of them. So there's a balance to be struck for your site's particular userbase.

yladiz 10y ago
I agree, that's a good point. I think it's a different cost/value proposition with Airbnb vs Netflix, since you paid for a, sometimes expensive, service and then when the service is rendered you can review it, whereas with Netflix, while you are paying for the service you're not paying directly for a movie/show. So, with Airbnb, since you invested a bit of money you feel like you should take the time and review, but with Netflix, if it's not too simple you may just skip the review.

scott_s 10y ago

I think that binary ratings are the purest form of rating. But you have to ask the right question. "Did you enjoy this Airbnb?" is not the right question, because you end up wanting to give a nuanced answer. But "Would you recommend this Airbnb to a friend?" is, I think, the right binary question because it forces you to distill all of your likes and dislikes into a single, unambiguous decision. The nuance comes when you aggregate many people's binary decisions. A value near 1 tells you many people felt the good way outweighed the bad; a value near 0 tells you the opposite; and a value near 0.5 tells you that people are split. I think that's far more useful than trying to ask many individuals to provide you with a granular score.

kartickv 10y ago

That may be true from your point of view as a rater, but from mine as someone deciding whether to watch a movie, I want you to condense your thoughts down to a yes or no by assigning an appropriate weight to each factor. Was the acting so bad that it ruined the movie despite the good cinematography? For example, I found Mirror Mask enjoyable and would recommend it. They dreamed up and brought to life a fantastical world full of magic and wonder. The visuals are excellent. The story is mediocre, so you should watch it with the right expectations, but watch it nevertheless. The ability to distill something complex and multi-dimensional to a clear conclusion ("enjoyable and I recommend it" in the above paragraph) is a sign of clarity of thinking and expression. If you can't do that, I'll go by other people's ratings. Remember that the goal isn't to accurately capture one person's opinion, but to help potential viewers decide whether to watch it.