11 ms·
As I posted in the dupe: This saddens me. >Users would rate documentaries with 5 stars, and silly movies with just 3 stars, but still watch silly movies more
by pizzetta 10y ago
As I posted in the dupe:
This saddens me.
>Users would rate documentaries with 5 stars, and silly movies with just 3 stars, but still watch silly movies more often than those high-rated documentaries
That's not incongruous to me. The stars are not about "enjoyment" factor, they are about perceived quality. I may have a go to cheap ice-cream and rate it 3-stars but rate a good affogato 5-stars and only have it once in a while.
They are diluting the meaning of quality and instead are opting for a saccharine "enjoyment" factor.
This binary choice does not sit well with me and I hope they abandon the idea soon.
- bluntfang 10y agoWhen I watch netflix, I use the stars as a measure of if I would enjoy something. I don't think most people use stars as a measure of "quality" because when they browse movies, they are looking for something they enjoy, not something that's high quality that they might not enjoy.
- bmelton 10y agoThis seems like pure difficulty for me. When I think of thumbs up / down metrics I think in terms of "would I endorse this?" A movie like "Suicide Squad" was (to me at least) highly enjoyable, though I thought it objectively had many problems with plot and delivery. I would probably give it 3 stars, which is relatively neutral for me (or leave it altogether unrated). If it were thumbs up/down, I would probably have to leave it unrated completely. Maybe that's an improvement, but the more discussion I see around this, the more I realize the futility of single-axis metrics for things like this, until and unless you're asking a single question "Did you enjoy this movie" is a much different question from "is this a good movie" -- and I don't really know which one I've been answering on Netflix all these years past.
- bluntfang 10y agoI think you are an anomalous user. I'm not sure why you're OK with rating 1-5 stars but not a thumbsup or down. I think you're just upset because they are changing.
- phy6 10y agoLets say a movie previously would have shown 3 out of 5 stars. Where does that fit in with the binary thumbs up/down? I think they are just doing this because recently some actors movies are being attacked with 1 star ratings for being politically vocal.
- CaptSpify 10y agoI like giving weight to my vote. 3 stars = "not a bad movie, but wouldn't recommend" 4 starts = "decent movie, would recommend" 5 stars = "great movie, would definitely recommend" A binary choice doesn't have any weight to it.
- __david__ 10y agoI agree. I even want a couple more increments between 2,3, and 4. There are a lot of movies I see that I think are better or worse than a 3 rating but don't justify a 2 or a 4. I feel there's a lot of room for nuance around the middle area.
- kartickv 10y agoI don't think there's a need for weight since there are many reviewers. The weight is the percentage of people who thumbed it up.
- sethammons 10y agoThe current suggested rating is influenced by your history of rating, with weight given to how highly you rated it. There are shows that are suggested at 4 stars for my wife but at 2 stars for me.
- CaptSpify 10y agoHow does weight have anything to do with amount of reviewers? Those are completely unrelated metrics.
- sethammons 10y agoThis. And based on my history of rating, Netflix had been pretty good at guessing if something will be 3+ stars. If they think it is 3 stars, I consider watching. If they think it is 4+, I tend to really enjoy the suggestion. Though, nearly every Netflix original is suggested at 5 stars, lol.
- wtetzner 10y agoI think a lot of people look at the stars as a measure of enjoyment when trying to find something to watch, but when actually rating something, as a measure of quality.
- bobwaycott 10y agoI use the stars to signal quality as a means of determining if I'd enjoy it. 5 stars? This is likely to be high-quality and enjoyable. 3 stars? This is going to be lower quality and I may not enjoy it. 1 star? This is likely shit quality, and I'll regret playing—or it's a cult classic and all the haters don't get it.
- bobwaycott 10y agoI use the stars to signal quality as a means of determining if I'd enjoy it. 5 stars? This is likely to be high-quality and enjoyable. 3 stars? This is going to be lower quality and I may not enjoy it. 1 star? This is likely shit quality, and I'll regret playing—or it's a cult classic and all the haters don't get it.
- SyneRyder 10y agoI've had this argument with my (tending towards elderly) parents while introducing them to Netflix. They don't understand rating for enjoyment, and keep rating for "quality". They'll watch a show, absolutely hate it, but give it a 4 or 5 star rating because "it was well produced and the camera work was good, but it isn't the sort of thing I personally like to watch". I'm worried about the thumbs up/down, as Netflix has increasingly recommended shows I do not like this year, and I don't think the thumbs solves the "rating for perceived quality" issue. I've been using Movielens for recommendations a lot instead, and it seems more accurate for me... and it has enough stats to show that on their site, most rating is not binary, much closer to a normal curve for average movies or a linear graph on really good movies. (It lets you see the distribution curve on individual movies, and other metrics.)
- johnny22 10y agointeresting. I always rated for enjoyment. I thought that Neflix made it pretty clear that the ratings were used to choose which content to show me. So I should only rate based on what I'd actually like to see.
- deleted 10y ago[deleted]
- ProAm 10y ago> They are diluting the meaning of quality and instead are opting for a saccharine "enjoyment" factor. They are just trying to increase number of views for a poor quality library. This will mask how bad most of the content is and increase their numbers.
- M_Grey 10y agoOr edge cases like me will get that little boost to just give up on Netflix. I already use them less than other streaming services, and I could always just turn it on and off again.
- karlshea 10y agoHonestly one of the only reasons I keep it is because I have so many movies that I've rated and didn't want that time investment to go to waste. I guess now that isn't going to be a factor.
- M_Grey 10y agoThat was definitely part of it or me too; the rating system has always been a draw. It took time to make it actually respond the way I needed it to, and now it's just... sigh
- pfd1986 10y agoThis! My dream is to build an open source app that gathers all these personal ratings (from Netflix to Pandora to GoodReads to restaurants) and puts it back on where it belongs: the users hand!! Unfortunately finding the time to work on this will be tricky.. Anyone knows if that's a thing already?
- bad_user 10y agoWhy on earth would you use Netflix as a rating service? First of all their library lacks a lot of great movies, or info about the movies they have. Netflix isn't IMDB.
- 10y ago
- JTon 10y agoI agree with you. And hat tip for the nice analogy. Perhaps they should implement a system analogous to facebook's Reactions (e.g. "disliked", "thought provoking", "easy watching", etc) as a way to differentiate between cheap/easy and high quality releases.
- lumisota 10y agoShouldn't this quality rating be within a genre? Not all documentaries are 5-star, and not all silly movies are 3-stars. How do you compare ice-cream with cheesecake?
- Pxtl 10y agoYeah, if they went with up/downs but allowed genre up/downs I think this would correct some anxiety about this change. "Yes, I still want to watch anime. No, the last 4 shows you recommended to me were not good."
- Fifer82 10y ago> The stars are not about "enjoyment" factor, they are about perceived quality I ignore ratings because of that. They don't mean anything and I am guilty. On Good Reads, I have given what is deemed as a trashy piece of writing 5 stars. At the same time, I have given Dune 1 star because the story was garbage. So what is this 5 star rating? Some people say it is the experience, some say it is the writing style, others say it is the story telling, or the plot. Lets also face it, when it comes to films, the news papers have their own agendas and personal commitments to other agencies. My film could be absolutely garbage, but since I dined with the Editor a few times, It gets 4/5 stars and then, it gets 4/5 stars because the rest of the population is told how to feel. I would even say that this whole system of rating based on Stars or Likes can simply be removed. If a film is good, you will know about it. If a film has been watched 1000000 times more this weekend than another, base it on that if anything.
- chiliap2 10y agoYour thesis is basically, ratings mean absolutely nothing, and how good something is should solely be based on how popular it is?
- masklinn 10y agoThe problem as I understand it is that the "rating space" is multidimensional (a piece can be funny with great plot but bad writing compensated for by a great lead and awesome special effects although the set design was inconsistent) but all of this has to be projected into a 1-dimensional system. And the rub is that not everybody uses the same projection, which means the utility of one person's rating for another unknown their projection system (the weight they give to each sub-component of the overall rating) is completely unknown.
- deleted 10y ago[deleted]
- Fifer82 10y agoA much more succinct way of making the point. Thank you.
- bryanlarsen 10y agoThey're optimizing the wrong metric. They should be optimizing "number of people who pay for Netflix". Instead they are optimizing "number of hours spent watching Netflix". Those two metrics are correlated, but definitely not fully. If I spend hundreds of hours a month watching meaningless action movies and TV on Netflix, I can cancel my subscription and save myself hundreds of hours and $10. I've got lots of other ways of filling my time with endless pablum. If I spend 2 hours watching a thought-provoking documentary, I say to myself "yup, that justified spending $10 on Netflix this month."
- sweezyjeezy 10y agoHow do you know what metric Netflix are optimising?
- scruple 10y agoThey tell you so in TFA. > Users would rate documentaries with 5 stars, and silly movies with just 3 stars, but still watch silly movies more often than those high-rated documentaries.
- s_m_mcguire 10y agoAccounting for your average person, who do you estimate is more likely to cancel: The person watching hundreds of hours of meaningless action movies and TV, or the person subscribing but rarely/never watching anything? By average I mean not prone to suddenly realizing they want/feel sufficiently motivated to spend those hundreds of hours on something else, or posting on hacker news. Speaking anecdotally, the one (temporary) time I canceled was when I found myself in the paying-but-not-using category.
- oculusthrift 10y agoit's interesting because the more someone uses netflix, the less money netflix makes.
- TylerE 10y agoNot necessarily. I strongly suspect (but can't prove) that they sell their rating data back to the studios.
- sweezyjeezy 10y agoNetflix have said for years that the star ratings they gather are not actually very good indicators for their recommendation engines compared to implicit signals such as - 'did the user watch the show / film to the end?' If they still believe this, it seems fairly irrelevant what they use.
- TeMPOraL 10y ago> implicit signals such as - 'did the user watch the show / film to the end?' I hope their version of that signal is better than what they're presenting in the UI. I have a few movies Netflix shows me as not finished, becaue I closed them just as the ending credits started.
- always_good 10y agoAlso, it seems incredibly delicate. Upon finishing the final season of Dexter, I clicked on an episode in the first season to look up something. I jumped to the 18 minute mark and watched about 10 seconds and closed the tab. Now Netflix asks me every time if I want to continue watching Dexter from where it now thinks I left off in season one even though I spent a whole year watching all of the seasons. So I have the same concern. If they have a more sophisticated model, it certainly doesn't get reflected in the UI...
- sundvor 10y agoThe UI... They can't even make an 'f' shortcut key in the Win10 app (app, not browser) to go fullscreen. And they force 21:9 movies to be played back inside the 16:9 portion of a 21:9 ultrawide monitor, with massive black borders as a result where there should be none. I'm seriously unimpressed with Netflix as a company, for what they do with all their resources. Sometimes the MVP just isn't good enough.
- sweezyjeezy 10y agoNetflix have said for years that the star ratings they gather are not actually very good indicators for their recommendation engines compared to implicit signals such as - 'did the user watch the show / film to the end?' If they still believe this, it seems fairly irrelevant what they use.
- caseysoftware 10y ago> The stars are not about "enjoyment" factor, they are about perceived quality. For you. But are you representative of the whole or even a majority? I personally use it to reflect whether I enjoyed it or considered it a waste of time. I give some of my favorites - like Hackers - 5 stars but would never consider it a "quality" movie. It's awful and silly but that's the fun of it.
- unsigner 10y agoIf you are sophisticated enough to appreciate "foreign documentaries", you should have little trouble discovering them by another means than Netflix rating algorithm. The latter is for the laziest, couch-potatoest audience, and it makes sense to optimize for that.
- chocolatebunny 10y agoBut Netflix doesn't care about the quality of a movie. They care about whether you would watch it or not.
- sfifs 10y agoWe've analysed product ratings in the past and found that the online ratings tend to have primarily bimodal distributions. People who rate online welf select and either love something and give very good ratings or they dislike and give poor ratings. Unlike surveys where people have to answer questions, only people with extreme reactions tend to answer these online rating questions and you don't get any meaningful distributions. So the only meaningful way to model say drivers from comments vs. ratings is to treat the ratings as binary and do a logistic type model. Ultimately the key reasons to have these ratings in the first place is not to let people express their feelings, it is to get data for the recommendation modeling to predict better. Perhaps showing a simple like/don't like increases the number of people rating something which gives you more training data and thus better business outcome from your model without compromising on midel quality since you had to anyway model it bi-modally before.
- fiddlerwoaroof 10y agoI'm far from an expert on statistics, but wouldn't the fact that star ratings tend to fall on a bimodal distribution just mean that the ratings that don't fit the bimodal distribution have a higher signal/noise ratio than the ones that do? I'd think you could use a 2-4 rating as an indicator of a thoughtful reviewer. Additionally, it seems like you could also use the fact that a particular reviewer doesn't fit the overall bimodal distribution as a sign that this reviewer's reviews are similarly more thought out.
- Indy_Dh 10y agoThe fact that a reviewer uses 2 and 4 ratings might not be too meanngful. One reviewer might use 2s and 4s the way another uses 1s and 5s. There may be reviewers who do not generally fit the bimodal distribution, but if they are the minority, why design your whole rating system around them? Better to optimize your system for the way most of your users behave.
- sfifs 10y agoyes scaled ratings do tend to have subjectivity which in practice we normalize by databasing surveys over time for categories x markets. The reason online ratings tend to be bimodal is because the respondents are self selecting sample rather than randomized. Since product ratings are not typically compulsory, the extreme likers/dis-likers tend to fill the survey question. When implemented as compulsory, it drives poor user experience and having to fill a survey to get something anyway introduces a bias (just fill something to get it over with). So no easy answer. This is also the reason traditional survey firms haven't really gone out of business even with the wealth of online data from Facebook, Twitter and product reviews available, though better/easier access to actual behavioral signals are certainly replacing surveys in some areas.
- svachalek 10y agoI don't work in the recommendations side of the house, but the way I understand things is that nowadays your star ratings (and now up/down ratings) are primarily to learn about you, what you liked and didn't like and therefore what you might like to see in the future. They're not really about creating a Rotten Tomatoes or IMDB style database of trying to rank all movies on some sort of objective "regardless of what you like, this is a high quality movie" sort of scale.
- Gracana 10y agoBut there's no difference in price between going to see a silly movie or a"good" movie.
- shiftb 10y agoSome friends of mine used to have a movie rating site which asked two questions: 1) How good was the movie? [1-5] 2) How likely are you to watch it again? [1-5] I think it was too many questions to ask, but trying to separate out the enjoyment from the quality struck me as a good idea.