7 ms·

Didn't ever expect to see this capability. A bit unrelated, but: anyone remember in the "early" days when Netflix held competitions to develop the most accurat

Didn't ever expect to see this capability. A bit unrelated, but: anyone remember in the "early" days when Netflix held competitions to develop the most accurate recommendation and ratings engines? Giving out million dollar prizes? And held possibly state-of-the-art tech for this? And then one day completely gutted the star rating system, replacing it with a much-despised "thumbs up/down" system, giving up their "so good it's magical" recommendations for something that feels less accurate than a coin flip...

Someone1234 7y ago
It was right around the time their first "Originals" came out. I guess accurate recommendations is less important than shows with cheaper licensing costs or home-grown originals.
- giancarlostoro 7y ago
  They coulda just made their originals part of the simpler rating system instead. As jacked up as it is, it would be nice to find shows that I'd more than likely actually like.
- hanniabu 7y ago
  While I'm sure this had a lot to do with it, I wouldn't be surprised if they also came up with a better rating system using statistics on the amount of views, how long it's been out, what percent watched it all the way through, etc.
  - Someone1234 7y ago
    If that were true, wouldn't the actual recommendations be better rather than substantially worse?
    - dwighttk 7y ago
      better for Netflix or better for us?
- ajmurmann 7y ago
  I sometimes think that they make it harder to discover content, so that it's also harder to discover that there is nothing you'll actually like. This will keep you browsing longer and starting some movies you'll stop watching. Shitty customer experience, but as far as I know they optimizer for time you spent with the platform.
scrooched_moose 7y ago
I'm pretty sure the new recommendation algorithm is basically just "recommend Netflix originals in a random order". Edit: I feel like I should add something besides a snarky comment: I was searching for a mid-2000s Best Picture Nominee a few days ago (something like Munich or Capote, but don't remember). Results 1 & 2 were The Irishman and Marriage Story. None of the other non-Netflix best picture nominees were anywhere in the results if that was the connection they were making.
- gdulli 7y ago
  There's a law of conservation of advertisement. It's not like you were going to get away from ads forever when switching from real TV. I prefer the purely interstitial (untargeted) commercials from real TV. With a DVR I can fast forward them, or always at least mute. Having Amazon/Netflix "recommend" things to me through what should be a functional unbiased system is more insidious.
  - briandear 7y ago
    Ads in the middle of the show? No way. It’s literally an interruption.
    - reaperducer 7y ago
      If it's for free OTA television, I'm OK with that. I don't know why people never rose up against it in cable/satellite-only channels that they pay for. /I'm OTA, Netflix, and the library. But I mostly only use Netflix for the DVD service. I'm fortunate enough to have enough going on in my life that I watch very little video.
  - spidermango 7y ago
    lol, the sheer stupidity in the mental gymnastics people will employ to back their outrageous behaviors in avoiding the "targeted ads" bogeyman
    - scarejunba 7y ago
deleted 7y ago
[deleted]
1123581321 7y ago
A lot more people use the thumb system than were using the star system.
- ixwt 7y ago
  They used the equivalent of the thumb system. Many people give 5 stars, or 1 star. There are some that give in between. Many places consider anything less than 5 star to mean there was a problem (Uber, Amazon, reviews for phone support).
  - t-writescode 7y ago
    Do you have statistics on that? I’m not sure I believe you, given Amazon ratings seem to have quite a broad range, so the pattern is established. edit: specifically, that people only give 1 or 5 stars
    - ixwt 7y ago
      No, it's anecdata. A lot of stuff I see tends to have a large amount of 5s and 1s. There is often 2-4s, but the amount is very often less than 5s or 1s. It's mostly just looking at trends. A few things I've purchased from Amazon come with a little advertising slip that says "if this product is anything less than 5 stars, let us know what we can do to make it better!" I've seen plenty of stories from people complaining that their less than 5 star reviews got removed for X reason (often because it's "dishonest"). Maybe that'll be my next learning exercise: scraping the count of reviews from Amazon to support this hypothesis.
      - gknoy 7y ago
        It might be that people are less motivated to take the effort to rate something if it was "meh, okay" -- whereas a 1 or 5 star rating is something you might want others to know about, so they can see/avoid the show.
- what_ever 7y ago
  Source?
taude 7y ago
Yes, I remember those days of the content. But, I don't really agree that the thumbs up/down system is giving me bad recommendations. I still find the 'percentage' that it thinks I'd like is probably really accurate. I have to assume that the recommendations work with more data than just the thumbs up/down interaction, since they probably don't fully trust the user input. And I imagine all the stuff I watched for 10 minutes but stopped is more valuable in determining usefulness than me going out of my way to rate it.
deleted 7y ago
[deleted]
minimaxir 7y ago
Netflix still does a ton of statistical experiments. A good one is video thumbnail selection. 2016 blog post: https://netflixtechblog.com/selecting-the-best-artwork-for-videos-through-a-b-testing-f6155c4595f6 https://netflixtechblog.com/selecting-the-best-artwork-for-v... 2018 presentation: https://www.youtube.com/watch?v=UjQMEjkrUGo https://www.youtube.com/watch?v=UjQMEjkrUGo
- daniel_iversen 7y ago
  But that’s more about making people click right? So while those experiments serve Netflix well, the old recommendations and ratings served both Netflix and users.
  - minimaxir 7y ago
    True; more was addressing the point about Netflix regressing from statistical rigor.
jdamon96 7y ago
what causes you to say much despised? I personally enjoy thumbs up v thumbs down - I find with 5 star ratings systems people resort to extremes and usually use it like a thumbs up/down anyway (1 star or 5 star)
- tele_ski 7y ago
  I'd prefer up/sideways/down. Sometimes I watch things that are brainless, they aren't thumbs up worthy but they are not totally thumbs down either. But if this was hiring a sideways is always down in my book, so I could see why it's not totally useful.
  - scott_s 7y ago
    "Sideways" exists: don't rate it.
    - gvjddbnvdrbv 7y ago
      Null !== 0 Not rating does not mean the same thing as rating something as average.
      - scott_s 7y ago
        Not in a literal sense, no. But I bet their model treats it similarly.
        deleted 7y ago
        [deleted]
mr_toad 7y ago
I only recall them ever giving out one one million dollar prize on Kaggle. As far as I know they never implemented the winning system because it would have been too difficult to engineer the pipeline. I’m not sure whether you’re trying to imply that non-ML based, non-personalised simple average ratings are better or worse than ML. Regardless, I have nothing nice to say about anecdotes.
psychometry 7y ago
They obviously determined that thumbs up/down either generated better training data or more training data. They wouldn't intentionally make their recommendations worse just for the sake of it.
- EpicEng 7y ago
  >They wouldn't intentionally make their recommendations worse just for the sake of it. reply As has been mentioned before, there is at least some motivation for doing so. One, it means their originals don't rate below third party titles (as far as we can tell), and two, as they lose content the average rating of what's left will likely fall.
- cle 7y ago
  > They wouldn't intentionally make their recommendations worse just for the sake of it. Why not? Every big platform that I know of has eventually done this with either 1p or “sponsored” content (Google, Amazon, Apple, etc.). In this case their incentive would be to push their own content above 3p content. What matters to them is their revenue, not maximizing the accuracy of their recommendations model.
  - Dylan16807 7y ago
    That still depends on a good recommendation system, it's just an extra finger on the balance. Giving a boost to first party content is very different from making it "just worse".
    - cle 7y ago
      From a customer's perspective, there's no difference. These are just rationalizations the business tells itself for those who need to reduce their cognitive dissonance.
- DoofusOfDeath 7y ago
  > They obviously determined that thumbs up/down either generated better training data or more training data. I'm guessing that the definition of "better" changed at some point, perhaps when they went from 1-5 stars to +-1 thumbs. In Netflix's early years, I was confident that their classifier's goal was to predict my future rating for a given video. I.e., it was part of a larger system designed around my viewing pleasure. Around the time they switched to thumbs, I started suspecting the classifier was being used to manipulate me towards optimizing Netflix's profitability, instead of my viewing pleasure.
joncp 7y ago
Remember when you didn't have to wade through that awful carousel view to find what you wanted to watch? Or when they didn't try to hide your "continue watching" list in a steaming heap of inaccurate recommendations? Except for a couple of shows, I've almost completely switched to Prime.
- ehsankia 7y ago
  "continue watching" being deep down is my biggest pet peeve. Nothing gets me angrier than having to scroll and search just to continue watching the same show every single day. Statistically, if each show is 8 episodes, then 88% of the time when I come to the homepage, I want to "continue watching" not "find a new show". Why the hell is continue watching not always the very first row.
  - forrestbrazeal 7y ago
    They deliberately switch around the placement of the homepage queues. The goal is to keep you engaging with the product and discovering new things. They don't want you holding a Netflix subscription just for Friends, or whatever.
    - wvenable 7y ago
      But why? Isn't watching Friends a viable reason to own that subscription?
      - Lammy 7y ago
        You're more likely to cancel when you're done watching that one show. They want to continuously give you a new reason to keep paying.
        DoofusOfDeath 7y ago
        Their awful UI was one of the reasons I dropped the service. I wonder what fraction of former customers are like me.
paxys 7y ago
1. They didn't want their originals to be rated below third-party titles 2. They didn't want users to see that highly rated content was slowly disappearing from the service
pgrote 7y ago
The supposedly killed it when the Amy Schumer special was bombed. https://www.whas11.com/article/news/entertainment-news/poof-netflix-has-deleted-all-user-reviews-from-its-website-citing-declining-use/417-585567573 https://www.whas11.com/article/news/entertainment-news/poof-...
- hanniabu 7y ago
  They really messed up by not giving that to Andrew Schulz
- AviationAtom 7y ago
  Was going to post this, but you beat me to it. It was neutered very quickly after her special was down-voted into oblivion, not long after it was gutted.
Carpetsmoker 7y ago
I don't really like star systems because I spend way too much time deciding if something is 3 or 4 stars, or 4 or 5 stars. I much prefer textual descriptions; for example "Terrible", "Don't like", "Okay", "Like", "Favourite", which has the same options as a 5-star rating system, but choosing between 3/4 and 4/5 is much easier IMHO. I don't really know of any system that uses this, except this one music player I wrote myself (which has "Crap", "Meh", "Okay", "Super").
- ehsankia 7y ago
  Yep, as much as people hate it, when it comes to recommendation, up/down is all the data you need. Trying to build a system out of a 5 star system just adds unnecessary complexity, and the reality was that people used the star system differently making it even harder. up/down thumb is explicit and cleaner to work with.
  - glitcher 7y ago
    The problem with the up/down system for me is not my own ability to like/dislike specific titles, but more the fact that Netflix no longer displays the average of all user votes. Sure different people used the 5 star system in different ways, and there were some who may have misused it by giving poor ratings to things they never intended on watching, but it was a great signal to me for the extremes. Scenario: I'm considering some odd looking sci-fi movie to watch that I never heard of before. Ratings between 2-4 stars might not tell me much, but very reliably titles with only one star were terrible movies. Now Netflix happily recommends any and all sci-fi titles, saying they are a "98% match" for me! Sure by category, but when the movie is a low budget dumpster fire I no longer have that instant signaling that the previous rating system gave me.
    - rpdillon 7y ago
      IIRC, Netflix never showed the average, but rather the rating they predicted you would give it, taking into account your previous viewing and rating.
  - 7y ago
ggggtez 7y ago
Yes, it was discovered that when you had to order movies in the mail, you had different preferences. If you could only watch one movie at a time, sure you want to make it good. But when it's streaming, people mostly just want to binge watch garbage. Who needs million dollar prizes when you can do just fine recommending 50 different shows, until the user gives up finding anything good and watches a Netflix original.
ajkjk 7y ago
I suspect there was a really good justification for this that they can't directly share with us. These sort of decisions tend to be like that. Probably getting rid of recommendations led to some massive increase in viewership that couldn't be ignored.
- ehsankia 7y ago
  One person probably rated a movie they hated 1 star, whereas someone else rated it 3 star or 4 star. The inconsistency makes it hard to build a model that works for everyone. Also, the current one technically has 3 state, down, up and no vote. That's much more explicit and easy to build a model around.
philipkglass 7y ago
The Netflix Prize competition launched in 2006, before they had started streaming anything. The problem was to recommend a huge DVD-by-mail catalog to millions of users. By the time the competition finished in 2009, streaming was already growing rapidly. Nowadays their DVD business is a small footnote to their streaming business. The streaming catalog is much smaller than the DVD catalog and the consequences for starting to stream a show/movie you don't like are much smaller; you can back out and immediately start something else. You don't have to wait days to return your unsatisfying DVD and get a different one in the mail. Netflix has ~6,000 items to recommend to streaming customers in the US [1]. This is about 6% as much content to recommend as the DVD-by-mail service has. Their streaming catalog size is actually shrinking over time [2]. Fine grained rating systems and better recommenders aren't going to move the needle much for Netflix's current business, because they now have a content availability problem much more than a discovery problem. [1] https://www.statista.com/statistics/1013571/netflix-library-size-worldwide/ https://www.statista.com/statistics/1013571/netflix-library-... [2] https://www.streamingobserver.com/netflix-movie-library-shrinking/ https://www.streamingobserver.com/netflix-movie-library-shri...
- pjc50 7y ago
  I'm sure there was a narrow window when you could stream seemingly every film, ancient and modern, legitimately. Then the studios twigged and had to kill it.
  - philipkglass 7y ago
    I don't even mind paying on demand to stream a film. It's annoying and economically baffling when you can't stream a film even if you have your credit card in hand, ready to pay. Last weekend I tried to find To Live and Die in L.A. -- from the 1980s but hardly obscure. I couldn't find it legitimately available to stream anywhere, for any amount of money. I even tried Kanopy, which has been my go-to source for films I can't find on paid streaming services, but they didn't have it either. https://en.wikipedia.org/wiki/To_Live_and_Die_in_L.A._(film) https://en.wikipedia.org/wiki/To_Live_and_Die_in_L.A._(film)
elamje 7y ago
A small detail many forget is that Netflix pays very different prices for different shows. I don’t know the exact negotiated rates, but basically you can envision a blockbuster is going to cost more per stream then a Netflix Original. Eventually Netflix was/is bound to game their own recommendations since it saves them costs. The only case I can see where this incentive doesn’t exist is for shows that Netflix pays a fixed price to stream rather than a unit price per stream.
- bostonfincs 7y ago
  I believe most if not all shows/movies on Netflix are acquired through a general use license for a period of time rather than a “royalty scheme” based on number of views. So while they definitely want to promote original content as in the long run it saves them money (in license acquisition costs) the per stream cost is relatively homogenous.
- matheusmoreira 7y ago
  > A small detail many forget is that Netflix pays very different prices for different shows. Yeah, it's extremely annoying. Netflix essentially mocks its customers by keeping the best works just out of their reach. Almost every time I search for a classic film, I find its sequels instead. The film I actually wanted to watch was apparently too good for Netflix's pay grade. Netflix has Terminator 1 and 3 but not 2; the first Mad Max but none of the others; Spider-Man 1 and 3 but not 2; several Hannibal movies and a TV series but not Silence of the Lambs; some Star Wars movies but not others; the list goes on...
jfengel 7y ago
Netflix now has much better sources of information than the self-reported star ratings. Self-reported ratings fail in a lot of ways. Users can upvote what they think they're supposed to upvote. They'll downvote something, then go watch the sequel, because "It's terrible but I love it". And since they're seeking out stuff that they suspect they're going to like, they either compress all of the ratings into the top tier, or they end up "hating" things that they actually thought were merely so-so. I suspect that even the up-down ratings get little attention in their recommendation engine. They have far more information, especially in the streaming service. Did you watch all of it? Did you watch it all at once? Did you watch it more than once? Did you watch it immediately after discovering it? That's all stuff that users can't fake or be confused about. You don't get users saying, "Citizen Kane is the greatest film of all time, but I really don't want to see it." It's more likely that you can tease out what it is that a user actually wants to see, rather than what they tell you they want. Netflix did run a competition back in the DVD days, when they had less interaction with the user. They did get a slight improvement, but at about the same time, they introduced streaming, and a whole lot of Big Data techniques appeared that could take advantage of that new data. So it wouldn't surprise me if the up/down buttons were completely ignored by the recommendation engine. Maybe it's just a reminder to you: "Oh, yeah, I did see that. It sucked." The new system clearly misses some things, but a lot of the problem is that the DVD catalog is immense and full of classics, but the streaming catalog is weak and full of crap. The whole thing may be back-ported into the DVD recommendation engine, but that's the redheaded stepchild at Netflix these days. They tried to spin it off entirely.
- reaperducer 7y ago
  All current user ratings systems for movies are terrible because new users can bury a great movie. There are classic, important, society-changing movies from the previous century that were highly praised by contemporary critics and won scads of awards. But then people today watch them with zero context or understanding of what was happening or what the world was like and give them bad ratings because they can only compare them to the latest shoot-em-up sequel of a sequel of a sequel. It's one of the reasons Rotten Tomatoes is worthless for anything more than 20 years old, and why the voices of movie critics are needed so badly today. See also: Yelp.
loudandskittish 7y ago
I really don't get what's up with that...2012 or so, Netflix was suggesting movies I'd never heard of that I ended up loving. Now when I log in, it just demands I watch Family Guy (ugh) ...and starts auto-playing it, of course.
low_key 7y ago
The algorithm requirements changed. It used to be to recommend the "best" content for you and now it is to recommend the cheapest content for them.
kgwxd 7y ago
They redefined their use of the word "recommended" from "we think you'll like this" to "we want you to watch this". That's what happens to every in-house rating system eventually, independent rating systems are better.
petee 7y ago
Their catalog has shrunk so much there is no need for a recommendation system; their classics category only has 42 movies in it.
beepboopbeep 7y ago
Star systems are largely trash anyways. What is 2 stars vs 3 or 4? but we know 1 Is bad, and that 5 is great. If anything a 3 star system is better, 1 - bad, 2 - neutral, 3 - good. Simple. Often the 5 star system just devolves into 1, 4, or 5 anyways. Or in the case of averaged ratings like your ride share apps, you get stuck as a floating point. Suddenly 4.3 is significant vs 4.8. It's all silly. I fully support the thumbs up or down. It's effectively a 3 star system anyways. 1 - Thumbs down, 2- neutral enough to not warrant a response, 3 - Thumbs up
fouc 7y ago
I feel like netflix will never solve the recommendations system in favor of the viewer. We need a 3rd party service or some sort of overlay in order to improve search results. I feel like netflix tends to recommend a lot of bottom of the barrel movies along with movies that I've already seen, there's no fine grained control.
- hombre_fatal 7y ago
  I mean, there are third-party review websites where you plug in shows you've watched and it recommends new ones, even showing you on which services they're available. I just doubt people actually care so much that they use them. It's a minor nice-to-have feature. But the law of triviality will suggest that everyone will argue about it every time Netflix comes up, and we do, even though I doubt we really care. I don't think it's any more useful than searching "best $genre $year" when you're looking for new things to watch.
neop1x 7y ago
Since I started using MovieLens.org and let it learn what I like, I get pretty decent recommendations. I am also adding tags after I watch the movie. Recommendations has to be highly personal. Thumbs up/down is not enough. And global rankings are totally useless for me. I rarely like top-rated movies. I hope no corporation buys them and rewrites/destroys the system as it is common with all acquisitions because the way it works now is perfect. They should open-source it to preserve it and help the society. I pay for Netflix but I rarely use it due to DRM (a bit complicated in Kodi) and lack of movies (too many shows).
guest__user 7y ago
i (naively?) thought that the thumbs up/down was coupled with info about what you watched, what you clicked on, what trailer you watched etc. is this not the case?

Someone1234 7y ago

It was right around the time their first "Originals" came out. I guess accurate recommendations is less important than shows with cheaper licensing costs or home-grown originals.

giancarlostoro 7y ago
They coulda just made their originals part of the simpler rating system instead. As jacked up as it is, it would be nice to find shows that I'd more than likely actually like.
hanniabu 7y ago
While I'm sure this had a lot to do with it, I wouldn't be surprised if they also came up with a better rating system using statistics on the amount of views, how long it's been out, what percent watched it all the way through, etc.
- Someone1234 7y ago
  If that were true, wouldn't the actual recommendations be better rather than substantially worse?
  - dwighttk 7y ago
    better for Netflix or better for us?
ajmurmann 7y ago
I sometimes think that they make it harder to discover content, so that it's also harder to discover that there is nothing you'll actually like. This will keep you browsing longer and starting some movies you'll stop watching. Shitty customer experience, but as far as I know they optimizer for time you spent with the platform.

scrooched_moose 7y ago

I'm pretty sure the new recommendation algorithm is basically just "recommend Netflix originals in a random order". Edit: I feel like I should add something besides a snarky comment: I was searching for a mid-2000s Best Picture Nominee a few days ago (something like Munich or Capote, but don't remember). Results 1 & 2 were The Irishman and Marriage Story. None of the other non-Netflix best picture nominees were anywhere in the results if that was the connection they were making.

gdulli 7y ago
There's a law of conservation of advertisement. It's not like you were going to get away from ads forever when switching from real TV. I prefer the purely interstitial (untargeted) commercials from real TV. With a DVR I can fast forward them, or always at least mute. Having Amazon/Netflix "recommend" things to me through what should be a functional unbiased system is more insidious.
- briandear 7y ago
  Ads in the middle of the show? No way. It’s literally an interruption.
  - reaperducer 7y ago
    If it's for free OTA television, I'm OK with that. I don't know why people never rose up against it in cable/satellite-only channels that they pay for. /I'm OTA, Netflix, and the library. But I mostly only use Netflix for the DVD service. I'm fortunate enough to have enough going on in my life that I watch very little video.
- spidermango 7y ago
  lol, the sheer stupidity in the mental gymnastics people will employ to back their outrageous behaviors in avoiding the "targeted ads" bogeyman
  - scarejunba 7y ago

deleted 7y ago

[deleted]

1123581321 7y ago

A lot more people use the thumb system than were using the star system.

ixwt 7y ago
They used the equivalent of the thumb system. Many people give 5 stars, or 1 star. There are some that give in between. Many places consider anything less than 5 star to mean there was a problem (Uber, Amazon, reviews for phone support).
- t-writescode 7y ago
  Do you have statistics on that? I’m not sure I believe you, given Amazon ratings seem to have quite a broad range, so the pattern is established. edit: specifically, that people only give 1 or 5 stars
  - ixwt 7y ago
    No, it's anecdata. A lot of stuff I see tends to have a large amount of 5s and 1s. There is often 2-4s, but the amount is very often less than 5s or 1s. It's mostly just looking at trends. A few things I've purchased from Amazon come with a little advertising slip that says "if this product is anything less than 5 stars, let us know what we can do to make it better!" I've seen plenty of stories from people complaining that their less than 5 star reviews got removed for X reason (often because it's "dishonest"). Maybe that'll be my next learning exercise: scraping the count of reviews from Amazon to support this hypothesis.
    - gknoy 7y ago
      It might be that people are less motivated to take the effort to rate something if it was "meh, okay" -- whereas a 1 or 5 star rating is something you might want others to know about, so they can see/avoid the show.
what_ever 7y ago
Source?

taude 7y ago

Yes, I remember those days of the content. But, I don't really agree that the thumbs up/down system is giving me bad recommendations. I still find the 'percentage' that it thinks I'd like is probably really accurate. I have to assume that the recommendations work with more data than just the thumbs up/down interaction, since they probably don't fully trust the user input. And I imagine all the stuff I watched for 10 minutes but stopped is more valuable in determining usefulness than me going out of my way to rate it.

deleted 7y ago

[deleted]

minimaxir 7y ago

Netflix still does a ton of statistical experiments. A good one is video thumbnail selection. 2016 blog post: https://netflixtechblog.com/selecting-the-best-artwork-for-videos-through-a-b-testing-f6155c4595f6 https://netflixtechblog.com/selecting-the-best-artwork-for-v... 2018 presentation: https://www.youtube.com/watch?v=UjQMEjkrUGo https://www.youtube.com/watch?v=UjQMEjkrUGo

daniel_iversen 7y ago
But that’s more about making people click right? So while those experiments serve Netflix well, the old recommendations and ratings served both Netflix and users.
- minimaxir 7y ago
  True; more was addressing the point about Netflix regressing from statistical rigor.

jdamon96 7y ago

what causes you to say much despised? I personally enjoy thumbs up v thumbs down - I find with 5 star ratings systems people resort to extremes and usually use it like a thumbs up/down anyway (1 star or 5 star)

tele_ski 7y ago
I'd prefer up/sideways/down. Sometimes I watch things that are brainless, they aren't thumbs up worthy but they are not totally thumbs down either. But if this was hiring a sideways is always down in my book, so I could see why it's not totally useful.
- scott_s 7y ago
  "Sideways" exists: don't rate it.
  - gvjddbnvdrbv 7y ago
    Null !== 0 Not rating does not mean the same thing as rating something as average.
    - scott_s 7y ago
      Not in a literal sense, no. But I bet their model treats it similarly.
      - deleted 7y ago
        [deleted]

mr_toad 7y ago

I only recall them ever giving out one one million dollar prize on Kaggle. As far as I know they never implemented the winning system because it would have been too difficult to engineer the pipeline. I’m not sure whether you’re trying to imply that non-ML based, non-personalised simple average ratings are better or worse than ML. Regardless, I have nothing nice to say about anecdotes.

psychometry 7y ago

They obviously determined that thumbs up/down either generated better training data or more training data. They wouldn't intentionally make their recommendations worse just for the sake of it.

EpicEng 7y ago
>They wouldn't intentionally make their recommendations worse just for the sake of it. reply As has been mentioned before, there is at least some motivation for doing so. One, it means their originals don't rate below third party titles (as far as we can tell), and two, as they lose content the average rating of what's left will likely fall.
cle 7y ago
> They wouldn't intentionally make their recommendations worse just for the sake of it. Why not? Every big platform that I know of has eventually done this with either 1p or “sponsored” content (Google, Amazon, Apple, etc.). In this case their incentive would be to push their own content above 3p content. What matters to them is their revenue, not maximizing the accuracy of their recommendations model.
- Dylan16807 7y ago
  That still depends on a good recommendation system, it's just an extra finger on the balance. Giving a boost to first party content is very different from making it "just worse".
  - cle 7y ago
    From a customer's perspective, there's no difference. These are just rationalizations the business tells itself for those who need to reduce their cognitive dissonance.
DoofusOfDeath 7y ago
> They obviously determined that thumbs up/down either generated better training data or more training data. I'm guessing that the definition of "better" changed at some point, perhaps when they went from 1-5 stars to +-1 thumbs. In Netflix's early years, I was confident that their classifier's goal was to predict my future rating for a given video. I.e., it was part of a larger system designed around my viewing pleasure. Around the time they switched to thumbs, I started suspecting the classifier was being used to manipulate me towards optimizing Netflix's profitability, instead of my viewing pleasure.

joncp 7y ago

Remember when you didn't have to wade through that awful carousel view to find what you wanted to watch? Or when they didn't try to hide your "continue watching" list in a steaming heap of inaccurate recommendations? Except for a couple of shows, I've almost completely switched to Prime.

ehsankia 7y ago
"continue watching" being deep down is my biggest pet peeve. Nothing gets me angrier than having to scroll and search just to continue watching the same show every single day. Statistically, if each show is 8 episodes, then 88% of the time when I come to the homepage, I want to "continue watching" not "find a new show". Why the hell is continue watching not always the very first row.
- forrestbrazeal 7y ago
  They deliberately switch around the placement of the homepage queues. The goal is to keep you engaging with the product and discovering new things. They don't want you holding a Netflix subscription just for Friends, or whatever.
  - wvenable 7y ago
    But why? Isn't watching Friends a viable reason to own that subscription?
    - Lammy 7y ago
      You're more likely to cancel when you're done watching that one show. They want to continuously give you a new reason to keep paying.
      - DoofusOfDeath 7y ago
        Their awful UI was one of the reasons I dropped the service. I wonder what fraction of former customers are like me.

paxys 7y ago

1. They didn't want their originals to be rated below third-party titles 2. They didn't want users to see that highly rated content was slowly disappearing from the service

pgrote 7y ago

The supposedly killed it when the Amy Schumer special was bombed. https://www.whas11.com/article/news/entertainment-news/poof-netflix-has-deleted-all-user-reviews-from-its-website-citing-declining-use/417-585567573 https://www.whas11.com/article/news/entertainment-news/poof-...

hanniabu 7y ago
They really messed up by not giving that to Andrew Schulz
AviationAtom 7y ago
Was going to post this, but you beat me to it. It was neutered very quickly after her special was down-voted into oblivion, not long after it was gutted.

Carpetsmoker 7y ago

I don't really like star systems because I spend way too much time deciding if something is 3 or 4 stars, or 4 or 5 stars. I much prefer textual descriptions; for example "Terrible", "Don't like", "Okay", "Like", "Favourite", which has the same options as a 5-star rating system, but choosing between 3/4 and 4/5 is much easier IMHO. I don't really know of any system that uses this, except this one music player I wrote myself (which has "Crap", "Meh", "Okay", "Super").

ehsankia 7y ago
Yep, as much as people hate it, when it comes to recommendation, up/down is all the data you need. Trying to build a system out of a 5 star system just adds unnecessary complexity, and the reality was that people used the star system differently making it even harder. up/down thumb is explicit and cleaner to work with.
- glitcher 7y ago
  The problem with the up/down system for me is not my own ability to like/dislike specific titles, but more the fact that Netflix no longer displays the average of all user votes. Sure different people used the 5 star system in different ways, and there were some who may have misused it by giving poor ratings to things they never intended on watching, but it was a great signal to me for the extremes. Scenario: I'm considering some odd looking sci-fi movie to watch that I never heard of before. Ratings between 2-4 stars might not tell me much, but very reliably titles with only one star were terrible movies. Now Netflix happily recommends any and all sci-fi titles, saying they are a "98% match" for me! Sure by category, but when the movie is a low budget dumpster fire I no longer have that instant signaling that the previous rating system gave me.
  - rpdillon 7y ago
    IIRC, Netflix never showed the average, but rather the rating they predicted you would give it, taking into account your previous viewing and rating.
- 7y ago

ggggtez 7y ago

Yes, it was discovered that when you had to order movies in the mail, you had different preferences. If you could only watch one movie at a time, sure you want to make it good. But when it's streaming, people mostly just want to binge watch garbage. Who needs million dollar prizes when you can do just fine recommending 50 different shows, until the user gives up finding anything good and watches a Netflix original.

ajkjk 7y ago

I suspect there was a really good justification for this that they can't directly share with us. These sort of decisions tend to be like that. Probably getting rid of recommendations led to some massive increase in viewership that couldn't be ignored.

ehsankia 7y ago
One person probably rated a movie they hated 1 star, whereas someone else rated it 3 star or 4 star. The inconsistency makes it hard to build a model that works for everyone. Also, the current one technically has 3 state, down, up and no vote. That's much more explicit and easy to build a model around.

philipkglass 7y ago

The Netflix Prize competition launched in 2006, before they had started streaming anything. The problem was to recommend a huge DVD-by-mail catalog to millions of users. By the time the competition finished in 2009, streaming was already growing rapidly. Nowadays their DVD business is a small footnote to their streaming business. The streaming catalog is much smaller than the DVD catalog and the consequences for starting to stream a show/movie you don't like are much smaller; you can back out and immediately start something else. You don't have to wait days to return your unsatisfying DVD and get a different one in the mail. Netflix has ~6,000 items to recommend to streaming customers in the US [1]. This is about 6% as much content to recommend as the DVD-by-mail service has. Their streaming catalog size is actually shrinking over time [2]. Fine grained rating systems and better recommenders aren't going to move the needle much for Netflix's current business, because they now have a content availability problem much more than a discovery problem. [1] https://www.statista.com/statistics/1013571/netflix-library-size-worldwide/ https://www.statista.com/statistics/1013571/netflix-library-... [2] https://www.streamingobserver.com/netflix-movie-library-shrinking/ https://www.streamingobserver.com/netflix-movie-library-shri...

pjc50 7y ago
I'm sure there was a narrow window when you could stream seemingly every film, ancient and modern, legitimately. Then the studios twigged and had to kill it.
- philipkglass 7y ago
  I don't even mind paying on demand to stream a film. It's annoying and economically baffling when you can't stream a film even if you have your credit card in hand, ready to pay. Last weekend I tried to find To Live and Die in L.A. -- from the 1980s but hardly obscure. I couldn't find it legitimately available to stream anywhere, for any amount of money. I even tried Kanopy, which has been my go-to source for films I can't find on paid streaming services, but they didn't have it either. https://en.wikipedia.org/wiki/To_Live_and_Die_in_L.A._(film) https://en.wikipedia.org/wiki/To_Live_and_Die_in_L.A._(film)

elamje 7y ago

A small detail many forget is that Netflix pays very different prices for different shows. I don’t know the exact negotiated rates, but basically you can envision a blockbuster is going to cost more per stream then a Netflix Original. Eventually Netflix was/is bound to game their own recommendations since it saves them costs. The only case I can see where this incentive doesn’t exist is for shows that Netflix pays a fixed price to stream rather than a unit price per stream.

bostonfincs 7y ago
I believe most if not all shows/movies on Netflix are acquired through a general use license for a period of time rather than a “royalty scheme” based on number of views. So while they definitely want to promote original content as in the long run it saves them money (in license acquisition costs) the per stream cost is relatively homogenous.
matheusmoreira 7y ago
> A small detail many forget is that Netflix pays very different prices for different shows. Yeah, it's extremely annoying. Netflix essentially mocks its customers by keeping the best works just out of their reach. Almost every time I search for a classic film, I find its sequels instead. The film I actually wanted to watch was apparently too good for Netflix's pay grade. Netflix has Terminator 1 and 3 but not 2; the first Mad Max but none of the others; Spider-Man 1 and 3 but not 2; several Hannibal movies and a TV series but not Silence of the Lambs; some Star Wars movies but not others; the list goes on...

jfengel 7y ago

Netflix now has much better sources of information than the self-reported star ratings. Self-reported ratings fail in a lot of ways. Users can upvote what they think they're supposed to upvote. They'll downvote something, then go watch the sequel, because "It's terrible but I love it". And since they're seeking out stuff that they suspect they're going to like, they either compress all of the ratings into the top tier, or they end up "hating" things that they actually thought were merely so-so. I suspect that even the up-down ratings get little attention in their recommendation engine. They have far more information, especially in the streaming service. Did you watch all of it? Did you watch it all at once? Did you watch it more than once? Did you watch it immediately after discovering it? That's all stuff that users can't fake or be confused about. You don't get users saying, "Citizen Kane is the greatest film of all time, but I really don't want to see it." It's more likely that you can tease out what it is that a user actually wants to see, rather than what they tell you they want. Netflix did run a competition back in the DVD days, when they had less interaction with the user. They did get a slight improvement, but at about the same time, they introduced streaming, and a whole lot of Big Data techniques appeared that could take advantage of that new data. So it wouldn't surprise me if the up/down buttons were completely ignored by the recommendation engine. Maybe it's just a reminder to you: "Oh, yeah, I did see that. It sucked." The new system clearly misses some things, but a lot of the problem is that the DVD catalog is immense and full of classics, but the streaming catalog is weak and full of crap. The whole thing may be back-ported into the DVD recommendation engine, but that's the redheaded stepchild at Netflix these days. They tried to spin it off entirely.

reaperducer 7y ago
All current user ratings systems for movies are terrible because new users can bury a great movie. There are classic, important, society-changing movies from the previous century that were highly praised by contemporary critics and won scads of awards. But then people today watch them with zero context or understanding of what was happening or what the world was like and give them bad ratings because they can only compare them to the latest shoot-em-up sequel of a sequel of a sequel. It's one of the reasons Rotten Tomatoes is worthless for anything more than 20 years old, and why the voices of movie critics are needed so badly today. See also: Yelp.

loudandskittish 7y ago

I really don't get what's up with that...2012 or so, Netflix was suggesting movies I'd never heard of that I ended up loving. Now when I log in, it just demands I watch Family Guy (ugh) ...and starts auto-playing it, of course.

low_key 7y ago

The algorithm requirements changed. It used to be to recommend the "best" content for you and now it is to recommend the cheapest content for them.

kgwxd 7y ago

They redefined their use of the word "recommended" from "we think you'll like this" to "we want you to watch this". That's what happens to every in-house rating system eventually, independent rating systems are better.

petee 7y ago

Their catalog has shrunk so much there is no need for a recommendation system; their classics category only has 42 movies in it.

beepboopbeep 7y ago

Star systems are largely trash anyways. What is 2 stars vs 3 or 4? but we know 1 Is bad, and that 5 is great. If anything a 3 star system is better, 1 - bad, 2 - neutral, 3 - good. Simple. Often the 5 star system just devolves into 1, 4, or 5 anyways. Or in the case of averaged ratings like your ride share apps, you get stuck as a floating point. Suddenly 4.3 is significant vs 4.8. It's all silly. I fully support the thumbs up or down. It's effectively a 3 star system anyways. 1 - Thumbs down, 2- neutral enough to not warrant a response, 3 - Thumbs up

fouc 7y ago

I feel like netflix will never solve the recommendations system in favor of the viewer. We need a 3rd party service or some sort of overlay in order to improve search results. I feel like netflix tends to recommend a lot of bottom of the barrel movies along with movies that I've already seen, there's no fine grained control.

hombre_fatal 7y ago
I mean, there are third-party review websites where you plug in shows you've watched and it recommends new ones, even showing you on which services they're available. I just doubt people actually care so much that they use them. It's a minor nice-to-have feature. But the law of triviality will suggest that everyone will argue about it every time Netflix comes up, and we do, even though I doubt we really care. I don't think it's any more useful than searching "best $genre $year" when you're looking for new things to watch.

neop1x 7y ago

Since I started using MovieLens.org and let it learn what I like, I get pretty decent recommendations. I am also adding tags after I watch the movie. Recommendations has to be highly personal. Thumbs up/down is not enough. And global rankings are totally useless for me. I rarely like top-rated movies. I hope no corporation buys them and rewrites/destroys the system as it is common with all acquisitions because the way it works now is perfect. They should open-source it to preserve it and help the society. I pay for Netflix but I rarely use it due to DRM (a bit complicated in Kodi) and lack of movies (too many shows).

guest__user 7y ago

i (naively?) thought that the thumbs up/down was coupled with info about what you watched, what you clicked on, what trailer you watched etc. is this not the case?