6 ms·

Netflix: Lessons in Experimentation

watwut 5y ago
The Netflix recommendations for me are quite bad. I literally aost always end up watching YouTube. It makes it is super difficult to discover movies, even if you have pretty good idea about what kind of movie you want.
travisjungroth 5y ago
We’re hiring to do more of what’s in the article: https://jobs.netflix.com/search?q=Experimentation https://jobs.netflix.com/search?q=Experimentation
- weq 5y ago
  Yes, someone with some basic recommendation engine knowledge please apply! A basic rip-off website streaming site has a better recommendation engine AND ux then this company. Was given a subscruption as a present and it dumped it in the bin a few weeks later after. It seems like the algorithm is a basic highlight whatever scam show is pumped by the bingers at the time. Not even the show i last watched (consitantly everyday) is easy to navigate back to if u close the app. I swapped back to the streaming sites and found more NetFlix content immedataly. Thx for observables though!
  - rawoke083600 5y ago
    Yup like https://www.wepickflix.com https://www.wepickflix.com :p - I tried to implemented a "group recommendation" - since watching "tv/shows/movies" is many times a "group activity".
- deleted 5y ago
  [deleted]
- dmitriid 5y ago
  Do create a button called "wipe my recommendations and start anew"
  - tedsanders 5y ago
    There already exists button for wiping your recommendations and starting anew: It's labeled 'add profile'. :) (Otherwise, you can scroll through you watch history clicking delete a zillion times.)
    - merlincorey 5y ago
      I assume you probably mean well and are trying to help the GP, but this comes off as a totally tone-deaf PR response. The GP doesn't want to create a new profile, but they still want to avoid "clicking delete a zillion times" in their watch history. In my opinion you should acknowledge that the user has an actual problem that your team could solve by providing them that button THEN offer the workaround of creating a new profile with apologies instead of a potentially snarky smile.
      - tedsanders 5y ago
        I don't work at Netflix. I'm not Netflix PR. The smile was meant in good spirit to acknowledge the hackiness of the solution. I don't see why this solution doesn't solve the problem. Creating a new profile is effectively the same as resetting your profile. It absolutely is a solution. Same thing with Facebook or any other service. If you want to "reset" your Facebook profile, just create a new one.
        travisjungroth 5y ago
        Maybe they mixed up your username with mine? And if this is the Ted Sanders I’m thinking of, good place to mention that you did previously work at Netflix. Offering help with a hacky solution and smiley face would be consistent with the Ted I met.
        5y ago
laxatives 5y ago
I would be a lot more interested in a technical article than hearing about the history of Netflix.
- rwilson4 5y ago
  You might be interested in Trustworthy Online Experiments by Kohavi et al then. I enjoyed the historical progression!
- travisjungroth 5y ago
  We've recently done a large blog series on experimentation. https://netflixtechblog.com/netflix-a-culture-of-learning-394bc7d0f94c https://netflixtechblog.com/netflix-a-culture-of-learning-39... Kinda covers everything and links out to posts that cover the software engineering in more detail and papers that cover the math in more detail.
  - gordon_freeman 5y ago
    the blog posts series on rec system is interesting. But when I watch Netflix I still run into simple issues. can you please take care of some basics first, such as if I press 'thumbs down' then Netflix would never recommend that title again or similar to such titles.
    - dmitriid 5y ago
      They can't do that. Netflix now has very little content since all the big players have pulled out and started their own streaming services. Netflix has to recommend what it has: old movies, subpar TV shows, things you don't like, Netflix-produced content.
smartbit 5y ago
If their experiment are zo good, why are the recommendations so bad that I had to turn off the feature that adds items to My List? Several series I’ve enjoyed watching several years ago now appear on Netflix. Too late for me on Netflix, but as I liked them I rate them with thumbs up, hoping that Netflix would give me better recommendations. Instead of recommending other movies or series, to my dismay Netflix starts recommending the series I’ve already watched elsewhere. As I can only rate a series I’ve watched, why on earth does Netflix recommend something I’ve already watched? The diversification of Netflix is great, but most of it is too little or too late for my taste. Still better than Disney or Apple TV+ where I could find 1 or 2 shows of my liking. Looking forward to a couple of months of HBO Max once it is available in my region. HBO Europe made some award winning documentaries and I’m hoping to find more that I haven’t watched. Luckily for me there is an alternative for the streaming platform as for the past several decades every week my local TV stations have 1 or 2 great movies and maybe a new serie, selected not by experiments but by real people of flesh and blod.
- ajmurmann 5y ago
  If you want something selected and curated by actual humans, I recommend checking out Mubi. Lots of odd ball, but phenomenal movies on there. They started out having thirty movies on there and replacing the oldest every day with a new movie. It's often times a few in a row by a specific director or around a theme. They now also a have decent library in addition. Almost every movie I find truly memorable in they just few years came to me via Mubi.
- onlyrealcuzzo 5y ago
  Recommendations aren't really chosen by "experiments". And certainly, they aren't A/B testing budgeting and development of shows. Recommendation engines can be improved with experiments. And they - along with budgeting and show development (at least at Netflix) - are "driven by big data". Further, it doesn't matter if you - personally - like a feature. If that feature is moving the numbers and passes the A/B test, it will make itself into the app. They run thousands of experiments. Of course there are going to be several that you - personally - do not like.
- dewey 5y ago
deathanatos 5y ago
> Netflix abandoned the ‘large catalog’ strategy for a ‘right for you, personalized catalog’ strategy. The "right for [me]" catalog seems to contain a lot of "get the viewer hooked on the first two seasons of a show; don't catalog the latter seasons".
jph 5y ago
Netflix experimentation seems to have suffered in the past year IMHO, with worsening recommendations and no simple way for users to provide feedback. And bugs are creeping in-- this week I've hit two different bugs in the Netflix recommendation engine, so something's amiss. The experiment I would love to see is an simple icon that means "I don't want to watch this".
- john___matrix 5y ago
  It's been pretty apparent for me that their who focus changed a couple of years ago from "here's some stuff you might enjoy watching" to "here's the stuff we want you to watch" or much like Google ads, "here's something you've already watched, watch it again" I'm almost at the point of cancelling my Netflix subscription simply because it's so much effort to find something of interest now.
- tedsanders 5y ago
  There already exists a simple icon for "I don't want to watch this": the thumbs down button. Clicking it is supposed to remove the show from your recommendations (after a day or so, to clear the caches).
  - jph 5y ago
    It turns out that Netflix thumbs down conflicts with thumbs up. For example, pick a movie that you like, yet don't ever want to watch on Netflix; perhaps you've already seen the movie on a different service. Do you click thumbs up ("I like this") or thumbs down ("Not for me")? Next, if you watch some of the movie, then it shows up in your "Currently Watching" list, with icons for thumbs up ("I like this but don't want to continue watching"), thumbs down ("I don't like this"), and a broom ("Just cleaning up"). What I am suggesting as an experiment is a choice that means "I like this but don't want it on my main page".
- Consultant32452 5y ago
  For the love of God, can we please get an "Only show me things in my native language" option?
birdyrooster 5y ago
The only reason Netflix is in FAANG is because without them, people would be saying a very naught word.
- thaumasiotes 5y ago
  The reason they're there is that they are known primarily for the high salaries they pay.
anarazel 5y ago
I find the recommendations rarely useful. But what really bugs me is that the same series / movie typically shows up in all kinds of lists. Over and over and over. Besides the categories - often filled with stuff not matching the label - I see the same stuff repeatedly in "trending now", "top 10 today", "only on Netflix", "popular on Netflix".

The Netflix recommendations for me are quite bad. I literally aost always end up watching YouTube. It makes it is super difficult to discover movies, even if you have pretty good idea about what kind of movie you want.

travisjungroth 5y ago

We’re hiring to do more of what’s in the article: https://jobs.netflix.com/search?q=Experimentation https://jobs.netflix.com/search?q=Experimentation

weq 5y ago
Yes, someone with some basic recommendation engine knowledge please apply! A basic rip-off website streaming site has a better recommendation engine AND ux then this company. Was given a subscruption as a present and it dumped it in the bin a few weeks later after. It seems like the algorithm is a basic highlight whatever scam show is pumped by the bingers at the time. Not even the show i last watched (consitantly everyday) is easy to navigate back to if u close the app. I swapped back to the streaming sites and found more NetFlix content immedataly. Thx for observables though!
- rawoke083600 5y ago
  Yup like https://www.wepickflix.com https://www.wepickflix.com :p - I tried to implemented a "group recommendation" - since watching "tv/shows/movies" is many times a "group activity".
deleted 5y ago
[deleted]
dmitriid 5y ago
Do create a button called "wipe my recommendations and start anew"
- tedsanders 5y ago
  There already exists button for wiping your recommendations and starting anew: It's labeled 'add profile'. :) (Otherwise, you can scroll through you watch history clicking delete a zillion times.)
  - merlincorey 5y ago
    I assume you probably mean well and are trying to help the GP, but this comes off as a totally tone-deaf PR response. The GP doesn't want to create a new profile, but they still want to avoid "clicking delete a zillion times" in their watch history. In my opinion you should acknowledge that the user has an actual problem that your team could solve by providing them that button THEN offer the workaround of creating a new profile with apologies instead of a potentially snarky smile.
    - tedsanders 5y ago
      I don't work at Netflix. I'm not Netflix PR. The smile was meant in good spirit to acknowledge the hackiness of the solution. I don't see why this solution doesn't solve the problem. Creating a new profile is effectively the same as resetting your profile. It absolutely is a solution. Same thing with Facebook or any other service. If you want to "reset" your Facebook profile, just create a new one.
      - travisjungroth 5y ago
        Maybe they mixed up your username with mine? And if this is the Ted Sanders I’m thinking of, good place to mention that you did previously work at Netflix. Offering help with a hacky solution and smiley face would be consistent with the Ted I met.
        5y ago

laxatives 5y ago

I would be a lot more interested in a technical article than hearing about the history of Netflix.

rwilson4 5y ago
You might be interested in Trustworthy Online Experiments by Kohavi et al then. I enjoyed the historical progression!
travisjungroth 5y ago
We've recently done a large blog series on experimentation. https://netflixtechblog.com/netflix-a-culture-of-learning-394bc7d0f94c https://netflixtechblog.com/netflix-a-culture-of-learning-39... Kinda covers everything and links out to posts that cover the software engineering in more detail and papers that cover the math in more detail.
- gordon_freeman 5y ago
  the blog posts series on rec system is interesting. But when I watch Netflix I still run into simple issues. can you please take care of some basics first, such as if I press 'thumbs down' then Netflix would never recommend that title again or similar to such titles.
  - dmitriid 5y ago
    They can't do that. Netflix now has very little content since all the big players have pulled out and started their own streaming services. Netflix has to recommend what it has: old movies, subpar TV shows, things you don't like, Netflix-produced content.

smartbit 5y ago

If their experiment are zo good, why are the recommendations so bad that I had to turn off the feature that adds items to My List? Several series I’ve enjoyed watching several years ago now appear on Netflix. Too late for me on Netflix, but as I liked them I rate them with thumbs up, hoping that Netflix would give me better recommendations. Instead of recommending other movies or series, to my dismay Netflix starts recommending the series I’ve already watched elsewhere. As I can only rate a series I’ve watched, why on earth does Netflix recommend something I’ve already watched? The diversification of Netflix is great, but most of it is too little or too late for my taste. Still better than Disney or Apple TV+ where I could find 1 or 2 shows of my liking. Looking forward to a couple of months of HBO Max once it is available in my region. HBO Europe made some award winning documentaries and I’m hoping to find more that I haven’t watched. Luckily for me there is an alternative for the streaming platform as for the past several decades every week my local TV stations have 1 or 2 great movies and maybe a new serie, selected not by experiments but by real people of flesh and blod.

ajmurmann 5y ago
If you want something selected and curated by actual humans, I recommend checking out Mubi. Lots of odd ball, but phenomenal movies on there. They started out having thirty movies on there and replacing the oldest every day with a new movie. It's often times a few in a row by a specific director or around a theme. They now also a have decent library in addition. Almost every movie I find truly memorable in they just few years came to me via Mubi.
onlyrealcuzzo 5y ago
Recommendations aren't really chosen by "experiments". And certainly, they aren't A/B testing budgeting and development of shows. Recommendation engines can be improved with experiments. And they - along with budgeting and show development (at least at Netflix) - are "driven by big data". Further, it doesn't matter if you - personally - like a feature. If that feature is moving the numbers and passes the A/B test, it will make itself into the app. They run thousands of experiments. Of course there are going to be several that you - personally - do not like.
dewey 5y ago

deathanatos 5y ago

> Netflix abandoned the ‘large catalog’ strategy for a ‘right for you, personalized catalog’ strategy. The "right for [me]" catalog seems to contain a lot of "get the viewer hooked on the first two seasons of a show; don't catalog the latter seasons".

jph 5y ago

Netflix experimentation seems to have suffered in the past year IMHO, with worsening recommendations and no simple way for users to provide feedback. And bugs are creeping in-- this week I've hit two different bugs in the Netflix recommendation engine, so something's amiss. The experiment I would love to see is an simple icon that means "I don't want to watch this".

john___matrix 5y ago
It's been pretty apparent for me that their who focus changed a couple of years ago from "here's some stuff you might enjoy watching" to "here's the stuff we want you to watch" or much like Google ads, "here's something you've already watched, watch it again" I'm almost at the point of cancelling my Netflix subscription simply because it's so much effort to find something of interest now.
tedsanders 5y ago
There already exists a simple icon for "I don't want to watch this": the thumbs down button. Clicking it is supposed to remove the show from your recommendations (after a day or so, to clear the caches).
- jph 5y ago
  It turns out that Netflix thumbs down conflicts with thumbs up. For example, pick a movie that you like, yet don't ever want to watch on Netflix; perhaps you've already seen the movie on a different service. Do you click thumbs up ("I like this") or thumbs down ("Not for me")? Next, if you watch some of the movie, then it shows up in your "Currently Watching" list, with icons for thumbs up ("I like this but don't want to continue watching"), thumbs down ("I don't like this"), and a broom ("Just cleaning up"). What I am suggesting as an experiment is a choice that means "I like this but don't want it on my main page".
Consultant32452 5y ago
For the love of God, can we please get an "Only show me things in my native language" option?

birdyrooster 5y ago

The only reason Netflix is in FAANG is because without them, people would be saying a very naught word.

thaumasiotes 5y ago
The reason they're there is that they are known primarily for the high salaries they pay.

anarazel 5y ago

I find the recommendations rarely useful. But what really bugs me is that the same series / movie typically shows up in all kinds of lists. Over and over and over. Besides the categories - often filled with stuff not matching the label - I see the same stuff repeatedly in "trending now", "top 10 today", "only on Netflix", "popular on Netflix".