5 ms·

The recommendation systems (i.e. Netflix) for movies and music are still just awful in terms of guessing my tastes. I still find that human based recommendatio

by fjabre 7y ago

The recommendation systems (i.e. Netflix) for movies and music are still just awful in terms of guessing my tastes. I still find that human based recommendations for things like music and movies consistently beat the algorithm at predicting what I'll like vs what I won't like just about every time. IMHO it will be many years before these recommendation engines consistently pick media I like to watch or listen to. They are still far inferior to human based recommendations. Guessing what someone will want to watch at any given moment is probably a harder problem than getting a car to drive autonomously. EDIT (source): I work on a mostly human powered recommendation engine/expert system @ lazyday.tv. It does not use ML to make recommendations. It takes some creative approaches to solving the movie recommendation problem using an expert system and optimizing search results.

stockholm 7y ago
Also taking the cost into account, it’s much more reasonable hiring 10 content curators dishing out 10 lists a day, as opposed to using a team of experts creating ML models.
softwaredoug 7y ago
I think a lot off preference towards humans is an issue of trust and a measure of your relationship with that person. Usually we get recommendations from friends and give them more thought and weight than some random ad we scroll by on Facebook. https://opensourceconnections.com/blog/2016/08/21/recommendations-systems-not-as-cool-as-friends/ https://opensourceconnections.com/blog/2016/08/21/recommenda...
jcims 7y ago
Is there any evidence that the recommendation systems are actually intent on putting something you would like to see in front of you instead of some other incentive? My YouTube subscriptions are all engineering/programming/space/guns/machining/electronics. I even see the same commenters in many of them. There is clearly a set of people with these interests that could be used to cross-pollinate interesting content. Yet my recommendations are all pop culture and sports. Spotify is the same way but i blame my kids for polluting their model of my interests, I’ve given up on Netflix. I can’t even find stuff i want to watch on there any more.
- fjabre 7y ago
  It's a fair point. Netflix for example has a subpar catalog to choose from (maybe 5000 titles max) and of course they are pushing their own content to the top. By comparison the known movie/tv universe of global movies and tv selections ever made is upwards of 500K.
- catalogia 7y ago
  > Is there any evidence that the recommendation systems are actually intent on putting something you would like to see in front of you instead of some other incentive? Something I've been wondering about recently is how infrastructure costs factor into this. Obviously videos get served from a local CDN node when possible, but I'm guessing sometimes the file you're requesting is uncommon enough that the local CDN node doesn't have it. Maybe this isn't the case with Netflix, but it's probably the case with youtube, just because they have that much more content. Adding more storage to the CDNs would cost money, and cache misses on the CDN also cost money, so does youtube have a financial incentive to bias their recommendations towards videos already cached near you?
  - jcims 7y ago
    Interesting. It would be a very Googly thing to do, so my guess is yes.
  - hooande 7y ago
    the cost of cache misses, even at google scale, is still trivial compared to their revenue
    - jcims 7y ago
      The infrastructure costs might be low, but transit performance back to Google is terrible for many ISPs (looking at you CenturyLink), so there might be 'quality' costs to consider as well.
- dxbydt 7y ago
  We had a bunch of people who worked on the recommendation system at netflix at the stat seminar. Their whole movie catalogue after hashing comfortably fits into a dataframe in R on your laptop. Even the audience-like matrix is not very big. They simply don’t have that much permanent content due to licensing issues. Parks & Rec for example is like one of their top 10 most streamed content - today - even though that thing aired like a decade ago! on top of which, its going to move to Peacock next quarter! So not only are they lagging in content contemporariness, they are losing what little they have because the networks who make the content are pulling the plug. An unrelated problem with movie recs is that the kind of movies & tv episodes being made today have homogenized because of international distribution issues & things like metoo. Your villains can’t be clearcut russians & chinese anymore, we sell to those countries. So it has to be some amorphous bad person with ambiguous intentions, which doesn’t work so well for james bond type stuff. Many of the older shows I love, I watch them on youtube & the comments there are mostly along the lines of “this would never ever get made in 2019. this is so bigoted. this is so mean spirited. this is openly sexist racist” etc etc. These are shows from 70s-80s, so its not like we had some genetic mutation and became a different species. We are still the same people, but those kinds of movies & shows aren’t happening anymore.
inertiatic 7y ago
Most recommendation systems are collaborative filtering + some fine tuning. I don't see how collaborative filtering can not work. Especially in the case of modern media providers where they have data about whether you actually finished watching something, how many videos of a series you watched etc.
- pps43 7y ago
  Collaborative filtering cannot recommend new stuff. By the time all your friends have watched and liked the new hotness it's already too late to recommend it to you.
  - inertiatic 7y ago
    By the time a single person with highly similar tastes to me has high engagement to a new item, that can be suggested, by providing a boost to new items. I'm not familiar with what the best approaches for the cold start problem of getting that first (few) interaction(s) are, but in the case of Netflix it's easily handled by a new on Netflix section.
- zwaps 7y ago
  The problem of recommending something (especially something genuinely new) is actually quite complex, which you realize by studying inferential/causal statistics. A statisticians would say that you are making "heroic assumptions" about both the taste of viewers, and the dimensions of the product, especially given that the products you have are actually quite sparse. Heck, there's even interesting literature in empirical economics (called empirical industrial organization) trying to estimate prices/demand/costs (which is, essentially the same issue as Netflix trying to estimate what you like) and dealing with the inherent curse of dimensionality. This becomes really obvious if you are actually write down explicitly what you are doing. Of course ML engineers are engineers and not really interested in causal inference, so Netflix wants me to watch 12 Hitler documentaries in a row and Amazon recommends that I better buy that second fridge right after the first. Where _does_ collaborative filtering actually work?
pps43 7y ago
Netflix is a bad example. After Netflix Prize they stopped trying to make a better recommendation system. They only used a few of the models from the winning ensemble, and now are using something even weaker instead. That said, the competition is hardly any better. Most use the same collaborative filtering from the Netflix Prize (Koren et al), which by design never recommends anything new. Most new "big data, machine learning" personalized recommendations just give you most popular plus some random noise. They perform well on some very artificial evaluation functions, but rarely A/B tested on actual users. When they are tested, they barely beat naive models like IMDB top 250.
- sida 7y ago
  Do you have evidence that they rarely hear naive models like IMDB top 250. I mean they have entire divisions of engineers dedicated to this so I assume they will disagree with you. So I am wondering if you have evidence that all those people are not doing anything to earn their pay
anjc 7y ago
What do you mean "human powered" rather than ML? How are the implicit/explicit judgements of relevance (e.g. likes, clicks, views, star ratings) that are used in rec algorithms any less human than, say, a recommendation from a single trusted expert? Personally I think it's amazing that even basic recommendation algorithms can instantly select 10 mostly relevant items from corpora of millions of irrelevant items. I think people are too used to receiving excellent recommendations and do not realise what receiving bad recommendations would be like. Hybrid systems such as Spotify's are really amazing when you think about it.
- fjabre 7y ago
  Human powered in the sense that the expert system we developed uses those human likes, views, star ratings and implements recommendation features to facilitate humans recommending titles to other humans in a very straightforward way. But there is no ML compiling a profile of every genre/title you've ever expressed interest in and then using that profile to guess what you want to see next like Netflix et al.
  - anjc 7y ago
    At some point you presumably have to assess user-similarity in order to let similar users recommend items to each other? How's this different from e.g. user-user CF in terms of compiling everyone's info into a single model? I'm not criticising but rather just curious
    - fjabre 7y ago
      I haven't found enough of a need for it at this point to make the application more useful as it stands without making great effort in training the necessary models to assess user-similarity for example. It is certainly faster to build an expert system that makes a lot of assumptions and iterate from there. There are plenty of use cases for ML and I hope to integrate it more into app some day to help automate repetitive tasks.
DonaldFisk 7y ago
Glad to hear that someone has still found a need for expert systems. Am I correct in assuming Lazyday has adopted a rule-based approach, using information about the movies (e.g. release date, genre, director, cast, etc.) and users (e.g. age, sex, nationality, current mood, etc.) that collaborative filtering systems don't use, and that you have evaluated the system against a collaborative filtering system which uses the ratings your users provided? I developed an online collaborative filtering movie recommendation system in the mid 1990s ( http://www.fmjlang.co.uk/morse/MORSE.html http://www.fmjlang.co.uk/morse/MORSE.html ), and analysed its performance, for my MSc project. The only information it used was how each user rated the movies they had seen. It used no other information about the movies or the users themselves. A surprising result I found was that using the ratings of the person whose tastes most closely matches your own to predict your ratings was very inaccurate. (Note that this is not necessarily the same as someone recommending you a movie, as they might know and take into account your tastes, and even if they didn't, they could still improve their recommendations simply by recommending the most popular movies, rather than the movies they liked.) Expanding from a single nearest neighbour to the average of the N nearest neighbours (for N = 25) gave quite good results. It seems that you're more similar to an average of all your friends than your closest friend. Using the algorithm I developed (described in detail in the paper) resulted in a further big improvement, which was unsurpassed until the Netflix prize. My data set was significantly smaller, though.
- fjabre 7y ago
  Very insightful. Thank you. Yes Lazyday is a rules based search engine. It makes best guesses based on what's currently popuplar, the movie's meta data and user search preferences and history. We have not evaluated it against a CF system in an A/B setup. Of course we have years of user frustration with Netflix / Prime recommends that has taught us a few valuable lessons about how to solve the problem of what to watch next. And yes it seems intuitive to assume that two people with the same tastes will like the same titles. The problem is, I might not be in the mood for movie A even though my virtual twin is, sometimes I would be in the mood for movie A if I had company that was also in the mood for movie A. So many edge cases when a recommendation engine is trying to guess what you want to watch next. Using the averages of your friends is very interesting. The issue would be building your dataset. Attaching it to a network like FB or building it yourself which would take even more time.
gav 7y ago
Netflix does such a bad job with recommendations for me that I can’t understand what they are doing. For example, I watch a lot of cooking shows. I just scrolled to the bottom of the recommendations on my tablet and they only suggest two—-and one of those I’ve watched every episode! They repeatedly fail to tell me about new series of shows I’ve watched, or really obvious connections like “Breakfast, Lunch & Dinner” as I’ve watched “Ugly Delicious”. The problem seems less of a ML one to me and more of a (lack of) metadata one. Edit: I plugged “Ugly Delicious” into lazyday.tv and the “Similar Titles” are very random.
- fjabre 7y ago
  Yes definitely lots of fine tuning still needs to be done for the similar titles playlist. It should be ok with most titles. I noticed that's a Netflix title so the algorithm maybe constraining the similar titles to the Netflix catalog. I have to look at that code again. Since there's less selection in Netflix the recommendations won't be as good because there's less to choose from. Some titles with less meta data are harder to get right as well.

by fjabre 7y ago

stockholm 7y ago

Also taking the cost into account, it’s much more reasonable hiring 10 content curators dishing out 10 lists a day, as opposed to using a team of experts creating ML models.

softwaredoug 7y ago

I think a lot off preference towards humans is an issue of trust and a measure of your relationship with that person. Usually we get recommendations from friends and give them more thought and weight than some random ad we scroll by on Facebook. https://opensourceconnections.com/blog/2016/08/21/recommendations-systems-not-as-cool-as-friends/ https://opensourceconnections.com/blog/2016/08/21/recommenda...

jcims 7y ago

Is there any evidence that the recommendation systems are actually intent on putting something you would like to see in front of you instead of some other incentive? My YouTube subscriptions are all engineering/programming/space/guns/machining/electronics. I even see the same commenters in many of them. There is clearly a set of people with these interests that could be used to cross-pollinate interesting content. Yet my recommendations are all pop culture and sports. Spotify is the same way but i blame my kids for polluting their model of my interests, I’ve given up on Netflix. I can’t even find stuff i want to watch on there any more.

fjabre 7y ago
It's a fair point. Netflix for example has a subpar catalog to choose from (maybe 5000 titles max) and of course they are pushing their own content to the top. By comparison the known movie/tv universe of global movies and tv selections ever made is upwards of 500K.
catalogia 7y ago
> Is there any evidence that the recommendation systems are actually intent on putting something you would like to see in front of you instead of some other incentive? Something I've been wondering about recently is how infrastructure costs factor into this. Obviously videos get served from a local CDN node when possible, but I'm guessing sometimes the file you're requesting is uncommon enough that the local CDN node doesn't have it. Maybe this isn't the case with Netflix, but it's probably the case with youtube, just because they have that much more content. Adding more storage to the CDNs would cost money, and cache misses on the CDN also cost money, so does youtube have a financial incentive to bias their recommendations towards videos already cached near you?
- jcims 7y ago
  Interesting. It would be a very Googly thing to do, so my guess is yes.
- hooande 7y ago
  the cost of cache misses, even at google scale, is still trivial compared to their revenue
  - jcims 7y ago
    The infrastructure costs might be low, but transit performance back to Google is terrible for many ISPs (looking at you CenturyLink), so there might be 'quality' costs to consider as well.
dxbydt 7y ago
We had a bunch of people who worked on the recommendation system at netflix at the stat seminar. Their whole movie catalogue after hashing comfortably fits into a dataframe in R on your laptop. Even the audience-like matrix is not very big. They simply don’t have that much permanent content due to licensing issues. Parks & Rec for example is like one of their top 10 most streamed content - today - even though that thing aired like a decade ago! on top of which, its going to move to Peacock next quarter! So not only are they lagging in content contemporariness, they are losing what little they have because the networks who make the content are pulling the plug. An unrelated problem with movie recs is that the kind of movies & tv episodes being made today have homogenized because of international distribution issues & things like metoo. Your villains can’t be clearcut russians & chinese anymore, we sell to those countries. So it has to be some amorphous bad person with ambiguous intentions, which doesn’t work so well for james bond type stuff. Many of the older shows I love, I watch them on youtube & the comments there are mostly along the lines of “this would never ever get made in 2019. this is so bigoted. this is so mean spirited. this is openly sexist racist” etc etc. These are shows from 70s-80s, so its not like we had some genetic mutation and became a different species. We are still the same people, but those kinds of movies & shows aren’t happening anymore.

inertiatic 7y ago

Most recommendation systems are collaborative filtering + some fine tuning. I don't see how collaborative filtering can not work. Especially in the case of modern media providers where they have data about whether you actually finished watching something, how many videos of a series you watched etc.

pps43 7y ago
Collaborative filtering cannot recommend new stuff. By the time all your friends have watched and liked the new hotness it's already too late to recommend it to you.
- inertiatic 7y ago
  By the time a single person with highly similar tastes to me has high engagement to a new item, that can be suggested, by providing a boost to new items. I'm not familiar with what the best approaches for the cold start problem of getting that first (few) interaction(s) are, but in the case of Netflix it's easily handled by a new on Netflix section.
zwaps 7y ago
The problem of recommending something (especially something genuinely new) is actually quite complex, which you realize by studying inferential/causal statistics. A statisticians would say that you are making "heroic assumptions" about both the taste of viewers, and the dimensions of the product, especially given that the products you have are actually quite sparse. Heck, there's even interesting literature in empirical economics (called empirical industrial organization) trying to estimate prices/demand/costs (which is, essentially the same issue as Netflix trying to estimate what you like) and dealing with the inherent curse of dimensionality. This becomes really obvious if you are actually write down explicitly what you are doing. Of course ML engineers are engineers and not really interested in causal inference, so Netflix wants me to watch 12 Hitler documentaries in a row and Amazon recommends that I better buy that second fridge right after the first. Where _does_ collaborative filtering actually work?

pps43 7y ago

Netflix is a bad example. After Netflix Prize they stopped trying to make a better recommendation system. They only used a few of the models from the winning ensemble, and now are using something even weaker instead. That said, the competition is hardly any better. Most use the same collaborative filtering from the Netflix Prize (Koren et al), which by design never recommends anything new. Most new "big data, machine learning" personalized recommendations just give you most popular plus some random noise. They perform well on some very artificial evaluation functions, but rarely A/B tested on actual users. When they are tested, they barely beat naive models like IMDB top 250.

sida 7y ago
Do you have evidence that they rarely hear naive models like IMDB top 250. I mean they have entire divisions of engineers dedicated to this so I assume they will disagree with you. So I am wondering if you have evidence that all those people are not doing anything to earn their pay

anjc 7y ago

What do you mean "human powered" rather than ML? How are the implicit/explicit judgements of relevance (e.g. likes, clicks, views, star ratings) that are used in rec algorithms any less human than, say, a recommendation from a single trusted expert? Personally I think it's amazing that even basic recommendation algorithms can instantly select 10 mostly relevant items from corpora of millions of irrelevant items. I think people are too used to receiving excellent recommendations and do not realise what receiving bad recommendations would be like. Hybrid systems such as Spotify's are really amazing when you think about it.

fjabre 7y ago
Human powered in the sense that the expert system we developed uses those human likes, views, star ratings and implements recommendation features to facilitate humans recommending titles to other humans in a very straightforward way. But there is no ML compiling a profile of every genre/title you've ever expressed interest in and then using that profile to guess what you want to see next like Netflix et al.
- anjc 7y ago
  At some point you presumably have to assess user-similarity in order to let similar users recommend items to each other? How's this different from e.g. user-user CF in terms of compiling everyone's info into a single model? I'm not criticising but rather just curious
  - fjabre 7y ago
    I haven't found enough of a need for it at this point to make the application more useful as it stands without making great effort in training the necessary models to assess user-similarity for example. It is certainly faster to build an expert system that makes a lot of assumptions and iterate from there. There are plenty of use cases for ML and I hope to integrate it more into app some day to help automate repetitive tasks.

DonaldFisk 7y ago

Glad to hear that someone has still found a need for expert systems. Am I correct in assuming Lazyday has adopted a rule-based approach, using information about the movies (e.g. release date, genre, director, cast, etc.) and users (e.g. age, sex, nationality, current mood, etc.) that collaborative filtering systems don't use, and that you have evaluated the system against a collaborative filtering system which uses the ratings your users provided? I developed an online collaborative filtering movie recommendation system in the mid 1990s ( http://www.fmjlang.co.uk/morse/MORSE.html http://www.fmjlang.co.uk/morse/MORSE.html ), and analysed its performance, for my MSc project. The only information it used was how each user rated the movies they had seen. It used no other information about the movies or the users themselves. A surprising result I found was that using the ratings of the person whose tastes most closely matches your own to predict your ratings was very inaccurate. (Note that this is not necessarily the same as someone recommending you a movie, as they might know and take into account your tastes, and even if they didn't, they could still improve their recommendations simply by recommending the most popular movies, rather than the movies they liked.) Expanding from a single nearest neighbour to the average of the N nearest neighbours (for N = 25) gave quite good results. It seems that you're more similar to an average of all your friends than your closest friend. Using the algorithm I developed (described in detail in the paper) resulted in a further big improvement, which was unsurpassed until the Netflix prize. My data set was significantly smaller, though.

fjabre 7y ago
Very insightful. Thank you. Yes Lazyday is a rules based search engine. It makes best guesses based on what's currently popuplar, the movie's meta data and user search preferences and history. We have not evaluated it against a CF system in an A/B setup. Of course we have years of user frustration with Netflix / Prime recommends that has taught us a few valuable lessons about how to solve the problem of what to watch next. And yes it seems intuitive to assume that two people with the same tastes will like the same titles. The problem is, I might not be in the mood for movie A even though my virtual twin is, sometimes I would be in the mood for movie A if I had company that was also in the mood for movie A. So many edge cases when a recommendation engine is trying to guess what you want to watch next. Using the averages of your friends is very interesting. The issue would be building your dataset. Attaching it to a network like FB or building it yourself which would take even more time.

gav 7y ago

Netflix does such a bad job with recommendations for me that I can’t understand what they are doing. For example, I watch a lot of cooking shows. I just scrolled to the bottom of the recommendations on my tablet and they only suggest two—-and one of those I’ve watched every episode! They repeatedly fail to tell me about new series of shows I’ve watched, or really obvious connections like “Breakfast, Lunch & Dinner” as I’ve watched “Ugly Delicious”. The problem seems less of a ML one to me and more of a (lack of) metadata one. Edit: I plugged “Ugly Delicious” into lazyday.tv and the “Similar Titles” are very random.

fjabre 7y ago
Yes definitely lots of fine tuning still needs to be done for the similar titles playlist. It should be ok with most titles. I noticed that's a Netflix title so the algorithm maybe constraining the similar titles to the Netflix catalog. I have to look at that code again. Since there's less selection in Netflix the recommendations won't be as good because there's less to choose from. Some titles with less meta data are harder to get right as well.