4 ms·

I wrote the recommendation system at Netflix (still in use after 5 years). Primary problem was company politics. Many groups were not happy that one person coul

I wrote the recommendation system at Netflix (still in use after 5 years). Primary problem was company politics. Many groups were not happy that one person could write a system that was better in A/B test, had more uptime and cheaper to run. All of it (ML, production, monitoring), was custom code.

chudi 8y ago
I always wondered how do you find the best artwork for the movies, is it multi armed bandits with thompson sampling? In my company navigating politics is always the hard part, the marketing team would love to spam everyone all the time and the product and sales team would love to sell some kind of upgraded recommendation, its hard to push back but with metrics of coverage, ctr, precision, etc we usually kept them quiet with this metrics
MaxLeiter 8y ago
Almost all of your top-level comments mention you did this
- adtac 8y ago
  Interestingly, his Medium post from two years ago [1] also says "5 years ago", and happens to be the only activity there. [1] https://medium.com/@sadikkapadia/i-wrote-the-recommendation-system-at-netflix-5-years-ago-705d02c6aa9f https://medium.com/@sadikkapadia/i-wrote-the-recommendation-...
  - sadikkapadia1 8y ago
    I don't keep track of time. That system is old technology. I did confirm from a Netflix employee that they still use it a few months ago. Deep learning, LDA (even one of Xavier's pet projects - k-means), did not do better.
- tschwimmer 8y ago
  hard to fault him in this particular case as it's directly pertinent to the original topic.
- sadikkapadia1 8y ago
  This is what working in a big company is like sometimes. Imagine a person like MaxLeiter working in Netflix. You would think that all staff would be happy that their work became easier. But a minority seem to have a zero-sum mindset. When I finish my current work I will talk about that also. Some of my older work is speech recognition. Download my thesis.
- deleted 8y ago
  [deleted]
theknight 8y ago
@sadikkapadia - Any idea why Recommendation Engine as a Service has not picked up? I realize that building a use-case specific recommendation engine is unique. However, I am wondering is there a recommendation engine as a service, which is similar to algolia available/possible. I see only 2 players - yusp and recombee. I'd appreciate any thought you have on this.
- sebst 8y ago
  > Any idea why Recommendation Engine as a Service has not picked up? These kind of services aren't so much exposed to the public and likely don't start at <100 bucks a month, which could be why those services are not that visible. However, there are some e-commerce services going in that direction, such like AgilOne...
  - sadikkapadia1 8y ago
    It is hard to sell technology to companies when they have their own teams (often using free libraries). Embedded teams are always experts and will often discredit better technology. Often the only method of testing is A/B. These can easily be manipulated. For instance at Netflix (ignoring more blatant practices), P-hacking (run thousands of simulations and report ones that worked), and HARKing (come up with a hypothesis after the results are known) are rampant. That is part of the reason the recommender has been degrading over the years.
    - prades 8y ago
      > That is part of the reason the recommender has been degrading over the years. Netflix is clearly promoting its original shows. Do you think your system is still in use now that they've moved to thumbs rating and percent match score?
      - sadikkapadia1 8y ago
        That is what I have been told. It is however clearly messed up.
amirouche 8y ago
Same situation with a smaller project. I still struggle to pull off it for the problem I face today. How did you succeed?
fratlas 8y ago
What was the tech stack? Did you use a graph db? I've built my own in neo4j for about 500M data points, but would love to know what you used.

chudi 8y ago

I always wondered how do you find the best artwork for the movies, is it multi armed bandits with thompson sampling? In my company navigating politics is always the hard part, the marketing team would love to spam everyone all the time and the product and sales team would love to sell some kind of upgraded recommendation, its hard to push back but with metrics of coverage, ctr, precision, etc we usually kept them quiet with this metrics

MaxLeiter 8y ago

Almost all of your top-level comments mention you did this

adtac 8y ago
Interestingly, his Medium post from two years ago [1] also says "5 years ago", and happens to be the only activity there. [1] https://medium.com/@sadikkapadia/i-wrote-the-recommendation-system-at-netflix-5-years-ago-705d02c6aa9f https://medium.com/@sadikkapadia/i-wrote-the-recommendation-...
- sadikkapadia1 8y ago
  I don't keep track of time. That system is old technology. I did confirm from a Netflix employee that they still use it a few months ago. Deep learning, LDA (even one of Xavier's pet projects - k-means), did not do better.
tschwimmer 8y ago
hard to fault him in this particular case as it's directly pertinent to the original topic.
sadikkapadia1 8y ago
This is what working in a big company is like sometimes. Imagine a person like MaxLeiter working in Netflix. You would think that all staff would be happy that their work became easier. But a minority seem to have a zero-sum mindset. When I finish my current work I will talk about that also. Some of my older work is speech recognition. Download my thesis.
deleted 8y ago
[deleted]

theknight 8y ago

@sadikkapadia - Any idea why Recommendation Engine as a Service has not picked up? I realize that building a use-case specific recommendation engine is unique. However, I am wondering is there a recommendation engine as a service, which is similar to algolia available/possible. I see only 2 players - yusp and recombee. I'd appreciate any thought you have on this.

sebst 8y ago
> Any idea why Recommendation Engine as a Service has not picked up? These kind of services aren't so much exposed to the public and likely don't start at <100 bucks a month, which could be why those services are not that visible. However, there are some e-commerce services going in that direction, such like AgilOne...
- sadikkapadia1 8y ago
  It is hard to sell technology to companies when they have their own teams (often using free libraries). Embedded teams are always experts and will often discredit better technology. Often the only method of testing is A/B. These can easily be manipulated. For instance at Netflix (ignoring more blatant practices), P-hacking (run thousands of simulations and report ones that worked), and HARKing (come up with a hypothesis after the results are known) are rampant. That is part of the reason the recommender has been degrading over the years.
  - prades 8y ago
    > That is part of the reason the recommender has been degrading over the years. Netflix is clearly promoting its original shows. Do you think your system is still in use now that they've moved to thumbs rating and percent match score?
    - sadikkapadia1 8y ago
      That is what I have been told. It is however clearly messed up.

amirouche 8y ago

Same situation with a smaller project. I still struggle to pull off it for the problem I face today. How did you succeed?

fratlas 8y ago

What was the tech stack? Did you use a graph db? I've built my own in neo4j for about 500M data points, but would love to know what you used.