4 ms·

Just curious, what is the state of the art of recommendation systems these days? Is this pretty much a solved problem or is there more to explore?

by pelcg 5y ago

mrfox321 5y ago
There's always some new neural network module that pushes the needle.
amelius 5y ago
And how does it perform compared to the solution that won the Netflix prize, years ago?
- kqr 5y ago
  Netflix gets away with mainly training on dense data from high-frequency users that binge on content. The techniques they use don't adapt very well to most other organisations.
  - whiplash451 5y ago
    Netflix _designs_ movies/series so that the majority of people will like them. And so Netflix' recommendation has become largely irrelevant now that the company is pushing its own content agenda. Their recommendation algorithm is operating under massive override by marketing rules.
kqr 5y ago
There's more to explore. In particular, using groups of similar products and users to infer recommendations given only very sparse data is powerful but underexplored in production usage. (Disclosure: I work at a a company which just expanded their product portfolio in this direction. The first few pilot customers show very good results, but there are a lot of aspects and nuances and alternative approaches we haven't had time to try yet.)
plafl 5y ago
Do you feel that recommendations given to you are perfect? That more or less should answer the question. Evaluating recommendation systems is hard because you actually require a human in the loop. Even worse, giving the recommendations alters the human behavior. Then you need to think what metric are you going to use. For training you will most probably use a proxy metric that correlates. Maybe you want to optimize different metrics and they actually need to be balanced. Then there are lot of confounding variables: maybe a better UX will improve the metrics than a better algorithm, or a change of products. It seems that with big enough data you can improve old models with deep learning but I think recommenders are very far from similar gains to other fields (NLP or CV for example). And most companies don't have that much data.
- stepbeek 5y ago
  Sure, but we don’t need to be sat next to them. A large e-commerce company can A/B test new models to improve. The success metric is revenue. For smaller customer bases this is a tricky problem, but I’d argue that automated recommendations don’t work at a small scale anyway so manual curation is king.
  - kqr 5y ago
    If you do it right, automated recommendations work also for smaller customer bases. There's a lot of redundancy in customer and product data that can be exploited to generalise the behaviour of groups of customers over groups of products. Doing this automatically is a huge positive ROI thing over manual curation. In fact, humans are not even that good at coming up with good recommendations. Manual re-arrangement has almost universally been an anti-relevance feature in A/B tests I've looked at. But of course, almost nobody does the automation right.
    - plafl 5y ago
      That's the good part: putting in place a simple system will beat manual recommendations. Sadly even moderately big businesses lack sometimes automated recommendations. There are several companies that offer the service but I don't know how many clients they get. Why the disinterest? No idea, I value good recommendations as a customer but maybe it's not that relevant in the big picture.
eachro 5y ago
Rec systems that tech giants use looks vastly different than what ML academics study. Industry rec systems are a constantly evolving beast. Why? Because these teams of ML engineers are constantly running A/B tests to tweak some aspect of the model. The resulting model likely doesnt fit nicely into something category of model you can describe as collaborative filtering/lambdamart/etc b/c it's some highly performant glued together mess.
- w1nk 5y ago
  Your point about academia studying different systems than industry rings quite true. Lots of the academic recommender systems are built on datasets and techniques that simply don't scale to real world data/applications. That said, this pytorch work looks to live more in the realm of application than academia. Building and scaling large neural embeddings is pretty close to industry practice these days and this library at least claims to solve some of the challenges in doing so.
no_time 5y ago
>Just curious, what is the state of the art of recommendation systems these days? Asking your friends for recommendations. In my opinion automated systems are still not nearly as good as another person knowing you and your preferences.
marcyb5st 5y ago
It is not solved. Let me go into a bit of details: the current state of the art is using two-tower like models that create dense embeddings both for users and items. These towers are, in essence, two models that need to keep into account histories, reviews, pictures, color, material, ... . So this problem is as solved as the models that can handle these problems are. Currently, I would expect that teams are experimenting with repurposing Attention-like architectures in some way to get better embeddings, especially from sequence like features.
- mrfox321 5y ago
  There are more complex models than the one you described. That query / item tower is cheap and cache-friendly. These tend to be used for high recall. You can echew those performance benefits in favor of neural networks that allow for the features of the query and item to interact. These models are goaling for high precision.
  - Mehdi2277 5y ago
    Two towers are not independent of neural networks. Most two tower models I encounter are neural networks for each tower with optionally a mixing towering/layer after the two. And sometimes it's not two but even more towers depending on how your features are grouped. I would consider two towers pretty much a necessity for large corpus retrieval as step 1 in any rec system with many items and many requests. Stage 2 or later models can be heavier and be whatever you like.
brian_spiering 5y ago
One area to explore is how to optimize a sequence of events (e.g., user journey). Reinforcement learning (RL) might be useful for that.
whiplash451 5y ago
You'll find a detailed answer on paperswithcode, selecting the "Recommendation task": https://paperswithcode.com/datasets?task=recommendation-systems&page=1 https://paperswithcode.com/datasets?task=recommendation-syst... Like many have already said, this is mostly an academic answer (even if some of the papers are written by the industry). In the industrial world, the answer is a lot more subtle. Each domain has its own constraints and the best method will vary. Also, keep in mind that in some domains like ad tech, the whole measurement process is messed up by the attribution mechanism, which attributes a sale to the last click (which clearly is a poor indication of whether the recommendation engine is doing a good job -- the industry settled on this attribution mechanism because it is easy to audit).
whiplash451 5y ago
Recommendation will never be solved (like nuclear fusion can be) because it is about human minds liking or disliking things, and so it is pretty much "psychology-complete". You will only see the field make steady, regular and infinite progress towards an unknown asymptote.

Just curious, what is the state of the art of recommendation systems these days? Is this pretty much a solved problem or is there more to explore?

by pelcg 5y ago

Just curious, what is the state of the art of recommendation systems these days? Is this pretty much a solved problem or is there more to explore?

mrfox321 5y ago

There's always some new neural network module that pushes the needle.

amelius 5y ago

And how does it perform compared to the solution that won the Netflix prize, years ago?

kqr 5y ago
Netflix gets away with mainly training on dense data from high-frequency users that binge on content. The techniques they use don't adapt very well to most other organisations.
- whiplash451 5y ago
  Netflix _designs_ movies/series so that the majority of people will like them. And so Netflix' recommendation has become largely irrelevant now that the company is pushing its own content agenda. Their recommendation algorithm is operating under massive override by marketing rules.

kqr 5y ago

There's more to explore. In particular, using groups of similar products and users to infer recommendations given only very sparse data is powerful but underexplored in production usage. (Disclosure: I work at a a company which just expanded their product portfolio in this direction. The first few pilot customers show very good results, but there are a lot of aspects and nuances and alternative approaches we haven't had time to try yet.)

plafl 5y ago

Do you feel that recommendations given to you are perfect? That more or less should answer the question. Evaluating recommendation systems is hard because you actually require a human in the loop. Even worse, giving the recommendations alters the human behavior. Then you need to think what metric are you going to use. For training you will most probably use a proxy metric that correlates. Maybe you want to optimize different metrics and they actually need to be balanced. Then there are lot of confounding variables: maybe a better UX will improve the metrics than a better algorithm, or a change of products. It seems that with big enough data you can improve old models with deep learning but I think recommenders are very far from similar gains to other fields (NLP or CV for example). And most companies don't have that much data.

stepbeek 5y ago
Sure, but we don’t need to be sat next to them. A large e-commerce company can A/B test new models to improve. The success metric is revenue. For smaller customer bases this is a tricky problem, but I’d argue that automated recommendations don’t work at a small scale anyway so manual curation is king.
- kqr 5y ago
  If you do it right, automated recommendations work also for smaller customer bases. There's a lot of redundancy in customer and product data that can be exploited to generalise the behaviour of groups of customers over groups of products. Doing this automatically is a huge positive ROI thing over manual curation. In fact, humans are not even that good at coming up with good recommendations. Manual re-arrangement has almost universally been an anti-relevance feature in A/B tests I've looked at. But of course, almost nobody does the automation right.
  - plafl 5y ago
    That's the good part: putting in place a simple system will beat manual recommendations. Sadly even moderately big businesses lack sometimes automated recommendations. There are several companies that offer the service but I don't know how many clients they get. Why the disinterest? No idea, I value good recommendations as a customer but maybe it's not that relevant in the big picture.

eachro 5y ago

Rec systems that tech giants use looks vastly different than what ML academics study. Industry rec systems are a constantly evolving beast. Why? Because these teams of ML engineers are constantly running A/B tests to tweak some aspect of the model. The resulting model likely doesnt fit nicely into something category of model you can describe as collaborative filtering/lambdamart/etc b/c it's some highly performant glued together mess.

w1nk 5y ago
Your point about academia studying different systems than industry rings quite true. Lots of the academic recommender systems are built on datasets and techniques that simply don't scale to real world data/applications. That said, this pytorch work looks to live more in the realm of application than academia. Building and scaling large neural embeddings is pretty close to industry practice these days and this library at least claims to solve some of the challenges in doing so.

no_time 5y ago

>Just curious, what is the state of the art of recommendation systems these days? Asking your friends for recommendations. In my opinion automated systems are still not nearly as good as another person knowing you and your preferences.

marcyb5st 5y ago

It is not solved. Let me go into a bit of details: the current state of the art is using two-tower like models that create dense embeddings both for users and items. These towers are, in essence, two models that need to keep into account histories, reviews, pictures, color, material, ... . So this problem is as solved as the models that can handle these problems are. Currently, I would expect that teams are experimenting with repurposing Attention-like architectures in some way to get better embeddings, especially from sequence like features.

mrfox321 5y ago
There are more complex models than the one you described. That query / item tower is cheap and cache-friendly. These tend to be used for high recall. You can echew those performance benefits in favor of neural networks that allow for the features of the query and item to interact. These models are goaling for high precision.
- Mehdi2277 5y ago
  Two towers are not independent of neural networks. Most two tower models I encounter are neural networks for each tower with optionally a mixing towering/layer after the two. And sometimes it's not two but even more towers depending on how your features are grouped. I would consider two towers pretty much a necessity for large corpus retrieval as step 1 in any rec system with many items and many requests. Stage 2 or later models can be heavier and be whatever you like.

brian_spiering 5y ago

One area to explore is how to optimize a sequence of events (e.g., user journey). Reinforcement learning (RL) might be useful for that.

whiplash451 5y ago

You'll find a detailed answer on paperswithcode, selecting the "Recommendation task": https://paperswithcode.com/datasets?task=recommendation-systems&page=1 https://paperswithcode.com/datasets?task=recommendation-syst... Like many have already said, this is mostly an academic answer (even if some of the papers are written by the industry). In the industrial world, the answer is a lot more subtle. Each domain has its own constraints and the best method will vary. Also, keep in mind that in some domains like ad tech, the whole measurement process is messed up by the attribution mechanism, which attributes a sale to the last click (which clearly is a poor indication of whether the recommendation engine is doing a good job -- the industry settled on this attribution mechanism because it is easy to audit).

whiplash451 5y ago

Recommendation will never be solved (like nuclear fusion can be) because it is about human minds liking or disliking things, and so it is pretty much "psychology-complete". You will only see the field make steady, regular and infinite progress towards an unknown asymptote.