4 ms·

What's HNs best sociological guess for why recommendation systems are still pretty crappy? It can't be that it's technically or mathematically too challenging t

by tw1010 8y ago

What's HNs best sociological guess for why recommendation systems are still pretty crappy? It can't be that it's technically or mathematically too challenging to figure out. There are too many smart people out there for this not to be cracked and then immediately spread to everyone. I feel like there's a Peter Thiel-esque explanation to it. Something about a lack of diversity of thought and/or the lack of an incentive to encourage first-principles inventions in software engineering (or a lack of investments into long-term innovations which benefit everyone).

tomnipotent 8y ago
> best sociological guess for why recommendation systems are still pretty crappy That what we like can't be boiled down to an optimization function over a sparse matrix.
- tw1010 8y ago
  With the right optimization function and the feature extractors, of course it could. I don't buy that at all.
  - tomnipotent 8y ago
    Because you're focusing on a technology solution. This isn't a technology problem. We cannot model the impossibly complicated measure of taste, the best we can do is optimize on captured data and generalize. This technique - using optimization functions over sparse matrices - has taken us as far as it can go.
a_lieb 8y ago
I think recommendation systems actually are pretty good at recommending stuff you already like. I think a lot of us tend to buzz right past those as "well, obviously I've seen that already" and don't take note that it actually hit the nail right on the head for the first 5-10 results. The trick is to find stuff that cuts across the space in some sense—stuff that's out of your current wheelhouse, but still has features that you would like. (E.g. I like the music of both David Rawlings and Mark Ribot, even though they are in wildly different genres, because they both have a certain ear for melody and relationship with dissonance that I like.) I don't think anyone is doing that well yet.
- DonaldFisk 8y ago
  Collaborative systems are designed to do this. If enough people who like A also like B, and you like A but haven't rated B, they should predict you will like B.
lottin 8y ago
One of the reasons, I suspect, is that there is a lack of feedback. If you recommend something to a friend, later on you can ask them about it and they'll tell you whether they liked it or not and why. Whereas these recommendation engines keep recommending things but have no way of knowing whether the recommendations they make are any good.
- mooreds 8y ago
  Wow, this is very perceptive and could possibly be mitigated. For instance, a lot of the recommendation engines I've looked at use basic actions (like clicks) to determine whether something is worth surfacing. But if you could to not just clicks but something more material (like time on a page, a purchase or social share) into the mix, I think recommendations would get better. Tying into feedback further removed from the initial action, if it could scale, would help the model as well.
  - chudi 8y ago
    The guys from yt explains this in a paper that talks about their recsys. What they actually do is once you have a candidate set of videos, pick the best ones that maximizes the total time of watch time so they can show you more ads. That's the reason the videos that are frequently recommended are the really long duration ones
    - mooreds 8y ago
      Ah. Of course what you optimize depends on what you consider valuable, so that makes total sense. But if you want to optimize for customer satisfaction, maybe the video watched to completion or the one shared is better than the one the total amount of watch time.
- DonaldFisk 8y ago
  This can be done by leave-one-out cross-validation in a collaborative filtering system. If jane saw schindler's list and gave it 8/10, delete this from the database and use the rest of the data to predict it, then compare the predicted score with the score she gave. I did this for a movie recommendation system i developed: http://www.fmjlang.co.uk/morse/MORSE-Paper.html http://www.fmjlang.co.uk/morse/MORSE-Paper.html
QML 8y ago
Could you elaborate on which recommendation systems do you find crappy?
JD557 8y ago
Based on the difficulties I had implementing recommender systems, I think it boils down to two main problems: 1. Evaluating recommender systems is REALLY hard. When it comes to evaluation, recommender systems are very different from other machine learning tasks: Let's assume the classic methodology of collecting a dataset of events, splitting it at a certain point in time to get a training and test set, and checking the precision and recall - If your recommender system works as expected, you will influence what your users do in the future, so your test set is probably not going to represent what your users would actually do. - Having a precision and recall of 1 is actually bad. That means your recommender system was perfect, but also useless (you only recommended stuff that the user was already going to pick anyway). One way to address this is to just use A/B tests and try to optimize some business metric (e.g. number of purchases). This is usually "good enough", but this will make your recommender focus on sales, not on user satisfaction. There are also some other metrics that can be used[1], but there are so many of them and some are not very practical to implement, so I guess that everyone just goes with the A/B testing approach. 2. Recommendations require an explanation A lot of recommender algorithms are black boxes. Sure, you can write "this was recommended based on users similar to you" when you use a collaborative filtering algorithm, but that doesn't help much. Recommendations without an explanation are not that useful. When a friend recommends you a movie, he'll also tell you "why". Otherwise, it's really hard to make users trust your recommendations (especially if the recommender system recommends something outside of the user's "comfort zone") I've noticed that a lot of sites have improved on this front though, and I do enjoy those recommendations a lot more. [1]: Herlocker et al., "Evaluating collaborative filtering recommender systems" http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.97.5270&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.97....
- tw1010 8y ago
  That's not a sociological explanation. I want something "elephant in the brain"[1]-esque, something that shines light on poor incentive equilibrium on the game theoretic/social plane. [1] http://elephantinthebrain.com/ http://elephantinthebrain.com/
- DonaldFisk 8y ago
  Whether someone likes a movie (or a song) is usually subjective, often emotional, and can at best only be rationalized. There are exceptions, e.g. documentaries. You could e.g. have the system recognize a preference for a director, actor, or genre, and add that to the predicted rating. But i don't think using that kind of information to make a prediction would add to a collaborative system's accuracy.
platz 8y ago
Because the current recommender/ML systems model the past as identical to the future as a matter of structural equivalence (There is a profound assumption of linearity between past and future e.g. in time, and linearity in space). They must assume the world is a stable & stationary distribution. Which doesn't leave a lot of room for modeling change. This actually makes recommender systems things that train humans, that render them more predictable, by grouping them into categories base on 'latent variables'. It makes everyone more boring by reducing diversity instead of encouraging it. You can easily see this in your youtube recommendations. Jaron Larnier calls these kinds of algorithms manipulative - they train people to optimize engagement metrics, rather than sometime qualitative like quality. https://livestream.com/internetsociety2/cybercon/videos/166218044 https://livestream.com/internetsociety2/cybercon/videos/1662...
grahar64 8y ago
A recommender system is super hard because humans are hard. Even if the problem is well defined, like movie recommendations, it is exceptionally hard. Read about the Netflix challenge https://en.wikipedia.org/wiki/Netflix_Prize https://en.wikipedia.org/wiki/Netflix_Prize which took some of the worlds best researchers (trying to win $1MM) 3 years to increase Netflixes algorithm by 10%.
setr 8y ago
Without any real experience or knowledge beyond dealing with their outcomes in various e-stores, and a bit of thinking on the matter, I'm pretty damn sure the goals are wrong, which leads to the whole setup being wrong. Eg netflix pushes you into having useless ratings that at best become 1 or 5, and nothing in between, by designing themselves in such a fashion: you can't look over your ratings properly, you can't edit them in any sensible fashion, you can't interact or update your ratings at all. And of course, your ratings are limited to whatever the hell they have. And then because the ratings come out so incredibly divorced from a user's taste, netflix naturally gives up on the rating system, and makes its judgement based on your interactions with the video: how much of a show you watched, how quickly you binged, how quickly you ditched it, etc. But your viewing habits are not indicative of your taste, and is often quite divorced from it. Everyone has their guilty pleasures; those trash shows you watch because its mindless entertainment. The vast majority of the stuff you watch is not what you wished you watched. That's not the shit you want to be recommended, if you care about being recommended things (otherwise, high popularity is more than sufficient). But thats because netflix does not have the goal of showing you good movies and shows, things you can watch and actually remember three days later. They have the goal of mindless entertainment (they need your time, not your brain), and the recommender system succeeds at that task. But its difficult to see that goal and think "yeah, thats the personal recommendation I always wanted". It's not what your friend would recommend to you; it's the absolute opposite. Part of the problem is that a proper catalogue of your ratings for various shows/movies is likely to be a very good indicator of taste, but afaik, the western media has no culture of it, no decent tooling, and no decent systems for it. I know of no decent system to aggregate and curate my movie collection, adding, maintaining, and noting a simplified opinion of it (a rating), for my personal use, beyond the physical/file system storage of it, and half-assed custom-built systems for it (eg text file). Which also implies there's no public dataset of it from which to build a proper recommendation system from. However, I know at least the anime community offers us a possibility of redemption. MyAnimeList (MAL) and co. do offer a decent system of it, and the community itself lends itself towards the cataloguing task[1]; in my own usage of MAL, I consistently go back through and update my ratings to reflect my current taste, and more importantly, the difference between an old 10-rating and a new one, when I see something that beats out by previous by a long shot, shifting the whole rating. And I'm willing to do the work because maintaining the list is relatively simple, and it offers me great utility; both things you won't find by amazon or netflix. And I expect others use the site in the same fashion. I think if you really want a decent recommender system, this is the kind of dataset you want. Something that actually happens to reflect a user's taste, because the users themselves are incentivized into maintaining a simplified snapshot of their taste, because they can actually derive value back out of it. I've had intent for some time now to do that, but never moved beyond the scraping stage (some 40GB of MAL lists sitting on an aws db for almost a year now). Also I'm pretty damn sure with such a dataset, you can get really far with a few simple intuitions (picked up off random papers and my own thinking): 1) User definitions of ratings is mostly arbitrary, but people trivially agree on at least this much: in a 0-10 scale, anything below 5 is disliked, and anything above is liked. 0 is trash, and 10 is good. Normalize on it. 2) Anime communities have a notion of "dropping" shows. Watch 3 episodes, and if you don't like it, move on. This is probably true in all media. These extremely disliked shows are likely not recorded (though MAL has an item for recording dropped), so ratings are inherently skewed positively. 3) The average taste is bland. You have to cluster before you can do anything. 4) No one wants the average. They want the best, in their own little zones. You don't want the centroid, you want the edges of the cluster. 5) No one wants only the shit they already know, and its easier to steal from someone close to the target (in taste) than to generate a personal recommendation. Find nearby neighbors in taste, then find the anime they like, but doesn't belong to the cluster. (this kind of find is by far the most valuable; thats when a recommender system actually offers something you probably wouldn't have found on your own) 6) People are naturally more willing to accept deficiencies in someone else's list, than a personally generated one. Instead of only generating a new list, give a list of the user's neighbor-lists as well. 7) Point 5 is good for the community: If their taste is that similar, shouldn't they be friends? Or enemies. 8) Rarity is far more important than rating at the extremes. If two users even list the same rare anime, they probably have a lot in common (they managed to find it, meaning they're in similar communities, which implies similar tastes, even if this specific one differed). The most popular anime is worthless: everyone has an opinion on it, and most likely not a useful one. (TF-IDF) One day I'll do something with this. Maybe. But I'm pretty damn sure you can get really far with very little. But the ratings systems are generally built poorly, and so the data they generate is worthless, which leads to a worthless recommender system. And to work around the poorly built data, they convolute it with all sorts of fancy ideas to get around it, but never actually fix the data generation issue. Notably, one thing I've never found a satisfactory answer for is how to evaluate success/improvement. The best metric I've heard yet is split an animelist in half (or more like 70/30), use one half to predict the other half. Eh. [1] My favorite description of this tendency is Otaku: Database Animals [2] Anime Booru's are basically an amazon turk-built ML-labaled dataset; community constructed long before ML was in any way notable. They are accidentally ideal for ML. I'm pretty sure the west accomplishes something similar with porn datasets, but to a much weaker degree.

by tw1010 8y ago

tomnipotent 8y ago

> best sociological guess for why recommendation systems are still pretty crappy That what we like can't be boiled down to an optimization function over a sparse matrix.

tw1010 8y ago
With the right optimization function and the feature extractors, of course it could. I don't buy that at all.
- tomnipotent 8y ago
  Because you're focusing on a technology solution. This isn't a technology problem. We cannot model the impossibly complicated measure of taste, the best we can do is optimize on captured data and generalize. This technique - using optimization functions over sparse matrices - has taken us as far as it can go.

a_lieb 8y ago

I think recommendation systems actually are pretty good at recommending stuff you already like. I think a lot of us tend to buzz right past those as "well, obviously I've seen that already" and don't take note that it actually hit the nail right on the head for the first 5-10 results. The trick is to find stuff that cuts across the space in some sense—stuff that's out of your current wheelhouse, but still has features that you would like. (E.g. I like the music of both David Rawlings and Mark Ribot, even though they are in wildly different genres, because they both have a certain ear for melody and relationship with dissonance that I like.) I don't think anyone is doing that well yet.

DonaldFisk 8y ago
Collaborative systems are designed to do this. If enough people who like A also like B, and you like A but haven't rated B, they should predict you will like B.

lottin 8y ago

One of the reasons, I suspect, is that there is a lack of feedback. If you recommend something to a friend, later on you can ask them about it and they'll tell you whether they liked it or not and why. Whereas these recommendation engines keep recommending things but have no way of knowing whether the recommendations they make are any good.

mooreds 8y ago
Wow, this is very perceptive and could possibly be mitigated. For instance, a lot of the recommendation engines I've looked at use basic actions (like clicks) to determine whether something is worth surfacing. But if you could to not just clicks but something more material (like time on a page, a purchase or social share) into the mix, I think recommendations would get better. Tying into feedback further removed from the initial action, if it could scale, would help the model as well.
- chudi 8y ago
  The guys from yt explains this in a paper that talks about their recsys. What they actually do is once you have a candidate set of videos, pick the best ones that maximizes the total time of watch time so they can show you more ads. That's the reason the videos that are frequently recommended are the really long duration ones
  - mooreds 8y ago
    Ah. Of course what you optimize depends on what you consider valuable, so that makes total sense. But if you want to optimize for customer satisfaction, maybe the video watched to completion or the one shared is better than the one the total amount of watch time.
DonaldFisk 8y ago
This can be done by leave-one-out cross-validation in a collaborative filtering system. If jane saw schindler's list and gave it 8/10, delete this from the database and use the rest of the data to predict it, then compare the predicted score with the score she gave. I did this for a movie recommendation system i developed: http://www.fmjlang.co.uk/morse/MORSE-Paper.html http://www.fmjlang.co.uk/morse/MORSE-Paper.html

QML 8y ago

Could you elaborate on which recommendation systems do you find crappy?

JD557 8y ago

Based on the difficulties I had implementing recommender systems, I think it boils down to two main problems: 1. Evaluating recommender systems is REALLY hard. When it comes to evaluation, recommender systems are very different from other machine learning tasks: Let's assume the classic methodology of collecting a dataset of events, splitting it at a certain point in time to get a training and test set, and checking the precision and recall - If your recommender system works as expected, you will influence what your users do in the future, so your test set is probably not going to represent what your users would actually do. - Having a precision and recall of 1 is actually bad. That means your recommender system was perfect, but also useless (you only recommended stuff that the user was already going to pick anyway). One way to address this is to just use A/B tests and try to optimize some business metric (e.g. number of purchases). This is usually "good enough", but this will make your recommender focus on sales, not on user satisfaction. There are also some other metrics that can be used[1], but there are so many of them and some are not very practical to implement, so I guess that everyone just goes with the A/B testing approach. 2. Recommendations require an explanation A lot of recommender algorithms are black boxes. Sure, you can write "this was recommended based on users similar to you" when you use a collaborative filtering algorithm, but that doesn't help much. Recommendations without an explanation are not that useful. When a friend recommends you a movie, he'll also tell you "why". Otherwise, it's really hard to make users trust your recommendations (especially if the recommender system recommends something outside of the user's "comfort zone") I've noticed that a lot of sites have improved on this front though, and I do enjoy those recommendations a lot more. [1]: Herlocker et al., "Evaluating collaborative filtering recommender systems" http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.97.5270&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.97....

tw1010 8y ago
That's not a sociological explanation. I want something "elephant in the brain"[1]-esque, something that shines light on poor incentive equilibrium on the game theoretic/social plane. [1] http://elephantinthebrain.com/ http://elephantinthebrain.com/
DonaldFisk 8y ago
Whether someone likes a movie (or a song) is usually subjective, often emotional, and can at best only be rationalized. There are exceptions, e.g. documentaries. You could e.g. have the system recognize a preference for a director, actor, or genre, and add that to the predicted rating. But i don't think using that kind of information to make a prediction would add to a collaborative system's accuracy.

platz 8y ago

Because the current recommender/ML systems model the past as identical to the future as a matter of structural equivalence (There is a profound assumption of linearity between past and future e.g. in time, and linearity in space). They must assume the world is a stable & stationary distribution. Which doesn't leave a lot of room for modeling change. This actually makes recommender systems things that train humans, that render them more predictable, by grouping them into categories base on 'latent variables'. It makes everyone more boring by reducing diversity instead of encouraging it. You can easily see this in your youtube recommendations. Jaron Larnier calls these kinds of algorithms manipulative - they train people to optimize engagement metrics, rather than sometime qualitative like quality. https://livestream.com/internetsociety2/cybercon/videos/166218044 https://livestream.com/internetsociety2/cybercon/videos/1662...

grahar64 8y ago

A recommender system is super hard because humans are hard. Even if the problem is well defined, like movie recommendations, it is exceptionally hard. Read about the Netflix challenge https://en.wikipedia.org/wiki/Netflix_Prize https://en.wikipedia.org/wiki/Netflix_Prize which took some of the worlds best researchers (trying to win $1MM) 3 years to increase Netflixes algorithm by 10%.

setr 8y ago

Without any real experience or knowledge beyond dealing with their outcomes in various e-stores, and a bit of thinking on the matter, I'm pretty damn sure the goals are wrong, which leads to the whole setup being wrong. Eg netflix pushes you into having useless ratings that at best become 1 or 5, and nothing in between, by designing themselves in such a fashion: you can't look over your ratings properly, you can't edit them in any sensible fashion, you can't interact or update your ratings at all. And of course, your ratings are limited to whatever the hell they have. And then because the ratings come out so incredibly divorced from a user's taste, netflix naturally gives up on the rating system, and makes its judgement based on your interactions with the video: how much of a show you watched, how quickly you binged, how quickly you ditched it, etc. But your viewing habits are not indicative of your taste, and is often quite divorced from it. Everyone has their guilty pleasures; those trash shows you watch because its mindless entertainment. The vast majority of the stuff you watch is not what you wished you watched. That's not the shit you want to be recommended, if you care about being recommended things (otherwise, high popularity is more than sufficient). But thats because netflix does not have the goal of showing you good movies and shows, things you can watch and actually remember three days later. They have the goal of mindless entertainment (they need your time, not your brain), and the recommender system succeeds at that task. But its difficult to see that goal and think "yeah, thats the personal recommendation I always wanted". It's not what your friend would recommend to you; it's the absolute opposite. Part of the problem is that a proper catalogue of your ratings for various shows/movies is likely to be a very good indicator of taste, but afaik, the western media has no culture of it, no decent tooling, and no decent systems for it. I know of no decent system to aggregate and curate my movie collection, adding, maintaining, and noting a simplified opinion of it (a rating), for my personal use, beyond the physical/file system storage of it, and half-assed custom-built systems for it (eg text file). Which also implies there's no public dataset of it from which to build a proper recommendation system from. However, I know at least the anime community offers us a possibility of redemption. MyAnimeList (MAL) and co. do offer a decent system of it, and the community itself lends itself towards the cataloguing task[1]; in my own usage of MAL, I consistently go back through and update my ratings to reflect my current taste, and more importantly, the difference between an old 10-rating and a new one, when I see something that beats out by previous by a long shot, shifting the whole rating. And I'm willing to do the work because maintaining the list is relatively simple, and it offers me great utility; both things you won't find by amazon or netflix. And I expect others use the site in the same fashion. I think if you really want a decent recommender system, this is the kind of dataset you want. Something that actually happens to reflect a user's taste, because the users themselves are incentivized into maintaining a simplified snapshot of their taste, because they can actually derive value back out of it. I've had intent for some time now to do that, but never moved beyond the scraping stage (some 40GB of MAL lists sitting on an aws db for almost a year now). Also I'm pretty damn sure with such a dataset, you can get really far with a few simple intuitions (picked up off random papers and my own thinking): 1) User definitions of ratings is mostly arbitrary, but people trivially agree on at least this much: in a 0-10 scale, anything below 5 is disliked, and anything above is liked. 0 is trash, and 10 is good. Normalize on it. 2) Anime communities have a notion of "dropping" shows. Watch 3 episodes, and if you don't like it, move on. This is probably true in all media. These extremely disliked shows are likely not recorded (though MAL has an item for recording dropped), so ratings are inherently skewed positively. 3) The average taste is bland. You have to cluster before you can do anything. 4) No one wants the average. They want the best, in their own little zones. You don't want the centroid, you want the edges of the cluster. 5) No one wants only the shit they already know, and its easier to steal from someone close to the target (in taste) than to generate a personal recommendation. Find nearby neighbors in taste, then find the anime they like, but doesn't belong to the cluster. (this kind of find is by far the most valuable; thats when a recommender system actually offers something you probably wouldn't have found on your own) 6) People are naturally more willing to accept deficiencies in someone else's list, than a personally generated one. Instead of only generating a new list, give a list of the user's neighbor-lists as well. 7) Point 5 is good for the community: If their taste is that similar, shouldn't they be friends? Or enemies. 8) Rarity is far more important than rating at the extremes. If two users even list the same rare anime, they probably have a lot in common (they managed to find it, meaning they're in similar communities, which implies similar tastes, even if this specific one differed). The most popular anime is worthless: everyone has an opinion on it, and most likely not a useful one. (TF-IDF) One day I'll do something with this. Maybe. But I'm pretty damn sure you can get really far with very little. But the ratings systems are generally built poorly, and so the data they generate is worthless, which leads to a worthless recommender system. And to work around the poorly built data, they convolute it with all sorts of fancy ideas to get around it, but never actually fix the data generation issue. Notably, one thing I've never found a satisfactory answer for is how to evaluate success/improvement. The best metric I've heard yet is split an animelist in half (or more like 70/30), use one half to predict the other half. Eh. [1] My favorite description of this tendency is Otaku: Database Animals [2] Anime Booru's are basically an amazon turk-built ML-labaled dataset; community constructed long before ML was in any way notable. They are accidentally ideal for ML. I'm pretty sure the west accomplishes something similar with porn datasets, but to a much weaker degree.