5 ms·

Movie Recommendations with k-Nearest Neighbors and Cosine Similarity

139 points by mcphilip 13y ago

lowglow 13y ago
Would anyone be interested in more ML stuff coming out of #SFHN? We have a new speaker in the pipeline wanting to speak to Simulated Annealing.
- robert_tweed 13y ago
  Yes.
  - lowglow 13y ago
    Sounds great. I think we'll aim for a May deadline on the ML talk.
- agibsonccc 13y ago
  Along that same note you may want to use the words genetic algorithms. It sounds cooler and they're in the same category. Look in to hill climbing algorithms if you're ever curious how basic optimization algorithms (re: games) work.
wavesum 13y ago
The Netflix challenge forums are a treasure trove when it comes to this particular ML problem. In the end the algorithms grew into huge monsters blending results from tens of algorithms, but the most interesting discoveries were made in the first year IMO. http://www.netflixprize.com/community/ http://www.netflixprize.com/community/
- thomasahle 13y ago
  Wow, I never thought of looking there. It's amazing reading tricks like that which people have come up with after years of thinking and experimenting. And they seem usable for quite a wide range of rating related problems.
rpicard 13y ago
I've been thinking about something like this for a while. My idea was to use data from IMDB to create a graph where the distance between movies is some ranking of similarity based on the people involved in the movie, the genre, the setting and any other information you could get from the data set. You could say, "I want to watch a movie like The Wolf of Wall Street" and it would find the closest 10 movies in the graph. It's still something I'd like to play with if I find the time.
- mcphilip 13y ago
  I started on something similar as a side project[1]. I decided to build a dataset from the AFI Top 100 Films list and persist it in neo4j. The goal was to find interesting questions to answer with this dataset that couldn't easily be googled. Most of my time thus far has been spent gathering the dataset, but I do have a few example cypher queries answering the following simple questions [2]: 1) What actors have appeared in the most AFI Top 100 films? 2) What are the genres of the top ten films? 3) Have any actors appeared in 2 or more of the top 25 films? I'm working on building a much larger data set using a combination of freebase and imdb so that I can have enough data to start exploring much more interesting interesting questions (e.g. graph the frequencies of genres over the past 60 years; for a given film, find movies with the greatest overlap in genres, actors, and directors; generalize the n-degrees-to-bacon problem to work on any two actors; etc). [1]https://github.com/mcphilip/film-graph https://github.com/mcphilip/film-graph [2]http://htmlpreview.github.io/?https://github.com/mcphilip/film-graph/blob/master/film-graph-overview.html#query1 http://htmlpreview.github.io/?https://github.com/mcphilip/fi...
  - rpicard 13y ago
    Very cool. Thanks for posting this. The n-degrees-to-bacon problem is actual what made me think of this in the first place. It would be great to be able to plug in two actors and have it spit out an answer with the shortest path.
- deleted 13y ago
  [deleted]
- 13y ago
hitlin37 13y ago
This looks good, i will give it a try.Also GraphGists looks good way, quite similar to ipython. I have one question: Why do i use neo4j when i can do k-NN in python with so many macine learning toolkits available? What benefit do i get when using neo4j for movie recommendation?
- mcphilip 13y ago
  This post was an entry in the Winter 2013 Neo4j GraphGist Contest. The author of this submission also has a blog post about doing k-NN in R [1]. In other words, this GraphGist is an example of how you could do basic ML in neo4j, but it's not meant to imply that neo4j should be chosen over python. http://nicolemargaretwhite.blogspot.com/2013/12/movie-recommendations-with-k-nn-and.html http://nicolemargaretwhite.blogspot.com/2013/12/movie-recomm...
- PaulRobinson 13y ago
  Neo4j as a data persistence layer has many, many advantages over other data persistence layers for graph-like problems (social, recommendation, etc.), principally from the point of view of performance. You can do it in Python but you're going to have to persist and query your data somehow, and I think at that point you'll "get" why a graph DB might be beneficial. It's not an ML-related thing though, it's a data query performance thing.
sgt101 13y ago
The ideas that are used here are heavily researched. As an example of current work have a look at http://www.eecs.qmul.ac.uk/~laurissa/Laurissas_Pages/Publications_files/5137a351.pdf http://www.eecs.qmul.ac.uk/~laurissa/Laurissas_Pages/Publica... Most of the challenge is in getting a way of assessing the value of innovations in the algorithms - how do you know how well it works ? Difficult unless you are running a large scale recommender that users can't opt out of (because pop goes your stats if the do!)
Radim 13y ago
Related: benchmarks of available k-NN libs using cosine similarity, in Python. http://radimrehurek.com/2013/12/performance-shootout-of-nearest-neighbours-contestants/ http://radimrehurek.com/2013/12/performance-shootout-of-near...
kenshiro_o 13y ago
This looks very good. I have not delved into maths stuff for quite some time so it is refreshing to read such an approachable article. Which other relatively simple techniques could we apply to find out who is similar to us in terms of movie taste, etc? Also, It may be better to compute cosine similarity based on movie type, as it may provide less noise. Moreover, I have never used R but it seems like a very neat language...
ManyNames 13y ago
I was wondering how this possibly got so high on HN with such a basic method, but the break-down and explanation given is great!
- UK-AL 13y ago
  Considering most stuff on HN is just web development/general programming its not that simple for some people on here. For anyone who's studied a bit of ML its simple though.
avaku 13y ago
KNN is not nearly as good as deeper probabilistic models. For example, if movie description mentions different subjects like "comedy" and "family", the model would not be able to differentiate them. It would only have a cluster "family comedies", but it won't be able to tell that a person likes "comedies". I've recently done a similar research on finding "company peers" using company descriptions. The descriptions are put through the LDA (http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation) to find the topics expressed in each description. Then, similar companies are identified using K-L divergence (http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_diverg...) between their topic distributions. It worked much better than cosine similarity in my tests. My guess is that it would work better for movies too. If interested, have a look at my results here: http://akuz.me/2014/03/finding-company-peers-using-lda/ http://akuz.me/2014/03/finding-company-peers-using-lda/

139 points by mcphilip 13y ago

Would anyone be interested in more ML stuff coming out of #SFHN? We have a new speaker in the pipeline wanting to speak to Simulated Annealing.

robert_tweed 13y ago
Yes.
- lowglow 13y ago
  Sounds great. I think we'll aim for a May deadline on the ML talk.
agibsonccc 13y ago
Along that same note you may want to use the words genetic algorithms. It sounds cooler and they're in the same category. Look in to hill climbing algorithms if you're ever curious how basic optimization algorithms (re: games) work.

wavesum 13y ago

The Netflix challenge forums are a treasure trove when it comes to this particular ML problem. In the end the algorithms grew into huge monsters blending results from tens of algorithms, but the most interesting discoveries were made in the first year IMO. http://www.netflixprize.com/community/ http://www.netflixprize.com/community/

thomasahle 13y ago
Wow, I never thought of looking there. It's amazing reading tricks like that which people have come up with after years of thinking and experimenting. And they seem usable for quite a wide range of rating related problems.

rpicard 13y ago

I've been thinking about something like this for a while. My idea was to use data from IMDB to create a graph where the distance between movies is some ranking of similarity based on the people involved in the movie, the genre, the setting and any other information you could get from the data set. You could say, "I want to watch a movie like The Wolf of Wall Street" and it would find the closest 10 movies in the graph. It's still something I'd like to play with if I find the time.

mcphilip 13y ago
I started on something similar as a side project[1]. I decided to build a dataset from the AFI Top 100 Films list and persist it in neo4j. The goal was to find interesting questions to answer with this dataset that couldn't easily be googled. Most of my time thus far has been spent gathering the dataset, but I do have a few example cypher queries answering the following simple questions [2]: 1) What actors have appeared in the most AFI Top 100 films? 2) What are the genres of the top ten films? 3) Have any actors appeared in 2 or more of the top 25 films? I'm working on building a much larger data set using a combination of freebase and imdb so that I can have enough data to start exploring much more interesting interesting questions (e.g. graph the frequencies of genres over the past 60 years; for a given film, find movies with the greatest overlap in genres, actors, and directors; generalize the n-degrees-to-bacon problem to work on any two actors; etc). [1]https://github.com/mcphilip/film-graph https://github.com/mcphilip/film-graph [2]http://htmlpreview.github.io/?https://github.com/mcphilip/film-graph/blob/master/film-graph-overview.html#query1 http://htmlpreview.github.io/?https://github.com/mcphilip/fi...
- rpicard 13y ago
  Very cool. Thanks for posting this. The n-degrees-to-bacon problem is actual what made me think of this in the first place. It would be great to be able to plug in two actors and have it spit out an answer with the shortest path.
deleted 13y ago
[deleted]
13y ago

hitlin37 13y ago

This looks good, i will give it a try.Also GraphGists looks good way, quite similar to ipython. I have one question: Why do i use neo4j when i can do k-NN in python with so many macine learning toolkits available? What benefit do i get when using neo4j for movie recommendation?

mcphilip 13y ago
This post was an entry in the Winter 2013 Neo4j GraphGist Contest. The author of this submission also has a blog post about doing k-NN in R [1]. In other words, this GraphGist is an example of how you could do basic ML in neo4j, but it's not meant to imply that neo4j should be chosen over python. http://nicolemargaretwhite.blogspot.com/2013/12/movie-recommendations-with-k-nn-and.html http://nicolemargaretwhite.blogspot.com/2013/12/movie-recomm...
PaulRobinson 13y ago
Neo4j as a data persistence layer has many, many advantages over other data persistence layers for graph-like problems (social, recommendation, etc.), principally from the point of view of performance. You can do it in Python but you're going to have to persist and query your data somehow, and I think at that point you'll "get" why a graph DB might be beneficial. It's not an ML-related thing though, it's a data query performance thing.

sgt101 13y ago

The ideas that are used here are heavily researched. As an example of current work have a look at http://www.eecs.qmul.ac.uk/~laurissa/Laurissas_Pages/Publications_files/5137a351.pdf http://www.eecs.qmul.ac.uk/~laurissa/Laurissas_Pages/Publica... Most of the challenge is in getting a way of assessing the value of innovations in the algorithms - how do you know how well it works ? Difficult unless you are running a large scale recommender that users can't opt out of (because pop goes your stats if the do!)

Radim 13y ago

Related: benchmarks of available k-NN libs using cosine similarity, in Python. http://radimrehurek.com/2013/12/performance-shootout-of-nearest-neighbours-contestants/ http://radimrehurek.com/2013/12/performance-shootout-of-near...

kenshiro_o 13y ago

This looks very good. I have not delved into maths stuff for quite some time so it is refreshing to read such an approachable article. Which other relatively simple techniques could we apply to find out who is similar to us in terms of movie taste, etc? Also, It may be better to compute cosine similarity based on movie type, as it may provide less noise. Moreover, I have never used R but it seems like a very neat language...

ManyNames 13y ago

I was wondering how this possibly got so high on HN with such a basic method, but the break-down and explanation given is great!

UK-AL 13y ago
Considering most stuff on HN is just web development/general programming its not that simple for some people on here. For anyone who's studied a bit of ML its simple though.

avaku 13y ago

KNN is not nearly as good as deeper probabilistic models. For example, if movie description mentions different subjects like "comedy" and "family", the model would not be able to differentiate them. It would only have a cluster "family comedies", but it won't be able to tell that a person likes "comedies". I've recently done a similar research on finding "company peers" using company descriptions. The descriptions are put through the LDA (http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation) to find the topics expressed in each description. Then, similar companies are identified using K-L divergence (http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_diverg...) between their topic distributions. It worked much better than cosine similarity in my tests. My guess is that it would work better for movies too. If interested, have a look at my results here: http://akuz.me/2014/03/finding-company-peers-using-lda/ http://akuz.me/2014/03/finding-company-peers-using-lda/