3 ms·

"We think you'd also like..." and the Math of Suggestion – Part 1 (from Directed Edge)

56 points by wheels 18y ago

wheels 18y ago
It's unfortunately split up into two parts due to requirements on the blog that published it (literally "Founder Scene"), but this hits on some of the basics of collaborative filtering. The next part gets into item-based collaborative filtering and a little on graph-based stuff. I'll post the second part here when it comes around. The article is definitely targeted at being a quick introduction to how recommendations work rather than being a how-to. Most Gründerszene's subscribers aren't developers so it's an attempt to present the stuff in a digestible format.
- thomasfl 18y ago
  Is there a part two out there yet?
  - wheels 18y ago
    No, it's written, but not published yet. It was written as one article, but since it was over their word limit, they split it into two parts. I think they usually space them out by about a week.
chrislo 18y ago
Ilya Grigorik wrote a great article on recommendation systems (with code in Ruby): http://www.igvita.com/2007/01/15/svd-recommendation-system-in-ruby/ http://www.igvita.com/2007/01/15/svd-recommendation-system-i...
- deleted 18y ago
  [deleted]
- davi 18y ago
  much better than the submitted article, thanks
  - wheels 18y ago
    For our own blog, which tends to be more tech-heavy, I'm open to writing on some topics that people have specific interest in in the recommendations world, so if there's stuff you're curious about I'm all ears. This article was naturally geared more towards the audience of the place publishing it.
ivyirwin 18y ago
It will be nice to see part two, but there is enough fodder in part one to talk about. There is no doubt recommendation engines are a great sales generator. But are they any good at what they do? Plotting points on a graph based on user ratings has always seemed artificial to me. I may like a book because of its rhythm and someone else may like the subject matter; will we like the same books? This is why I think netflix created their competition. They're not looking for better math, but for a better approach all together. At the end of the day, it probably has to be a combination of factors. For example, think of pandora's engine. It is based on analysis of the music, not just user preferences. And yes, math will be involved, which is why part two of this article will be worth reading.
- wheels 18y ago
  The funny thing is that we're not using the standard approaches to recommendations, though in an introduction to how the systems work in general, it seemed best to take the simplest explanation. Part II has something on graph-based approaches, but in general, the class of algorithms we're using tends to, at least in my opinion, produce better results in a couple orders of magnitude less time. I loved Greg's post here: http://cacm.acm.org/blogs/blog-cacm/22925-what-is-a-good-recommendation-algorithm/fulltext http://cacm.acm.org/blogs/blog-cacm/22925-what-is-a-good-rec... It's a valid question to ask, "Will these algorithms really suss out my personal taste?" And I think the answer is, a lot of the time, they'll get close enough to be interesting. When we started Directed Edge, one of the tipping point moments was seeing how bad some of the related films shown in IMDB for my favorite film were. It was a real watershed moment this week when I was running our algorithm against the Netflix dataset after adding 15 of my own ratings to the system and my two favorite films (which I had not rated) came out at #1 and #5, respectively, and at that, with 300x less computation time than a standard item-to-item recommender requires. (We've got implementations of a lot of standard algorithms that we use for benchmarking our own.)

56 points by wheels 18y ago

It's unfortunately split up into two parts due to requirements on the blog that published it (literally "Founder Scene"), but this hits on some of the basics of collaborative filtering. The next part gets into item-based collaborative filtering and a little on graph-based stuff. I'll post the second part here when it comes around. The article is definitely targeted at being a quick introduction to how recommendations work rather than being a how-to. Most Gründerszene's subscribers aren't developers so it's an attempt to present the stuff in a digestible format.

thomasfl 18y ago
Is there a part two out there yet?
- wheels 18y ago
  No, it's written, but not published yet. It was written as one article, but since it was over their word limit, they split it into two parts. I think they usually space them out by about a week.

chrislo 18y ago

Ilya Grigorik wrote a great article on recommendation systems (with code in Ruby): http://www.igvita.com/2007/01/15/svd-recommendation-system-in-ruby/ http://www.igvita.com/2007/01/15/svd-recommendation-system-i...

deleted 18y ago
[deleted]
davi 18y ago
much better than the submitted article, thanks
- wheels 18y ago
  For our own blog, which tends to be more tech-heavy, I'm open to writing on some topics that people have specific interest in in the recommendations world, so if there's stuff you're curious about I'm all ears. This article was naturally geared more towards the audience of the place publishing it.

ivyirwin 18y ago

It will be nice to see part two, but there is enough fodder in part one to talk about. There is no doubt recommendation engines are a great sales generator. But are they any good at what they do? Plotting points on a graph based on user ratings has always seemed artificial to me. I may like a book because of its rhythm and someone else may like the subject matter; will we like the same books? This is why I think netflix created their competition. They're not looking for better math, but for a better approach all together. At the end of the day, it probably has to be a combination of factors. For example, think of pandora's engine. It is based on analysis of the music, not just user preferences. And yes, math will be involved, which is why part two of this article will be worth reading.

wheels 18y ago
The funny thing is that we're not using the standard approaches to recommendations, though in an introduction to how the systems work in general, it seemed best to take the simplest explanation. Part II has something on graph-based approaches, but in general, the class of algorithms we're using tends to, at least in my opinion, produce better results in a couple orders of magnitude less time. I loved Greg's post here: http://cacm.acm.org/blogs/blog-cacm/22925-what-is-a-good-recommendation-algorithm/fulltext http://cacm.acm.org/blogs/blog-cacm/22925-what-is-a-good-rec... It's a valid question to ask, "Will these algorithms really suss out my personal taste?" And I think the answer is, a lot of the time, they'll get close enough to be interesting. When we started Directed Edge, one of the tipping point moments was seeing how bad some of the related films shown in IMDB for my favorite film were. It was a real watershed moment this week when I was running our algorithm against the Netflix dataset after adding 15 of my own ratings to the system and my two favorite films (which I had not rated) came out at #1 and #5, respectively, and at that, with 300x less computation time than a standard item-to-item recommender requires. (We've got implementations of a lot of standard algorithms that we use for benchmarking our own.)