3 ms·
I work for reddit and here I’ve used a similar technique to your jaccard distance but with one twist: divide by the size of the smaller subreddit (in your case,
by ketralnis 7y ago
I work for reddit and here I’ve used a similar technique to your jaccard distance but with one twist: divide by the size of the smaller subreddit (in your case, the number of unique posters that you’ve recorded). That gives you a directional relatedness, that is programming->python but not necessarily python->programming. Used this way you account for the giant subreddit problem automatically but now the results are less “amitheasshole is related to askreddit” and more like “linguisticshumor is a more niche version of linguistics”.
The great thing is that it’s actually more actionable as far as recommendations go! Everybody has already heard of the bigger version of this subreddit, but they probably haven’t heard of the smaller versions. And it’s self-correcting. As a subreddit gets bigger we are less likely to recommend it (which is great because it needs our help less)
- jcims 7y agoCan people still make their upvotes/downvotes public? I found that useful for sussing out related subreddits when it was possible back in the day.
- anvaka 7y agoThis is super awesome, thank you for sharing! If you guys are interested in seeing how your recommendation work for the entire reddit, I'd be happy to build you a spaceship similar to this one https://github.com/anvaka/word2vec-graph https://github.com/anvaka/word2vec-graph . I couldn't find an easy way to download the entire recommendation graph, but it would be awesome if we could make it work. My email is the same as this account at gmail, and twitter is all open: https://twitter.com/anvaka https://twitter.com/anvaka
- yantrams 7y agoIn case you haven't come across it already, here is a very exhaustive list of distance measures for dealing with problems of this kind - http://www.iiisci.org/journal/CV$/sci/pdfs/GS315JG.pdf http://www.iiisci.org/journal/CV$/sci/pdfs/GS315JG.pdf I fooled around a bit with lastfm data for band recommendation and found this sheet quite helpful. If you are interested in learning more about asymmetrical similarity, here is a great primer by Tversky - http://www.cogsci.ucsd.edu/~coulson/203/tversky-features.pdf http://www.cogsci.ucsd.edu/~coulson/203/tversky-features.pdf
- anvaka 7y agoThis is absolute treasure trove. Thank you so much!
- bobosha 7y agoNormalizing by the user count should help. My suggestion would be something much simpler ie. to try to compare content itself e.g. top 1000 posts from each subreddit, and estimate (say) cosine sim + Tfidf. Wouldn't that be a better indicator? Also, instead of pairwise comparison, you could try clustering (HDBSCAN for example) to reduce computational complexity. But great work, love your visualizations!
- ethn 7y agoI’ve also rediscovered this in similar work I do. But upon later research I’ve learned this is called containment.