3 ms·
You indicated that you used the Pushshift.io datasets, but how did you compute Jaccard Similarity on a dataset of 38M?
by minimaxir 8y ago
You indicated that you used the Pushshift.io datasets, but how did you compute Jaccard Similarity on a dataset of 38M?
- scrollbar 8y agoCheck out Graphlab Create's recommender toolkit, pretty fast for sets of that size https://turi.com/products/create/docs/graphlab.toolkits.recommender.html https://turi.com/products/create/docs/graphlab.toolkits.reco...
- Smithalicious 8y ago+1 for this recommendation, but it's called turicreate now and can be found here: https://github.com/apple/turicreate https://github.com/apple/turicreate
- anvaka 8y agoI didn't use pushshift, sorry. The data was collected from bigquery, stored locally into CSV files, and then I just wrote a node.js script to compute similarities.
- Scaevolus 8y agoDid you simply collect "user has posted to X, Y, and Z subreddits", or did you look at frequency too?
- anvaka 8y agoI didn't look into frequency. Is there a version of jaccard similarity that accounts for frequencies?
- yorwba 8y agoThere's a weighted variant: https://en.wikipedia.org/wiki/Jaccard_index#Weighted_Jaccard_similarity_and_distance https://en.wikipedia.org/wiki/Jaccard_index#Weighted_Jaccard...
- minimaxir 8y agoThe reason I asked the question is because back in 2016 I had a similar (now out of date) approach to finding related subreddits at scale using Jaccard similarity: https://minimaxir.com/2016/06/reddit-related-subreddits/ https://minimaxir.com/2016/06/reddit-related-subreddits/ There, I only built a user edge if a given user commented on 5 distinct threads in a subreddit, since a lot of subreddit interaction was due to brigading.