4 ms·

I didn't use pushshift, sorry. The data was collected from bigquery, stored locally into CSV files, and then I just wrote a node.js script to compute similariti

by anvaka 8y ago

I didn't use pushshift, sorry. The data was collected from bigquery, stored locally into CSV files, and then I just wrote a node.js script to compute similarities.

Scaevolus 8y ago
Did you simply collect "user has posted to X, Y, and Z subreddits", or did you look at frequency too?
- anvaka 8y ago
  I didn't look into frequency. Is there a version of jaccard similarity that accounts for frequencies?
  - yorwba 8y ago
    There's a weighted variant: https://en.wikipedia.org/wiki/Jaccard_index#Weighted_Jaccard_similarity_and_distance https://en.wikipedia.org/wiki/Jaccard_index#Weighted_Jaccard...
- minimaxir 8y ago
  The reason I asked the question is because back in 2016 I had a similar (now out of date) approach to finding related subreddits at scale using Jaccard similarity: https://minimaxir.com/2016/06/reddit-related-subreddits/ https://minimaxir.com/2016/06/reddit-related-subreddits/ There, I only built a user edge if a given user commented on 5 distinct threads in a subreddit, since a lot of subreddit interaction was due to brigading.