3 ms·
You guys have built quite a nice and useful product. I was thinking of a similar project which required crawling all of reddit. But presently there are some hic
by frontman1988 5y ago
You guys have built quite a nice and useful product. I was thinking of a similar project which required crawling all of reddit. But presently there are some hiccups due to the large amount of data involved.
I wanted to ask how you guys manage the reddit data? Do you use praw/pushshift to get all the comments/post titles or is it limited to selected subreddits? Including all comments and all subreddits is quite a lot of data I guess. Any suggestions to collect that data and then work with it effectively? Thanks.
- mojoe 5y agoWe use praw, it works great if you fill out Reddit's forms to register your user-agent as per their terms of service. We get data about all posts, but don't get the comments themselves, just the post metadata (titles, likes, number of comments, etc). Good luck!