4 ms·
You should probably add how exactly you retrieved the 4 million Reddit posts.
by minimaxir 7y ago
You should probably add how exactly you retrieved the 4 million Reddit posts.
- toyg 7y agoProbably with the official Reddit API? There are several libraries for it.
- Arimbr 7y agoRight! I used the official Reddit API. I created an APP, got the credentials for the API. Then used the Python library PRAW to consume the API. https://praw.readthedocs.io/en/latest/ https://praw.readthedocs.io/en/latest/ It took me 36 hours to collect the 4M posts. Reddit API returns results in batches of 100 results, and then sleeps for 2 seconds. You can find some more details on how it was built here https://blog.valohai.com/machine-learning-pipeline-classifying-reddit-posts https://blog.valohai.com/machine-learning-pipeline-classifyi... I can publish on Github the repository that runs two commands to collect the data if your are interested.
- minimaxir 7y agoThanks for the answer! I was curious since it seems like it was using the BigQuery dataset, but PRAW works too.
- dan1234 7y agoThe official API is well documented. https://www.reddit.com/dev/api/ https://www.reddit.com/dev/api/
- minimaxir 7y agoTrue, but 4M posts is heavy (even when bypassing the API itself and using unauthenticated requests), so was curious.