4 ms·

Yeah the thing I most wanted out of the post is a torrent to the data.

faizshah 12y ago
I have been working on a small shell script to get the top 500 posts of the top 5000 subreddits. The big issue is the api call limit is 60 req/min for OAuth2 tokens. It would save a lot of time if someone made a torrent of a data set like that, I will probably post the data I collect somewhere as well but it would be easier if reddit just hosted a data set like the one I described. They could probably post one on AWS public data sets. https://aws.amazon.com/datasets/ https://aws.amazon.com/datasets/
- bduerst 12y ago
  What kind of data are you pulling? Have you tried scraping the subreddit RSS feeds?
  - faizshah 12y ago
    I haven't tried that, I am looking for comments attached to usernames attached to articles/other comments in a tree and the subreddit that they are contained within. In order to define each user as an agent who has a probabilistic membership to the set of all interest groups (subpopulations) within the data set defined by a mixture model. My pipeline right now looks like: list of most popular SFW subreddits of all time -> gnu parallel -> curl on comment listings of top articles within subreddit -> json processing with jq -> gzip -> store on filesystem So at the end I should have a directory of 5000 subreddits containing a gzipped json file of the comment threads of their top 500 posts. I'm new to this kind of data processing but this is the easiest way I have found to do a short term, one time scrape of this data.
- HarryN 12y ago
  Could you not just request the .json feed whenever you like without limits? ( by appending .json to the reddit URL )
  - minimaxir 12y ago
    Note that almost all subreddits only let you travel back 1,000 posts using the .json feed. The exception is /r/all, which had infinite pagination, but the low API limits prevent much usability.
  - faizshah 12y ago
    I'm a little new to this but my understanding is that that would be the same as doing a GET request to the REST API. Which would have the same penalty for going over 30 req/s without an OAuth token. For example, this is what my get request to the REST endpoint looks like without URL parameters: GET reddit.com/r/explainlikeimfive/comments/2w1xzo.json Which returns the same response as appending .json to the the URL: https://www.reddit.com/r/explainlikeimfive/comments/2w1xzo/eli5_why_are_people_allowed_to_request_their_face.json https://www.reddit.com/r/explainlikeimfive/comments/2w1xzo/e...

I have been working on a small shell script to get the top 500 posts of the top 5000 subreddits. The big issue is the api call limit is 60 req/min for OAuth2 tokens. It would save a lot of time if someone made a torrent of a data set like that, I will probably post the data I collect somewhere as well but it would be easier if reddit just hosted a data set like the one I described. They could probably post one on AWS public data sets. https://aws.amazon.com/datasets/ https://aws.amazon.com/datasets/

bduerst 12y ago
What kind of data are you pulling? Have you tried scraping the subreddit RSS feeds?
- faizshah 12y ago
  I haven't tried that, I am looking for comments attached to usernames attached to articles/other comments in a tree and the subreddit that they are contained within. In order to define each user as an agent who has a probabilistic membership to the set of all interest groups (subpopulations) within the data set defined by a mixture model. My pipeline right now looks like: list of most popular SFW subreddits of all time -> gnu parallel -> curl on comment listings of top articles within subreddit -> json processing with jq -> gzip -> store on filesystem So at the end I should have a directory of 5000 subreddits containing a gzipped json file of the comment threads of their top 500 posts. I'm new to this kind of data processing but this is the easiest way I have found to do a short term, one time scrape of this data.
HarryN 12y ago
Could you not just request the .json feed whenever you like without limits? ( by appending .json to the reddit URL )
- minimaxir 12y ago
  Note that almost all subreddits only let you travel back 1,000 posts using the .json feed. The exception is /r/all, which had infinite pagination, but the low API limits prevent much usability.
- faizshah 12y ago
  I'm a little new to this but my understanding is that that would be the same as doing a GET request to the REST API. Which would have the same penalty for going over 30 req/s without an OAuth token. For example, this is what my get request to the REST endpoint looks like without URL parameters: GET reddit.com/r/explainlikeimfive/comments/2w1xzo.json Which returns the same response as appending .json to the the URL: https://www.reddit.com/r/explainlikeimfive/comments/2w1xzo/eli5_why_are_people_allowed_to_request_their_face.json https://www.reddit.com/r/explainlikeimfive/comments/2w1xzo/e...