5 ms·

I'm working on a project relevant to this. Does anyone know if the author has shared this data set anywhere? Or does anyone know of any data sets that could be

I'm working on a project relevant to this. Does anyone know if the author has shared this data set anywhere? Or does anyone know of any data sets that could be used for developing mixture models to classify users into interest groups (like photographers, programmers etc)?

placeybordeaux 12y ago
Yeah the thing I most wanted out of the post is a torrent to the data.
- faizshah 12y ago
  I have been working on a small shell script to get the top 500 posts of the top 5000 subreddits. The big issue is the api call limit is 60 req/min for OAuth2 tokens. It would save a lot of time if someone made a torrent of a data set like that, I will probably post the data I collect somewhere as well but it would be easier if reddit just hosted a data set like the one I described. They could probably post one on AWS public data sets. https://aws.amazon.com/datasets/ https://aws.amazon.com/datasets/
  - bduerst 12y ago
    What kind of data are you pulling? Have you tried scraping the subreddit RSS feeds?
    - faizshah 12y ago
      I haven't tried that, I am looking for comments attached to usernames attached to articles/other comments in a tree and the subreddit that they are contained within. In order to define each user as an agent who has a probabilistic membership to the set of all interest groups (subpopulations) within the data set defined by a mixture model. My pipeline right now looks like: list of most popular SFW subreddits of all time -> gnu parallel -> curl on comment listings of top articles within subreddit -> json processing with jq -> gzip -> store on filesystem So at the end I should have a directory of 5000 subreddits containing a gzipped json file of the comment threads of their top 500 posts. I'm new to this kind of data processing but this is the easiest way I have found to do a short term, one time scrape of this data.
  - HarryN 12y ago
    Could you not just request the .json feed whenever you like without limits? ( by appending .json to the reddit URL )
    - minimaxir 12y ago
      Note that almost all subreddits only let you travel back 1,000 posts using the .json feed. The exception is /r/all, which had infinite pagination, but the low API limits prevent much usability.
    - faizshah 12y ago
      I'm a little new to this but my understanding is that that would be the same as doing a GET request to the REST API. Which would have the same penalty for going over 30 req/s without an OAuth token. For example, this is what my get request to the REST endpoint looks like without URL parameters: GET reddit.com/r/explainlikeimfive/comments/2w1xzo.json Which returns the same response as appending .json to the the URL: https://www.reddit.com/r/explainlikeimfive/comments/2w1xzo/eli5_why_are_people_allowed_to_request_their_face.json https://www.reddit.com/r/explainlikeimfive/comments/2w1xzo/e...
aroch 12y ago
If I'm remembering correctly, the raw data was given under an NDA/DND as a one time only deal. There was a subreddit associated with the data and collection but its since been banned.
- minimaxir 12y ago
  I was the one with this data. The subreddit which discussed provided the data went private. I have the raw data but it's infeasable to distribute due to size and the original source got in trouble for making it easily accessible.
Houshalter 12y ago
The admins released a bunch of anonymize voting data once. And several people have distributed datasets of scraped comments. Sorry I don't have links handy. Check through /r/redditdev
- faizshah 12y ago
  Thanks for the tip, I found the link you're referring to: https://github.com/umbrae/reddit-top-2.5-million https://github.com/umbrae/reddit-top-2.5-million
stared 12y ago
The author released data, links are at the bottom of the article (great thanks to him!).
- faizshah 12y ago
  Thanks for the tip, I just started downloading the data.

placeybordeaux 12y ago

Yeah the thing I most wanted out of the post is a torrent to the data.

faizshah 12y ago
I have been working on a small shell script to get the top 500 posts of the top 5000 subreddits. The big issue is the api call limit is 60 req/min for OAuth2 tokens. It would save a lot of time if someone made a torrent of a data set like that, I will probably post the data I collect somewhere as well but it would be easier if reddit just hosted a data set like the one I described. They could probably post one on AWS public data sets. https://aws.amazon.com/datasets/ https://aws.amazon.com/datasets/
- bduerst 12y ago
  What kind of data are you pulling? Have you tried scraping the subreddit RSS feeds?
  - faizshah 12y ago
    I haven't tried that, I am looking for comments attached to usernames attached to articles/other comments in a tree and the subreddit that they are contained within. In order to define each user as an agent who has a probabilistic membership to the set of all interest groups (subpopulations) within the data set defined by a mixture model. My pipeline right now looks like: list of most popular SFW subreddits of all time -> gnu parallel -> curl on comment listings of top articles within subreddit -> json processing with jq -> gzip -> store on filesystem So at the end I should have a directory of 5000 subreddits containing a gzipped json file of the comment threads of their top 500 posts. I'm new to this kind of data processing but this is the easiest way I have found to do a short term, one time scrape of this data.
- HarryN 12y ago
  Could you not just request the .json feed whenever you like without limits? ( by appending .json to the reddit URL )
  - minimaxir 12y ago
    Note that almost all subreddits only let you travel back 1,000 posts using the .json feed. The exception is /r/all, which had infinite pagination, but the low API limits prevent much usability.
  - faizshah 12y ago
    I'm a little new to this but my understanding is that that would be the same as doing a GET request to the REST API. Which would have the same penalty for going over 30 req/s without an OAuth token. For example, this is what my get request to the REST endpoint looks like without URL parameters: GET reddit.com/r/explainlikeimfive/comments/2w1xzo.json Which returns the same response as appending .json to the the URL: https://www.reddit.com/r/explainlikeimfive/comments/2w1xzo/eli5_why_are_people_allowed_to_request_their_face.json https://www.reddit.com/r/explainlikeimfive/comments/2w1xzo/e...

aroch 12y ago

If I'm remembering correctly, the raw data was given under an NDA/DND as a one time only deal. There was a subreddit associated with the data and collection but its since been banned.

minimaxir 12y ago
I was the one with this data. The subreddit which discussed provided the data went private. I have the raw data but it's infeasable to distribute due to size and the original source got in trouble for making it easily accessible.

Houshalter 12y ago

The admins released a bunch of anonymize voting data once. And several people have distributed datasets of scraped comments. Sorry I don't have links handy. Check through /r/redditdev

faizshah 12y ago
Thanks for the tip, I found the link you're referring to: https://github.com/umbrae/reddit-top-2.5-million https://github.com/umbrae/reddit-top-2.5-million

stared 12y ago

The author released data, links are at the bottom of the article (great thanks to him!).

faizshah 12y ago
Thanks for the tip, I just started downloading the data.