4 ms·

Are you sure? The dataset is from here: https://archive.org/details/2015_reddit_comments_corpus https://archive.org/details/2015_reddit_comments_corpus Looking

Are you sure? The dataset is from here: https://archive.org/details/2015_reddit_comments_corpus https://archive.org/details/2015_reddit_comments_corpus Looking at the thread from the release, I see no explanation of how he got the data, but I see several people commenting that they finally have a way to get comments beyond the 1000 per account: https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/ https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_eve...

jedberg 10y ago
It looks like they got clever, so what they are doing is getting 1000 links from each subreddit, indexing the comments and linking it back to the user. So yeah, they did figure out how to got back more than 1000 comments for some users, but they'll still be limited by 1000 links per reddit per sort. So for a very active reddit, you can't get really old comments.
- _wmd 10y ago
  IIRC you can use the search API to enumerate 1000 at a time, using a "created_at>=..." restriction to move the cursor starting position. I had to poke in the code to find out how to talk Amazon's search syntax directly
  - jedberg 10y ago
    Yeah, that might work too. That would be pretty clever.
- MasterScrat 10y ago
  What about an official data dump, like StackExchange does? Wouldn't that be beneficial to all parties? Are there legal restrictions?
  - i336_ 10y ago
    I haven't poured over the code, but reddit is most definitely more than one server so there's gotta be a sync protocol in there somewhere. So, technically speaking, no restrictions there. Practically speaking, on the other hand, reddit likely never took user content licensing into account to begin with - they never knew they'd get all of digg a couple years in, and 2006 was a different time besides. The only way reddit would be able to release the data now would either be a specific clause (added from the start) allowing such in the TOS, or a general-purpose "we can do w/e we want with your data"; nobody thought to add the former (as noted), and of course the latter would have been roasted alive the moment it was noticed. So, me thinks licensing issues. Modmailing /r/reddit.com is the canonical way to reach the admins (for any reason), if you'd like more info.

jedberg 10y ago

It looks like they got clever, so what they are doing is getting 1000 links from each subreddit, indexing the comments and linking it back to the user. So yeah, they did figure out how to got back more than 1000 comments for some users, but they'll still be limited by 1000 links per reddit per sort. So for a very active reddit, you can't get really old comments.

_wmd 10y ago
IIRC you can use the search API to enumerate 1000 at a time, using a "created_at>=..." restriction to move the cursor starting position. I had to poke in the code to find out how to talk Amazon's search syntax directly
- jedberg 10y ago
  Yeah, that might work too. That would be pretty clever.
MasterScrat 10y ago
What about an official data dump, like StackExchange does? Wouldn't that be beneficial to all parties? Are there legal restrictions?
- i336_ 10y ago
  I haven't poured over the code, but reddit is most definitely more than one server so there's gotta be a sync protocol in there somewhere. So, technically speaking, no restrictions there. Practically speaking, on the other hand, reddit likely never took user content licensing into account to begin with - they never knew they'd get all of digg a couple years in, and 2006 was a different time besides. The only way reddit would be able to release the data now would either be a specific clause (added from the start) allowing such in the TOS, or a general-purpose "we can do w/e we want with your data"; nobody thought to add the former (as noted), and of course the latter would have been roasted alive the moment it was noticed. So, me thinks licensing issues. Modmailing /r/reddit.com is the canonical way to reach the admins (for any reason), if you'd like more info.