7 ms·

As a reference the archive of all Reddit comments from October 2007 to May 2015 is around 1 terabyte uncompressed. You could do exhaustive analysis on that dat

As a reference the archive of all Reddit comments from October 2007 to May 2015 is around 1 terabyte uncompressed. You could do exhaustive analysis on that dataset fully in memory.

jedberg 10y ago
Your point is accurate, but I'd like to point out that the dataset isn't actually all the comments on Reddit -- it's inly what they could scrape, which is limited to 1000 comments per account. So basically it's missing a lot of the historical comments of the oldest accounts. I only point this out to try and correct a common error I see. You're absolutely right that it is awesome that the entire data set can be analyzed in RAM!
- samstave 10y ago
  It would be interested to see the distribution of 1000 comments from each account over the period of 12 months. Some people go dormant - like vacation, or depression, or lack of interest in topics - then cluster a bunch of comments when, say, they are on a drunken rage binge. What time of day the accounts most frequently comment. (I'd bet there is an interesting grouping of those that post while at work during the day, and those who post from home at night. or what subreddits people comment in most during the day vs which /r/ they post to at night ;)
  - MasterScrat 10y ago
    Some of those have already been done. Sounds like you'd enjoy reading this guy's stuff: https://www.reddit.com/user/minimaxir/submitted/?sort=top https://www.reddit.com/user/minimaxir/submitted/?sort=top
    - samstave 10y ago
      Love it thanks!
- MasterScrat 10y ago
  Are you sure? The dataset is from here: https://archive.org/details/2015_reddit_comments_corpus https://archive.org/details/2015_reddit_comments_corpus Looking at the thread from the release, I see no explanation of how he got the data, but I see several people commenting that they finally have a way to get comments beyond the 1000 per account: https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/ https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_eve...
  - jedberg 10y ago
    It looks like they got clever, so what they are doing is getting 1000 links from each subreddit, indexing the comments and linking it back to the user. So yeah, they did figure out how to got back more than 1000 comments for some users, but they'll still be limited by 1000 links per reddit per sort. So for a very active reddit, you can't get really old comments.
    - _wmd 10y ago
      IIRC you can use the search API to enumerate 1000 at a time, using a "created_at>=..." restriction to move the cursor starting position. I had to poke in the code to find out how to talk Amazon's search syntax directly
      - jedberg 10y ago
        Yeah, that might work too. That would be pretty clever.
    - MasterScrat 10y ago
      What about an official data dump, like StackExchange does? Wouldn't that be beneficial to all parties? Are there legal restrictions?
      - i336_ 10y ago
        I haven't poured over the code, but reddit is most definitely more than one server so there's gotta be a sync protocol in there somewhere. So, technically speaking, no restrictions there. Practically speaking, on the other hand, reddit likely never took user content licensing into account to begin with - they never knew they'd get all of digg a couple years in, and 2006 was a different time besides. The only way reddit would be able to release the data now would either be a specific clause (added from the start) allowing such in the TOS, or a general-purpose "we can do w/e we want with your data"; nobody thought to add the former (as noted), and of course the latter would have been roasted alive the moment it was noticed. So, me thinks licensing issues. Modmailing /r/reddit.com is the canonical way to reach the admins (for any reason), if you'd like more info.
ers35 10y ago
You may be interested in an SQLite version of the dataset that is 553 GB vs. the 908 GB JSON: https://archive.org/details/2015_reddit_comments_corpus_sqlite https://archive.org/details/2015_reddit_comments_corpus_sqli... The storage format of a dataset can make a big difference in memory usage.
flamedoge 10y ago
I would like to know how much of that is memes and shitposts

As a reference the archive of all Reddit comments from October 2007 to May 2015 is around 1 terabyte uncompressed. You could do exhaustive analysis on that dataset fully in memory.

jedberg 10y ago

Your point is accurate, but I'd like to point out that the dataset isn't actually all the comments on Reddit -- it's inly what they could scrape, which is limited to 1000 comments per account. So basically it's missing a lot of the historical comments of the oldest accounts. I only point this out to try and correct a common error I see. You're absolutely right that it is awesome that the entire data set can be analyzed in RAM!

samstave 10y ago
It would be interested to see the distribution of 1000 comments from each account over the period of 12 months. Some people go dormant - like vacation, or depression, or lack of interest in topics - then cluster a bunch of comments when, say, they are on a drunken rage binge. What time of day the accounts most frequently comment. (I'd bet there is an interesting grouping of those that post while at work during the day, and those who post from home at night. or what subreddits people comment in most during the day vs which /r/ they post to at night ;)
- MasterScrat 10y ago
  Some of those have already been done. Sounds like you'd enjoy reading this guy's stuff: https://www.reddit.com/user/minimaxir/submitted/?sort=top https://www.reddit.com/user/minimaxir/submitted/?sort=top
  - samstave 10y ago
    Love it thanks!
MasterScrat 10y ago
Are you sure? The dataset is from here: https://archive.org/details/2015_reddit_comments_corpus https://archive.org/details/2015_reddit_comments_corpus Looking at the thread from the release, I see no explanation of how he got the data, but I see several people commenting that they finally have a way to get comments beyond the 1000 per account: https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/ https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_eve...
- jedberg 10y ago
  It looks like they got clever, so what they are doing is getting 1000 links from each subreddit, indexing the comments and linking it back to the user. So yeah, they did figure out how to got back more than 1000 comments for some users, but they'll still be limited by 1000 links per reddit per sort. So for a very active reddit, you can't get really old comments.
  - _wmd 10y ago
    IIRC you can use the search API to enumerate 1000 at a time, using a "created_at>=..." restriction to move the cursor starting position. I had to poke in the code to find out how to talk Amazon's search syntax directly
    - jedberg 10y ago
      Yeah, that might work too. That would be pretty clever.
  - MasterScrat 10y ago
    What about an official data dump, like StackExchange does? Wouldn't that be beneficial to all parties? Are there legal restrictions?
    - i336_ 10y ago
      I haven't poured over the code, but reddit is most definitely more than one server so there's gotta be a sync protocol in there somewhere. So, technically speaking, no restrictions there. Practically speaking, on the other hand, reddit likely never took user content licensing into account to begin with - they never knew they'd get all of digg a couple years in, and 2006 was a different time besides. The only way reddit would be able to release the data now would either be a specific clause (added from the start) allowing such in the TOS, or a general-purpose "we can do w/e we want with your data"; nobody thought to add the former (as noted), and of course the latter would have been roasted alive the moment it was noticed. So, me thinks licensing issues. Modmailing /r/reddit.com is the canonical way to reach the admins (for any reason), if you'd like more info.

ers35 10y ago

You may be interested in an SQLite version of the dataset that is 553 GB vs. the 908 GB JSON: https://archive.org/details/2015_reddit_comments_corpus_sqlite https://archive.org/details/2015_reddit_comments_corpus_sqli... The storage format of a dataset can make a big difference in memory usage.

flamedoge 10y ago

I would like to know how much of that is memes and shitposts