11 ms·
A dataset of every Reddit comment
TLDR; magnet link:
magnet:?xt=urn:btih:7690f71ea949b868080401c749e878f98de34d3d&dn=reddit%5Fdata&tr=http%3A%2F%2Ftracker.pushshift.io%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80
Someone has already put it on Google Big Query - https://bigquery.cloud.google.com/table/fh-bigquery:reddit_comments.2015_05
Link to original Reddit thread - https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/
- avinassh 11y agoClickables: https://bigquery.cloud.google.com/table/fh-bigquery:reddit_comments.2015_05 https://bigquery.cloud.google.com/table/fh-bigquery:reddit_c... https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/ https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_eve...
- avinassh 11y agoSome insights: Relationship between Reddit Comment Score and Comment Length for 1.66 Billion Comments [0] and the Github repo [1]. Reddit cliques N°2 - deeper into the subs [2] [0] - https://www.reddit.com/r/dataisbeautiful/comments/3cjyvb/relationship_between_reddit_comment_score_and/ https://www.reddit.com/r/dataisbeautiful/comments/3cjyvb/rel... [1] - https://github.com/minimaxir/reddit-comment-length https://github.com/minimaxir/reddit-comment-length [2] - https://www.reddit.com/r/dataisbeautiful/comments/3cofju/reddit_cliques_n2_deeper_into_the_subs_oc/ https://www.reddit.com/r/dataisbeautiful/comments/3cofju/red...
- deleted 11y ago[deleted]
- akhilcacharya 11y agoThis should be pretty great for research - the Reddit API is actually fairly limited in my experience, I wasn't even able to get my full comment history last time I tried.
- tspike 11y agoThis could be amazing as input to a question answering engine.
- aluhut 11y agoI would love to feed my megahal with that.
- douche 11y agoI think Reddit was one of the places they had to wall off Watson from data-mining, because it devolved into foul-mouthed memes. I wish they hadn't, nothing would be better than a billion dollar piece of technology deciding to reenact Sean Connery Jeopardy skits, on Jeopardy. [2001 HAL voice]: I have a question about the penis mightier. Does it work?
- Houshalter 11y agoI actually built an IRC bot that did this once. It searched reddit for your question, took the first result, and posted the top comment. It worked very well for certain kinds of questions. Especially if it came from the better subreddits like askscience, but even just reddit in general. I improved on it a lot with a whitelist of subreddits and some machine learning to select the best thread. But I was only touching on what is possible with that data. The scope of the discussions on reddit is huge. Despite a lot of jokes in the comments here, the quality of the comments on average, is pretty decent. And the metadata like subreddit and score are extremely useful for filtering it down more.
- jesrui 11y agoThere was also someone that created a dump of all HN data a while ago: https://github.com/sytelus/HackerNewsData https://github.com/sytelus/HackerNewsData
- minimaxir 11y agoI have a GitHub repository for getting all the HN data the hard way using Python and the Algolia API, and storing them in PostgreSQL: https://github.com/minimaxir/get-all-hacker-news-submissions-comments https://github.com/minimaxir/get-all-hacker-news-submissions...
- fit2rule 11y agoAnyone know how big it is yet?
- iaw 11y agoThis is an awesome dump, what types of analysis could we use this for? tspike mentioned a QA engine which would be awesome, what else can people think of?
- roflchoppa 11y agoIf anyone is working with the data, we should see how many users actually hivemind comment on things vs OC
- minimaxir 11y agoYou can't determine what is OC from comments, only from submission titles. But by sheer coincidence, I have made a chart comparing average scores for OC submissions vs. non-OC submissions a few months ago: https://www.reddit.com/r/dataisbeautiful/comments/2rv76z/oc_reddit_submissions_receive_almost_double_the/ https://www.reddit.com/r/dataisbeautiful/comments/2rv76z/oc_...
- stuck_in_the_ma 11y agoHey there! I know you! Posts are coming soon!
- minimaxir 11y agoGood to hear! :)
- thomasfromcdnjs 11y agoSomeone should try make a sentient twitter bot that learns from the Reddit data
- avinassh 11y agoHow does one go about doing that? I know how to program, but I have written anything which do 'intelligent' stuff.
- thomasfromcdnjs 11y agoI have no idea either, I would also love for someone to chime in with some first steps.
- Ianvdl 11y agoMarkov chains would probably be a good start. See reddit.com/r/subredditsimulator (not always safe for work)
- deleted 11y ago[deleted]
- jesrui 11y ago"The Analytics Edge" from edx [1], which is running now, could interest you. Great MOOC by the way. There is even a lecture around this idea [2]. [1] https://www.edx.org/course/analytics-edge-mitx-15-071x-0 https://www.edx.org/course/analytics-edge-mitx-15-071x-0 [2] https://www.youtube.com/watch?v=JzaoJZNCVWA https://www.youtube.com/watch?v=JzaoJZNCVWA
- sneak 11y agoPeople have been trying to make sentient bots of any kind for some time now.
- shiggerino 11y agoI'd be extremely impressed with a somewhat general AI at the insect or nematode level, let alone human.
- minimaxir 11y agoAs linked below, I've played around a bit with this dataset: https://www.reddit.com/r/dataisbeautiful/comments/3cjyvb/relationship_between_reddit_comment_score_and/ https://www.reddit.com/r/dataisbeautiful/comments/3cjyvb/rel... BigQuery is the best interface for it. Can resolve queries on the entire dataset in less than a few seconds (however, you only get 1TB processing free per month. Since the full dataset is ~285GB, you only get 4 queries per month. Plan ahead on the May 2015 dataset, which is only 8GB.) I can answer any other questions that people have.
- daave 11y ago> Since the full dataset is ~285GB, you only get 4 queries per month. That's only true if your 4 queries need to read every single column. One of the big advantages of BigQuery's column-oriented storage is that you only pay to read the columns that are actually needed to answer your query. For example, this query to extract the top 10 authors only cost me 19GB to run (and took 7.0s): SELECT author, COUNT(*) AS COUNT FROM TABLE_QUERY([fh-bigquery:reddit_comments], "table_id CONTAINS '20' AND LENGTH(table_id)<8") GROUP EACH BY author ORDER BY COUNT DESC LIMIT 10; author,COUNT [deleted],228425822 AutoModerator,3677774 conspirobot,575576 ModerationLog,547671 autowikibot,402076 PoliticBot,388395 imgurtranscriber,360248 dogetipbot,358093 qkme_transcriber,301968 TweetPoster,293309
- minimaxir 11y agoGood point. Although if you're doing analysis on the body column itself, it'll use the majority proportion of the data set, of course.
- ilyaeck 11y agoIs this data in a format that allows to recover threads? I.e., comment X was a response to comment Y ?
- minimaxir 11y agoparent_id is a column, so you could recreate hierarchy by JOINING a table on itself.
- nathan_f77 11y agoI think it's fascinating that if anyone uses this dataset to train an AI, some tiny piece of my personality might make a contribution.
- Widdershin 11y agoYou might enjoy http://reddit.com/r/subredditsimulator http://reddit.com/r/subredditsimulator, a markov chain powered simulation of Reddit, fueled by all the different subreddits.
- kristopolous 11y agoIt's going in my /raid/datasets with about 1.4TB of other comment data. So the answer is going to be "yes" ... well probably by Tuesday morning it will be. I've used older reddit dumps before ... it's not great data for most things, but there's an awful lot of it!
- zo1 11y agoWhat are you going to use it for? If you don't mind letting us in on some details, however vague albeit interesting.
- kristopolous 11y agosure. http://frustrometer.com http://frustrometer.com
- zo1 11y agoInteresting, thanks!
- danso 11y agoNot trying to throw cold water on this...but are the Reddit execs OK with this? I mean, Twitter and Facebook would most likely issue takedowns for similar kinds of data dumps...but is there something in the Reddit API TOS that says this is OK?...I wouldn't be surprised if there a fairly liberal license, as that would be aligned with the early spirit of the site. And an onerous TOS would have likely curbed the active user-run bot ecosystem that helps manage and monitor Reddit's discussions.
- hackcasual 11y agoI believe reddit comments are made under a CC license
- mattrepl 11y agoClosest thing to an answer I've found is from the licensing page: https://www.reddit.com/wiki/licensing https://www.reddit.com/wiki/licensing But it doesn't specifically address collection and distribution of API results as a dataset. (From the page:) A licensing agreement is required in order to: * use the reddit API for commercial purposes. Use of the API is considered "commercial" if you are earning money from it, including via in-app advertising or in-app purchases. Open source use is generally considered non-commercial. * use the reddit alien logo ("snoo") in your app or for its thumbnail. Any new apps you create must be approved as well before usage. The circular "r" logo is reserved solely for use by reddit, Inc. * allow users to subscribe to reddit gold via in-app purchases. If your platform allows for it, we encourage you to work with us to make this happen. We see gold purchases as a way for you to help reddit and to give back to the reddit community.
- curiousjorge 11y agohow to access the BigQuery data? I get this screen Welcome to BigQuery! What is BigQuery? but it doesn't show the data.
- minimaxir 11y agoYou'll need to set up a project in BigQuery first. (you don't have to give billing information to use the free quota)
- curiousjorge 11y agogot it running! I just did a query for my reddit handle and it took 6.5 seconds to retrieve all of my comments. Kind of a snowden moment but this is super interesting, first time I played with Big Query. Would love to run some google API for sentiment analysis.
- TTPrograms 11y agoThis would be a really amazing way to make factually backed statements about the nature of Reddit for news sources given the recent publicity. I.e. the frequency of comments of a certain nature, typical karma scores for those comments, breakdown by subreddit etc.
- Lerc 11y agoYou have to be careful determining the nature of a certain comment algorithmically. I saw one analysis of subreddits that checked for negativity. /r/PathOfExile came in as one of the most negative subreddits, which could easily be turned into a narrative about gaming culture. To a person familiar with the context it seems far more likely that game concepts were skewing the data. Discussions involved killing, physical damage, life leech, killing, dying etc. Not to mention creature names themselves Devourers, Plummeting Ursa, Miscreations,
- mbell 11y agoWow, that is much, much, smaller than I thought it would be. Anyone have a recent link on Reddit's infrastructure? Given the small size I would think it easily fits in memory so I'm a bit curious how they handle it.
- sdrothrock 11y agoThis is the newest (2010) one I know of, and probably the one everyone's seen (I think it was HN back when it was first posted: http://highscalability.com/blog/2010/5/17/7-lessons-learned-while-building-reddit-to-270-million-page.html http://highscalability.com/blog/2010/5/17/7-lessons-learned-...
- Houshalter 11y agoI really want to get my full commenting history from this, but the download is enormous. Anyone know how I could go about this with cloud services?
- stuck_in_the_ma 11y agoHey there! I'm the one that uploaded the original dataset. I'm creating API endpoints that will easily allow you to do this. It should be completed in 1-2 weeks.
- stuck_in_the_ma 11y agoI just realized my username on here is ... oh god.
- x0 11y agoBeautiful!
- ytdht 11y agohere it is, only selected the body of your comments though, out of 4005 comments... http://s000.tinyupload.com/?file_id=09238386475544637092 http://s000.tinyupload.com/?file_id=09238386475544637092 here is the query that I used: SELECT author, body FROM [fh-bigquery:reddit_comments.2007], [fh-bigquery:reddit_comments.2008], [fh-bigquery:reddit_comments.2009], [fh-bigquery:reddit_comments.2010], [fh-bigquery:reddit_comments.2011], [fh-bigquery:reddit_comments.2012], [fh-bigquery:reddit_comments.2013], [fh-bigquery:reddit_comments.2014], [fh-bigquery:reddit_comments.2015_01], [fh-bigquery:reddit_comments.2015_02], [fh-bigquery:reddit_comments.2015_03], [fh-bigquery:reddit_comments.2015_04], [fh-bigquery:reddit_comments.2015_05] WHERE author = 'Houshalter' this is where you can run another query: https://bigquery.cloud.google.com/table/fh-bigquery:reddit_comments.2015_05 https://bigquery.cloud.google.com/table/fh-bigquery:reddit_c... I did the same for my account and some comments are definitely missing... The first one that I cross-referenced is missing (5 month old and 1 point comment). But I have other comments with only 1 point and also 5 months old and those are there.
- alexggordon 11y agoI used this[0] query to find the top ten[1] most downvoted comments of all time on Reddit. The most downvoted comment[2] is ironically in iAMA, by a mod of iAMA (ironic because of the recent drama). I'd find the top ten most upvoted, but I ran out of free bandwidth on BigQuery :(. [0] https://gist.github.com/alexggordon/7b56353dcf8044a7a5f9 https://gist.github.com/alexggordon/7b56353dcf8044a7a5f9 [1] https://drive.google.com/file/d/0Bzxo-UKxFmN-eWticy1BR2tCRDQ/view?usp=sharing https://drive.google.com/file/d/0Bzxo-UKxFmN-eWticy1BR2tCRDQ... [2] https://www.reddit.com/r/IAmA/comments/s5guk/iam_bad_luck_brian_ama/c4b8m3u https://www.reddit.com/r/IAmA/comments/s5guk/iam_bad_luck_br...
- whoopdedo 11y agoCould this be used to model the lifecycle of a Reddit post? What makes one thread get more votes and be replied to more often versus another similar thread posted at the same time. Is there a time when a comment is more likely to be voted for, based on wall clock or relative to the start of the thread. What topics are more likely to receive votes than others. Is there a minimum time between when reposts can be posted and still get upvotes. What are the most reposted things on Reddit. Why do people keep voting for reposts.
- linkmotif 11y agoDoes anyone know if this includes the spam? This dataset is obviously very interesting in its own right without spam, but far more useful for me would be all of it, including the spam. EDIT: regardless, does anyone know if similar datasets with spam?
- jedberg 11y agoIf they grabbed it via the API then no, it doesn't. It also won't include a lot of older stuff which is no longer accessible through the API.
- colordrops 11y agoDoes this data include private subreddits?
- visarga 11y agoI grep-ed my own comments in it and only shows 39 out of thousands. Is it really complete?
- ers35 11y agoI wanted the data in a form I could easily query, so I wrote a program to convert the JSON to an SQLite database: https://gist.github.com/ers35/3b615a75fa0ed5e6d5cc https://gist.github.com/ers35/3b615a75fa0ed5e6d5cc I have the program running on Amazon EC2 right now converting the whole dataset. I plan to upload the database to the Internet Archive when it completes.
- justwannasing 11y agoSince 80% of all Reddit comments are vitriol, nonsense by 15-year old kids, why would anyone do this?
- ska 11y agoHow does complaining about the SNR on reddit in a way that decreases the SNR here help anyone?
- joelthelion 11y agoDoes someone have the equivalent for reddit links (with associated title)?
- avinashgupta 11y agoits good