4 ms·
As linked below, I've played around a bit with this dataset: https://www.reddit.com/r/dataisbeautiful/comments/3cjyvb/relationship_between_reddit_comment_score_
by minimaxir 11y ago
As linked below, I've played around a bit with this dataset: https://www.reddit.com/r/dataisbeautiful/comments/3cjyvb/relationship_between_reddit_comment_score_and/ https://www.reddit.com/r/dataisbeautiful/comments/3cjyvb/rel...
BigQuery is the best interface for it. Can resolve queries on the entire dataset in less than a few seconds (however, you only get 1TB processing free per month. Since the full dataset is ~285GB, you only get 4 queries per month. Plan ahead on the May 2015 dataset, which is only 8GB.)
I can answer any other questions that people have.
- daave 11y ago> Since the full dataset is ~285GB, you only get 4 queries per month. That's only true if your 4 queries need to read every single column. One of the big advantages of BigQuery's column-oriented storage is that you only pay to read the columns that are actually needed to answer your query. For example, this query to extract the top 10 authors only cost me 19GB to run (and took 7.0s): SELECT author, COUNT(*) AS COUNT FROM TABLE_QUERY([fh-bigquery:reddit_comments], "table_id CONTAINS '20' AND LENGTH(table_id)<8") GROUP EACH BY author ORDER BY COUNT DESC LIMIT 10; author,COUNT [deleted],228425822 AutoModerator,3677774 conspirobot,575576 ModerationLog,547671 autowikibot,402076 PoliticBot,388395 imgurtranscriber,360248 dogetipbot,358093 qkme_transcriber,301968 TweetPoster,293309
- minimaxir 11y agoGood point. Although if you're doing analysis on the body column itself, it'll use the majority proportion of the data set, of course.
- ilyaeck 11y agoIs this data in a format that allows to recover threads? I.e., comment X was a response to comment Y ?
- minimaxir 11y agoparent_id is a column, so you could recreate hierarchy by JOINING a table on itself.
- deleted 11y ago[deleted]
- rcpt 11y agoI want all the url submissions in a given subreddit, but all I can find in the tables is "link_id". How do I map link_ids to urls?
- minimaxir 11y agoThere is no submission data yet. /u/Stuck_in_the_Matrix commented on another thread that he's working on it.