8 ms·
What interests Reddit? A network analysis of 84M comments by 200K users
- stared 12y agoI like a lot such analyses based on the network structure (not long ago I made something similar for Stack Exchange - http://stared.github.io/tagoverflow/; http://stared.github.io/tagoverflow/; continuation of my older one https://github.com/stared/tag-graph-map-of-stackexchange/wiki https://github.com/stared/tag-graph-map-of-stackexchange/wik...). Though, technology-wise, it is one use-case where SVG beats pixel graphics, both in terms of usability and interface (whether it is custom D3.js or something graph-oriented as http://sigmajs.org/ http://sigmajs.org/).
- wamatt 12y agoif you set tag coloring to "% answered", an interesting pattern emerges http://i.imgur.com/ZLmWHrq.png http://i.imgur.com/ZLmWHrq.png responsiveness of the community in order of most to least - oldschool hacker (c/c++,bash,perl,regex) - web dev (jquery, javascript, html, css) - app dev (ios, objectivec, android, java)
- stared 12y ago^ edit: http://stared.github.io/tagoverflow/ http://stared.github.io/tagoverflow/ Link "ate" the semicolon.
- faizshah 12y agoI'm working on a project relevant to this. Does anyone know if the author has shared this data set anywhere? Or does anyone know of any data sets that could be used for developing mixture models to classify users into interest groups (like photographers, programmers etc)?
- placeybordeaux 12y agoYeah the thing I most wanted out of the post is a torrent to the data.
- faizshah 12y agoI have been working on a small shell script to get the top 500 posts of the top 5000 subreddits. The big issue is the api call limit is 60 req/min for OAuth2 tokens. It would save a lot of time if someone made a torrent of a data set like that, I will probably post the data I collect somewhere as well but it would be easier if reddit just hosted a data set like the one I described. They could probably post one on AWS public data sets. https://aws.amazon.com/datasets/ https://aws.amazon.com/datasets/
- bduerst 12y agoWhat kind of data are you pulling? Have you tried scraping the subreddit RSS feeds?
- faizshah 12y agoI haven't tried that, I am looking for comments attached to usernames attached to articles/other comments in a tree and the subreddit that they are contained within. In order to define each user as an agent who has a probabilistic membership to the set of all interest groups (subpopulations) within the data set defined by a mixture model. My pipeline right now looks like: list of most popular SFW subreddits of all time -> gnu parallel -> curl on comment listings of top articles within subreddit -> json processing with jq -> gzip -> store on filesystem So at the end I should have a directory of 5000 subreddits containing a gzipped json file of the comment threads of their top 500 posts. I'm new to this kind of data processing but this is the easiest way I have found to do a short term, one time scrape of this data.
- HarryN 12y agoCould you not just request the .json feed whenever you like without limits? ( by appending .json to the reddit URL )
- minimaxir 12y agoNote that almost all subreddits only let you travel back 1,000 posts using the .json feed. The exception is /r/all, which had infinite pagination, but the low API limits prevent much usability.
- aroch 12y agoIf I'm remembering correctly, the raw data was given under an NDA/DND as a one time only deal. There was a subreddit associated with the data and collection but its since been banned.
- minimaxir 12y agoI was the one with this data. The subreddit which discussed provided the data went private. I have the raw data but it's infeasable to distribute due to size and the original source got in trouble for making it easily accessible.
- Houshalter 12y agoThe admins released a bunch of anonymize voting data once. And several people have distributed datasets of scraped comments. Sorry I don't have links handy. Check through /r/redditdev
- faizshah 12y agoThanks for the tip, I found the link you're referring to: https://github.com/umbrae/reddit-top-2.5-million https://github.com/umbrae/reddit-top-2.5-million
- stared 12y agoThe author released data, links are at the bottom of the article (great thanks to him!).
- faizshah 12y agoThanks for the tip, I just started downloading the data.
- fspacef 12y agoSalute the effort put into this, quite thought provoking
- okasaki 12y agoMaybe I'm just stupid, but I don't see anything thought provoking. In fact I feel that a better way to see what redditors are interested in would be to just find (there may even be stats on reddit on this) the ~50 most active subreddits.
- zipppy 12y agoIf the 50 most active only constituted 10% of all reddit activity, though, they wouldn't necessarily paint an accurate picture of all of reddit. Maybe nothing could paint that picture, but if there are themes prevalent independent of the subreddit topics themselves, this kind of analysis could shed light on them.
- _sword 12y agoIt's fascinating to see the words connected between men and women in this network analysis. At first glance a portion of it looks to be MRA buzzwords and / or the typical circlejerk reddit reaction to feminism.
- deleted 12y ago[deleted]
- 6stringmerc 12y agoI get the feeling that Conde Nast may not like this type of approach when they're not directly profiting from it. A study of the language between the SFW and NSFW type tags might be pretty interesting, or, well, not very pleasant. I did participate in a couple music communities for a while, but there's something in the stew over there that I'm glad I closed my account and never looked back. YMMV.
- brandonwamboldt 12y agoContrary to popular belief, Condé Nast no longer owns Reddit. Since 2012, Reddit operates as an independent company (Advanced Publications, the parent company of Condé Nast is a majority share holder though). See: http://www.redditblog.com/2013/08/reddit-myth-busters_6.html#independent-reddit-inc http://www.redditblog.com/2013/08/reddit-myth-busters_6.html...
- thieving_magpie 12y agoSo they don't own it, but they are the majority share holder? That doesn't feel very independent. Maybe I'm misunderstanding.
- brandonwamboldt 12y agoYou are indeed misunderstanding. Reddit is an independent entity, not a subsidiary of Condé Nast (like it used to be) and not a subsidiary of Advanced Publications (like it used to be). It is an independent corporation, with it's own board of directors, and control of its own finances. Just being a majority stakeholder doesn't mean you control the company either. There are a lot of details like share types and company by-laws that determine that.
- bhayden 12y agoIt is probably safe to say Condé Nast has a huge influence in who the board of directors are, and therefore controls reddit still.
- grabcocque 12y agoMisogyny seems to be big in Reddit comments.
- seany 12y agoYou misspelled misandry.
- sliverstorm 12y agoI'm thinking misanthropy is more accurate
- hooo 12y agoI find these network visualizations nice to look at, but not all that insightful. They're generally hard to read and track relations outside of the main clusters. Am I missing something?
- th0ma5 12y agoNo I don't think so. A lot of people call these things "hairballs," and probably a more useful interface would be some kind of faceted browser that allows you to do pivots and look at aggregate stats of the various lenses you can put on top of a graph. Additionally, measurements such as node separation, "betweenness," or perhaps even looking at common chain patterns are probably more useful ways of trying to dissect graph structures.
- Chronic31 12y agoLet's say you compute how distant/similar two concepts are. Then what? You update a link on Wikipedia?
- th0ma5 12y agoYeah I don't know! :D I guess I was talking about graph processing in general: https://en.wikipedia.org/wiki/Betweenness_centrality https://en.wikipedia.org/wiki/Betweenness_centrality
- Real_S 12y agoYou might enjoy mapequation.org The methods used there are awesome! Also, open source with a great website.
- SwellJoe 12y agoWhat interests reddit? Casual racism and misogyny. Also cats. Seriously though, it's interesting how interconnected some things can be in this view. I'm not sure what sense I can make of those interconnections, though. Mousing around, while being a very frequent redditor (so my own neural network is making connections based on experience), I can kinda infer order out of things like the "government->state" topics connected to "force" and "property" among others (hints at the libertarian-leaning general population), and the "women" topic connecting to a whole host of stuff...the cyan colored section off to the top right might even kinda hint at the casual misogyny thing (which was a "ha ha only serious" kind of joke), with words like "bullshit", "logic", "proof", "assumption", "reasoning", and "evidence", being connected to "women" but not to "men". But, without having spent years on reddit, and without my particular flavor of reddit (the subs I'm subscribed to), maybe I'd interpret the data very differently. I never quite know how to interpret network graphs like this, honestly, short of for things that are networks. i.e. a computer network topology on a graph shows useful data...the hops from one machine to the next. When connecting up one word to the next, it seems difficult to draw meaningful conclusions. Like my interpretation of the meaning of "government->state->property" as being a hint at the libertarian leanings of many subreddits, or the connection of "women->reasoning->evidence" as being a hint of many redditors belief that women are illogical liars (which is the impression many of my female friends have of reddit, in general, particularly when topics like date rape or the "friend zone" come up). Is that actually the context in which these connections are made? I wouldn't really know how to check. It'd be cool to be able to drill down to conversations in which the connections where made, but presenting that in a coherent UI seems challenging.
- thejaredhooper 12y agoI agree. It would be nice to drill down into the data in order to further analyze everything. I also feel there was a particular sort of censorship in the dataset, an indicator of which was the explicit racist and misogynistic words that were absent. There was a large lack of swears and bad terms in this analysis (bitch being a particularly obvious cut) and I, for one, see examples of these slurs prevalently used by young men far too often on the site. Perhaps the data was tailored when it was provided to the analyst, or it was censored after reception, but this felt too "PG-13" for an analysis of reddit's "interests".
- jedberg 12y agoFun fact: We did this exact analysis at reddit many years ago, and used it to figure out which subreddits were related to each other. We never got around to productizing it, unfortunately, but the idea was to use it to suggest new reddits to you.
- sinemetu11 12y agoI guess this might get into some special sauce territory, but was there a specific reason why this type of recommendation system was deprecated?
- Houshalter 12y agoAll the reddit subreddit recommenders I've seen produce garbage recommendations. Outside of a handful of popular, general subreddits which everyone already knows about, everything is niche special interest stuff that you need to find on your own.
- bduerst 12y agoSubreddit specificity is so messily complex that it would be very difficult to do any recommendations based on your own subscriptions. Without reddit's cooperation in categorization (unlikely) it's probably not going to happen.
- jedberg 12y agoAs others already said below, the feedback loop was dangerous. Also it took a lot of resources to calculate and we just didn't have the time to build efficient map/reduce jobs to do it regularly. It was done by hand in Mathmatica.
- swalsh 12y agoOne potential downside of using an algorithm like that is the possibility of a feedback loop.
- eevilspock 12y ago
- erroneousfunk 12y agoSmall point: Is it really considered "scraping" (" I scraped approximately 84 million comments") if you used a Python library that uses the Reddit API, not the actual site directly?