6 ms·

This is a gorgeous post and I shiver at the thought of how much work went into this! Thoughtful, detailed and chock full of great visualizations of the data. I'

by phillmv 12y ago

This is a gorgeous post and I shiver at the thought of how much work went into this! Thoughtful, detailed and chock full of great visualizations of the data. I'd be interested in a similar analysis of HN, as I'm interested in the editorial intent thus revealed. On tangential note, though, I was a bit surprised by the hypothesis. 'Meritocracy' to begin with is a dubious fiction, but especially so when mapped on to the vote distribution of a given post in a semi-decentralized human-moderated environment that is constantly being gamed. "Merit" just seems like a big value judgement over such a noisy channel.

minimaxir 12y ago
A note about the graphics: all the charts were made with R/ggplot2. Not only ggplot2, but basic ggplot2: the charts in the blog post can be made with about 5 lines of code maximum each. You can customize ggplot2 for even more beautiful charts, such as the ones used in my analysis of HN comments: http://minimaxir.com/2014/10/hn-comments-about-comments/ http://minimaxir.com/2014/10/hn-comments-about-comments/ Note that reproducing this analysis for the HN front page is harder since there is a lot that happens behind the scenes that cannot be generalized. Here's an attempt at performing such an analysis (note dang's comments): https://news.ycombinator.com/item?id=8533757 https://news.ycombinator.com/item?id=8533757
- mikecb 12y ago
  My favorite example of ggplots ease and customizability comes from xkcd style graphs: http://stackoverflow.com/questions/12675147/how-can-we-make-xkcd-style-graphs-in-r http://stackoverflow.com/questions/12675147/how-can-we-make-...
- beambot 12y ago
  Not all the charts are ggplot2. It looks like the big interactive one is javascript, specifically HighCharts.
- jedmeyers 12y ago
  By the way, there is an awesome course track on Data Science from Johns Hopkins on Coursera right now, which includes introduction to R. https://www.coursera.org/specialization/jhudatascience/1?utm_source=spark&utm_medium=banner https://www.coursera.org/specialization/jhudatascience/1?utm...
- pit 12y ago
  Thanks! I love those chunky graphs, definitely going to play around with ggplot2 a bit.
- 12423gsd 12y ago
  I really enjoy using ggplot2, but the basic "theme" is terrible. - They tried too hard to do a Tuft* - but it's actually way too flashy. - Basically the issue is that it's not generic enough. Whenever you see a ggplot2 plot it's always screaming at you "I WAS MADE IN ggplot2!!" Just KISS... - The grey background is completely unusable if you want to print out your charts! - The default pastel colors (ie. PowerPoint 2012) are always a disaster for readability. In meetings people constantly can't tell them apart and they are definitely not colorblind friendly. It' a shame b/c each script I write ends up having to have an obnoxiously long theme declaration to make it look "normal" * I think most people miss the Tuft's point. He presents a way about thinking about data presentation. Instead people just look at it and think "oooo! That looks pretty. Let me copy what he did". The guy has his own personal style - it's definitely nice, but the point isn't to copy it. EDIT: For those thinking about learning ggplot2.. maybe wait off for a bit. It seems like it'll be deprecated soon and replaces with ggvis.
  - minimaxir 12y ago
    NB: Here is a paper the derivation of the ggplot styles: http://vita.had.co.nz/papers/layered-grammar.pdf http://vita.had.co.nz/papers/layered-grammar.pdf I would recommend using theme_bw(), which helps solve some of the problem with the gray background. EDIT: This is ggvis: http://blog.rstudio.org/2014/06/23/introducing-ggvis/ http://blog.rstudio.org/2014/06/23/introducing-ggvis/ This is news to me, so I'll look into it to see how it differs from ggplot2. EDIT 2: It seems more like that the difference is that ggvis is more for interactive charts, but then it requires a dependency on Shiny, which is not optimal for blog posts.
    - 12423gsd 12y ago
      Thanks for linking the paper! I'll definitely read over that. I've heard it mentioned before ... I'm definitely not an R guru or "in" on the latest news, but it seems that Hadley Wickham (who probably single handedly is the reason R is still relevant) now works for the RStudio guys and he's reworking his tools. plyr is now dplyr and gpplot2 is now ggvis. And there also another tool called tinyr. My understanding is that they're still in development, but they'll ultimately provide an "integrated" ecosystem for processing data.frames (with hooks into the RStudio IDE) He talks about it at the beginning of this video https://www.youtube.com/watch?v=wki0BqlztCo https://www.youtube.com/watch?v=wki0BqlztCo
gus_massa 12y ago
A deep analysis was published in http://www.righto.com/2013/11/how-hacker-news-ranking-really-works.html http://www.righto.com/2013/11/how-hacker-news-ranking-really... (HN discussion: https://news.ycombinator.com/item?id=6799854 https://news.ycombinator.com/item?id=6799854 (920 points, 345 days ago, 190 comments)). AFAIK there are no filters to bind the maximal post of a domain in the front page, but some domains have automatic penalties.
- 001sky 12y ago
  that (referenced) analysis is out of date
moron4hire 12y ago
It appears he just meant to say "democratic".
stcredzero 12y ago
I'd be interested in a similar analysis of HN, as I'm interested in the editorial intent thus revealed. HN no longer functions entirely off of the "reddit algorithm." I know that it's now curated to some extent. "Merit" just seems like a big value judgement over such a noisy channel. I think the takeaway from such investigations -- and all of human history -- is that the system will be gamed. To survive, systems must be robust in the face of corruption and manipulation.
- dang 12y ago
  > it's now curated to some extent It was before, too. We just explain it more now.
hiou 12y ago
I like as describe Reddit and HN to be voter nominated, not voter selected. Votes nominate stories, but then the algorithm decides which ones to pick from that pile.
- tedks 12y ago
  And for HN, the "algorithm" is the subjective judgment of the moderation staff, as witnessed by countless articles on the misogyny in tech (as an example) getting axed from the front page.
  - buckbova 12y ago
    Or maybe those subjective articles on misogyny in tech are flagged by users like me.
  - dang 12y ago
    The biggest factor there by far is user flags.
    - tedks 12y ago
      This is a nice way to punt, but you've (in the plural sense, not you specifically) have still admitted to removing them from the frontpage.

by phillmv 12y ago

minimaxir 12y ago

A note about the graphics: all the charts were made with R/ggplot2. Not only ggplot2, but basic ggplot2: the charts in the blog post can be made with about 5 lines of code maximum each. You can customize ggplot2 for even more beautiful charts, such as the ones used in my analysis of HN comments: http://minimaxir.com/2014/10/hn-comments-about-comments/ http://minimaxir.com/2014/10/hn-comments-about-comments/ Note that reproducing this analysis for the HN front page is harder since there is a lot that happens behind the scenes that cannot be generalized. Here's an attempt at performing such an analysis (note dang's comments): https://news.ycombinator.com/item?id=8533757 https://news.ycombinator.com/item?id=8533757

mikecb 12y ago
My favorite example of ggplots ease and customizability comes from xkcd style graphs: http://stackoverflow.com/questions/12675147/how-can-we-make-xkcd-style-graphs-in-r http://stackoverflow.com/questions/12675147/how-can-we-make-...
beambot 12y ago
Not all the charts are ggplot2. It looks like the big interactive one is javascript, specifically HighCharts.
jedmeyers 12y ago
By the way, there is an awesome course track on Data Science from Johns Hopkins on Coursera right now, which includes introduction to R. https://www.coursera.org/specialization/jhudatascience/1?utm_source=spark&utm_medium=banner https://www.coursera.org/specialization/jhudatascience/1?utm...
pit 12y ago
Thanks! I love those chunky graphs, definitely going to play around with ggplot2 a bit.
12423gsd 12y ago
I really enjoy using ggplot2, but the basic "theme" is terrible. - They tried too hard to do a Tuft* - but it's actually way too flashy. - Basically the issue is that it's not generic enough. Whenever you see a ggplot2 plot it's always screaming at you "I WAS MADE IN ggplot2!!" Just KISS... - The grey background is completely unusable if you want to print out your charts! - The default pastel colors (ie. PowerPoint 2012) are always a disaster for readability. In meetings people constantly can't tell them apart and they are definitely not colorblind friendly. It' a shame b/c each script I write ends up having to have an obnoxiously long theme declaration to make it look "normal" * I think most people miss the Tuft's point. He presents a way about thinking about data presentation. Instead people just look at it and think "oooo! That looks pretty. Let me copy what he did". The guy has his own personal style - it's definitely nice, but the point isn't to copy it. EDIT: For those thinking about learning ggplot2.. maybe wait off for a bit. It seems like it'll be deprecated soon and replaces with ggvis.
- minimaxir 12y ago
  NB: Here is a paper the derivation of the ggplot styles: http://vita.had.co.nz/papers/layered-grammar.pdf http://vita.had.co.nz/papers/layered-grammar.pdf I would recommend using theme_bw(), which helps solve some of the problem with the gray background. EDIT: This is ggvis: http://blog.rstudio.org/2014/06/23/introducing-ggvis/ http://blog.rstudio.org/2014/06/23/introducing-ggvis/ This is news to me, so I'll look into it to see how it differs from ggplot2. EDIT 2: It seems more like that the difference is that ggvis is more for interactive charts, but then it requires a dependency on Shiny, which is not optimal for blog posts.
  - 12423gsd 12y ago
    Thanks for linking the paper! I'll definitely read over that. I've heard it mentioned before ... I'm definitely not an R guru or "in" on the latest news, but it seems that Hadley Wickham (who probably single handedly is the reason R is still relevant) now works for the RStudio guys and he's reworking his tools. plyr is now dplyr and gpplot2 is now ggvis. And there also another tool called tinyr. My understanding is that they're still in development, but they'll ultimately provide an "integrated" ecosystem for processing data.frames (with hooks into the RStudio IDE) He talks about it at the beginning of this video https://www.youtube.com/watch?v=wki0BqlztCo https://www.youtube.com/watch?v=wki0BqlztCo

gus_massa 12y ago

A deep analysis was published in http://www.righto.com/2013/11/how-hacker-news-ranking-really-works.html http://www.righto.com/2013/11/how-hacker-news-ranking-really... (HN discussion: https://news.ycombinator.com/item?id=6799854 https://news.ycombinator.com/item?id=6799854 (920 points, 345 days ago, 190 comments)). AFAIK there are no filters to bind the maximal post of a domain in the front page, but some domains have automatic penalties.

001sky 12y ago
that (referenced) analysis is out of date

moron4hire 12y ago

It appears he just meant to say "democratic".

stcredzero 12y ago

I'd be interested in a similar analysis of HN, as I'm interested in the editorial intent thus revealed. HN no longer functions entirely off of the "reddit algorithm." I know that it's now curated to some extent. "Merit" just seems like a big value judgement over such a noisy channel. I think the takeaway from such investigations -- and all of human history -- is that the system will be gamed. To survive, systems must be robust in the face of corruption and manipulation.

dang 12y ago
> it's now curated to some extent It was before, too. We just explain it more now.

hiou 12y ago

I like as describe Reddit and HN to be voter nominated, not voter selected. Votes nominate stories, but then the algorithm decides which ones to pick from that pile.

tedks 12y ago
And for HN, the "algorithm" is the subjective judgment of the moderation staff, as witnessed by countless articles on the misogyny in tech (as an example) getting axed from the front page.
- buckbova 12y ago
  Or maybe those subjective articles on misogyny in tech are flagged by users like me.
- dang 12y ago
  The biggest factor there by far is user flags.
  - tedks 12y ago
    This is a nice way to punt, but you've (in the plural sense, not you specifically) have still admitted to removing them from the frontpage.