5 ms·

Words growing or shrinking in Hacker News titles: a tidy analysis

118 points by var_explained 9y ago

minimaxir 9y ago
Hmm, the BigQuery HN dataset is now updated daily and contains comments as well as stories? That's new, and I'll certainly give it another look at for my projects. With the bigrquery R package (https://github.com/rstats-db/bigrquery https://github.com/rstats-db/bigrquery), you can access the HN dataset directly from R, using dplyr syntax too. (for simple queries atleast; you can pass the raw SQL for complex queries) As noted, the resulting dataset of words is large, so mapping the words in BigQuery itself may be more practical (using a combo of SPLIT and UNNEST with standard SQL), although of course you can't do complex operations like logistic regression or splines there.
robteix 9y ago
I'm surprised both "NSA" and "surveillance" are two of the fastest shrinking words. I thought we saw more now than ever. Shows how perception doesn't always match reality.
- jdminhbg 9y ago
  When the Snowden leaks first dropped, the front page was absolutely overwhelmed with NSA news, to the exclusion of nearly everything else. Would not be possible to keep that level of interest up without making this is an exclusively NSA/surveillance-driven site.
  - Houshalter 9y ago
    IIRC the mods also soft banned it because of that. So posts with the word "NSA" in the title get penalized and ranked much lower than other posts. Hence the shrinking.
    - dang 9y ago
      We did that for a while but stopped already years ago. It was only needed until the story barrage leveled off.
aswanson 9y ago
A more interesting analysis would be comment length.
- minimaxir 9y ago
  In my old analysis (http://minimaxir.com/2014/10/hn-comments-about-comments/ http://minimaxir.com/2014/10/hn-comments-about-comments/), it's not that interesting. Comments are getting longer over time on average (http://minimaxir.com/img/hn-comments/monthly_average_words.png http://minimaxir.com/img/hn-comments/monthly_average_words.p...), and there is a slight positive correlation between comment score and comment length (http://minimaxir.com/img/hn-comments/distribution_comment_points_words.png http://minimaxir.com/img/hn-comments/distribution_comment_po...), but that can't be remade with the BigQuery dataset since comment scores are no longer public.
joecool1029 9y ago
I am extremely surprised rust wasn't included in here.
- var_explained 9y ago
  I've got a followup coming about what words lead to upvotes, and rust features quite prominently there!
drenvuk 9y ago
It would be nice to see a comparison of fastest growing words between the last 5 years vs 10 years ago. I'm wondering about the demographics of this site and if they've changed.
deleted 9y ago
[deleted]
flavio81 9y ago
A frequent post then would be: "Using VR to train a deep learning neural network on driving and react correctly to unexpected conditions, a bot implemented via a microservices stack using aws as a container and of course connected with cars and related traffic devices via the IoT, logging unexpected events into a blockchain."
- forgot-my-pw 9y ago
  Missing "HN" somewhere.
- Mz 9y ago
  Just by eyeballing it, I am pretty sure that exceeds 80 chars.
SippinLean 9y ago
>I don’t currently have a guess for why “million” and “billion” had sudden dropoffs in 2014. Is it some artifact of the Hacker News policy, with the word becoming edited or deleted in newer posts? Or is it a real change in what the site discusses? Any guesses on this one?

118 points by var_explained 9y ago

Hmm, the BigQuery HN dataset is now updated daily and contains comments as well as stories? That's new, and I'll certainly give it another look at for my projects. With the bigrquery R package (https://github.com/rstats-db/bigrquery https://github.com/rstats-db/bigrquery), you can access the HN dataset directly from R, using dplyr syntax too. (for simple queries atleast; you can pass the raw SQL for complex queries) As noted, the resulting dataset of words is large, so mapping the words in BigQuery itself may be more practical (using a combo of SPLIT and UNNEST with standard SQL), although of course you can't do complex operations like logistic regression or splines there.

robteix 9y ago

I'm surprised both "NSA" and "surveillance" are two of the fastest shrinking words. I thought we saw more now than ever. Shows how perception doesn't always match reality.

jdminhbg 9y ago
When the Snowden leaks first dropped, the front page was absolutely overwhelmed with NSA news, to the exclusion of nearly everything else. Would not be possible to keep that level of interest up without making this is an exclusively NSA/surveillance-driven site.
- Houshalter 9y ago
  IIRC the mods also soft banned it because of that. So posts with the word "NSA" in the title get penalized and ranked much lower than other posts. Hence the shrinking.
  - dang 9y ago
    We did that for a while but stopped already years ago. It was only needed until the story barrage leveled off.

aswanson 9y ago

A more interesting analysis would be comment length.

minimaxir 9y ago
In my old analysis (http://minimaxir.com/2014/10/hn-comments-about-comments/ http://minimaxir.com/2014/10/hn-comments-about-comments/), it's not that interesting. Comments are getting longer over time on average (http://minimaxir.com/img/hn-comments/monthly_average_words.png http://minimaxir.com/img/hn-comments/monthly_average_words.p...), and there is a slight positive correlation between comment score and comment length (http://minimaxir.com/img/hn-comments/distribution_comment_points_words.png http://minimaxir.com/img/hn-comments/distribution_comment_po...), but that can't be remade with the BigQuery dataset since comment scores are no longer public.

joecool1029 9y ago

I am extremely surprised rust wasn't included in here.

var_explained 9y ago
I've got a followup coming about what words lead to upvotes, and rust features quite prominently there!

drenvuk 9y ago

It would be nice to see a comparison of fastest growing words between the last 5 years vs 10 years ago. I'm wondering about the demographics of this site and if they've changed.

deleted 9y ago

[deleted]

flavio81 9y ago

A frequent post then would be: "Using VR to train a deep learning neural network on driving and react correctly to unexpected conditions, a bot implemented via a microservices stack using aws as a container and of course connected with cars and related traffic devices via the IoT, logging unexpected events into a blockchain."

forgot-my-pw 9y ago
Missing "HN" somewhere.
Mz 9y ago
Just by eyeballing it, I am pretty sure that exceeds 80 chars.

SippinLean 9y ago

>I don’t currently have a guess for why “million” and “billion” had sudden dropoffs in 2014. Is it some artifact of the Hacker News policy, with the word becoming edited or deleted in newer posts? Or is it a real change in what the site discusses? Any guesses on this one?