6 ms·

What gets to the front page of Hacker News? A data project

dredmorbius 3y ago
So, oddly enough, I've also been looking at HN front-page characteristics, based on the same corpus (the "past" page links). And that whole section on caveats over what that archive represents is something I could have written... The front page, both in its dynamic and archived forms is strongly subject to many influences in complex ways. A couple of tips: - It's possible to crawl the page using wget, given a reasonable delay. The full collection from 2007 to present (I'd done my first crawl in late May of this year) took a couple of days. Updates to that happen in seconds. - I break down data by date, story position (e.g., rank 1--30), submitted site (if present), points (votes), comments, and submitter, as well as title. - I'm working on classifying titles. The original question prompting my analysis was what US states get the most love from HN (NY, CA, WA*, TX, and CO are the top 5). I'd expanded that US and globally-significant cities, and been doing some tuple-based ngram analysis, though that gets pretty hairy. For 2022 (most recent complete year), the top 40 submitted front-page sites are: 2022: Distinct sites: 6446 Site Stories Points ( mean ) Comments ( mean ) ------------------------------ ------- ------ ---------- -------- ---------- n/a 432 167275 ( 386.32 ) 125304 ( 289.39 ) youtube.com 105 27243 ( 257.01 ) 12489 ( 117.82 ) nature.com 80 17694 ( 218.44 ) 11716 ( 144.64 ) wikipedia.org 68 12258 ( 177.65 ) 5855 ( 84.86 ) nytimes.com 67 21190 ( 311.62 ) 21765 ( 320.07 ) arstechnica.com 63 18319 ( 286.23 ) 12057 ( 188.39 ) ieee.org 53 9432 ( 174.67 ) 5933 ( 109.87 ) reuters.com 53 28360 ( 525.19 ) 29033 ( 537.65 ) theguardian.com 49 12228 ( 244.56 ) 8677 ( 173.54 ) quantamagazine.org 48 11293 ( 230.47 ) 5519 ( 112.63 ) science.org 47 12485 ( 260.10 ) 7655 ( 159.48 ) economist.com 46 12504 ( 266.04 ) 17324 ( 368.60 ) bloomberg.com 43 20037 ( 455.39 ) 20630 ( 468.86 ) lwn.net 43 10566 ( 240.14 ) 5912 ( 134.36 ) theverge.com 43 16313 ( 370.75 ) 14335 ( 325.80 ) arxiv.org 39 7415 ( 185.38 ) 3559 ( 88.97 ) washingtonpost.com 39 15778 ( 394.45 ) 18117 ( 452.93 ) bbc.com 37 11600 ( 305.26 ) 8696 ( 228.84 ) newyorker.com 37 7577 ( 199.39 ) 6549 ( 172.34 ) wsj.com 36 10920 ( 295.14 ) 11646 ( 314.76 ) wired.com 35 9104 ( 252.89 ) 6738 ( 187.17 ) archive.org 32 8011 ( 242.76 ) 4626 ( 140.18 ) gist.github.com 32 10287 ( 311.73 ) 5456 ( 165.33 ) reddit.com 30 12579 ( 405.77 ) 8457 ( 272.81 ) theregister.com 29 8288 ( 276.27 ) 4586 ( 152.87 ) apple.com 28 13245 ( 456.72 ) 12917 ( 445.41 ) github.blog 26 8398 ( 311.04 ) 4242 ( 157.11 ) cnbc.com 23 8568 ( 357.00 ) 10356 ( 431.50 ) phys.org 23 4918 ( 204.92 ) 2380 ( 99.17 ) theatlantic.com 23 7518 ( 313.25 ) 10643 ( 443.46 ) axios.com 22 8903 ( 387.09 ) 8616 ( 374.61 ) news.mit.edu 22 6181 ( 268.74 ) 2887 ( 125.52 ) smithsonianmag.com 22 4964 ( 215.83 ) 2988 ( 129.91 ) stanford.edu 22 8461 ( 367.87 ) 4720 ( 205.22 ) krebsonsecurity.com 21 6299 ( 286.32 ) 3331 ( 151.41 ) microsoft.com 21 7809 ( 354.95 ) 4392 ( 199.64 ) atlasobscura.com 20 2789 ( 132.81 ) 1637 ( 77.95 ) cnn.com 19 4704 ( 235.20 ) 4252 ( 212.60 ) righto.com 19 2568 ( 128.40 ) 795 ( 39.75 ) simonwillison.net 17 4878 ( 271.00 ) 1553 ( 86.28 ) TechCrunch, BTW, lands at #41: techcrunch.com 17 8681 ( 482.28 ) 8224 ( 456.89 ) (The "mean" values are the arithmetic mean of points (votes) and comments by domain.) For 2023, there've only been 10 TechCrunch items (through 21-6-2023), well below trend: Ubuntu 22.04 LTS servers and phased apt updates Twitterrific has been discontinued DuckDB – An in-process SQL OLAP database management system Shane Pitman, leader of the warez group Razor 1911: life after prison (2005) Nearly 40% of software engineers will only work remotely Htmx 1.9.0 has been released Geometry Central: library of data structures, algorithms for geometry processing Google Authenticator now supports Google Account synchronization I Wrote an Activitypub Server in OCaml: Lessons Learnt, Weekends Lost In New Paradox, Black Holes Appear to Evade Heat Death I'll note that breaking stories down by site will tend to obscure categories, as frequently-submitted sites (say, NY Times) will crowd out many individual blogs. I could probably do some manual classification based on sites, including, say, all categories of Twitter (currently broken out by user/account), and might look into that. One of the most surprising facts to jump out to me is how much nytimes.com has fallen since 2019. It had previously been in the top-4 submitted sites pretty consistently, and single top for 2014--2019, but fell to 7th in 2020 and 9th in 2021, recovering to 5 in 2022. I've also paired my own analysis with a 2022 study published by Whaly.io based on the HN API and all content submitted: <https://whaly.io/posts/hacker-news-2021-retrospective https://whaly.io/posts/hacker-news-2021-retrospective> I've been somewhat live-bloogging my analysis on the Fediverse under the #HackerNewsAnalytics hashtag: <https://toot.cat/@dredmorbius/tagged/HackerNewsAnalytics https://toot.cat/@dredmorbius/tagged/HackerNewsAnalytics> That includes a number of findings (and testing/debugging notes), including: mentions of Reddit by year, mentions of the FP-500 companies (top-10: Apple, Microsoft, Amazon, Intel, Tesla, Netflix, IBM, Adobe, Oracle, and AT&T, though Google under various terms (Google, Alphabet, YouTube, Android) nearly doubles the top-ranked Apple, and no, adding in iPhone, iPad, MacBook, etc., doesn't help), trends in votes and comments by story position (interesting IMO), overall submission success rate (a hair under 3%), mentions of the FP Top 100 Global Thinkers in titles (reprising an old study of mine of numerous online sites), a look at the Leaders characteristics, what HN cares about being down, and, well, ... things: <https://toot.cat/@dredmorbius/110454128168815763 https://toot.cat/@dredmorbius/110454128168815763> ________________________________ Notes: * "Washington" can of course designate both a city and a state, amongst other things, and it turns out that the string is dominated by references to the Washington Post, much as "New York" is by the New York Times. But the list gives the naive ranking. Adding in "Silicon Valley" and "San Francisco" put California well on top. Edits: Some in situ updates as I think of things. Sorry!
- dredmorbius 3y ago
  And for an overall activity summary: Years: 16 Start year: 2007 End year: 2023 Total Stories: 178882 Distinct Sites: 52642 Distinct Submitters: 43648 Year Stories Points ( mean ) Comments ( mean ) Sites Submitters ---- ------- ------ --------- -------- --------- ----- ---------- 2007 9382 92264 ( 9.83 ) 61207 ( 6.52 ) 2644 1163 2008 10980 294775 ( 26.85 ) 186339 ( 16.97 ) 3458 2021 2009 10950 608603 ( 55.58 ) 303962 ( 27.76 ) 4157 3017 2010 10950 1062763 ( 97.06 ) 491718 ( 44.91 ) 4397 3987 2011 10949 1657004 ( 151.34 ) 632724 ( 57.79 ) 4830 4889 2012 10980 1829402 ( 166.61 ) 778634 ( 70.91 ) 5047 5537 2013 10950 2132819 ( 194.78 ) 998387 ( 91.18 ) 5250 5975 2014 10905 2057628 ( 188.69 ) 916438 ( 84.04 ) 5338 5920 2015 10950 2001269 ( 182.76 ) 845719 ( 77.23 ) 5301 5313 2016 10977 2521394 ( 229.70 ) 1137575 ( 103.63 ) 5238 5341 2017 10950 2776064 ( 253.52 ) 1259899 ( 115.06 ) 5236 5274 2018 10950 2762928 ( 252.32 ) 1262654 ( 115.31 ) 4986 5038 2019 10950 3051011 ( 278.63 ) 1447141 ( 132.16 ) 5302 5123 2020 10980 3338150 ( 304.02 ) 1734703 ( 157.99 ) 5938 5564 2021 10950 3376829 ( 308.39 ) 1859933 ( 169.86 ) 6178 5339 2022 10950 3308025 ( 302.10 ) 1986265 ( 181.39 ) 6446 5443 2023 5160 1555335 ( 301.42 ) 879401 ( 170.43 ) 3253 2851 Data through 2023-6-21.
- gagejustins 3y ago
  Oh man, this is awesome. I learned a lot from collecting this data and one of the big takeaways for me was how diverse the set of news sources on HN is (to your point, very little "traditional" journalism here). Glad you're doing this!
  - dredmorbius 3y ago
    Drop me a line if you'd like to discuss this / share w/ reports. username at Protonmail. I'm sort of a Can Haz All The Tables sort of guy, and I'm largely processing via awk (and a few other shell tools). So pasting that here would get a bit tedious... It's also been interesting to look at how HN has, and hasn't, changed over the years. Your categorical analysis would be an interesting filter to look at over time, especially regarding accusations that HN is drifting in various directions. The other bit that stands out to me is how constrained a set the front page is (30 slots per day, 10,950 per year, 10,980 in a leap year), as well as how thin submission titles are for gleaning meaning and context (I'm ... somewhat frustrated by this). Though there is clearly signal that gets through. I don't have time-of-day granularity, but can look at day-of-week (and have) and month-of-year (not yet) looking for seasonality. DoW has been interesting (usually peaks Tue/Wed, starts trailing off on Fri, Sat & Sun are low points, based on votes/comments, but give higher odds of a given submission landing). You might want to look at Whaly's work as well (I'd edited it into my larger top-level comment above: <https://whaly.io/posts/hacker-news-2021-retrospective https://whaly.io/posts/hacker-news-2021-retrospective>).

So, oddly enough, I've also been looking at HN front-page characteristics, based on the same corpus (the "past" page links). And that whole section on caveats over what that archive represents is something I could have written... The front page, both in its dynamic and archived forms is strongly subject to many influences in complex ways. A couple of tips: - It's possible to crawl the page using wget, given a reasonable delay. The full collection from 2007 to present (I'd done my first crawl in late May of this year) took a couple of days. Updates to that happen in seconds. - I break down data by date, story position (e.g., rank 1--30), submitted site (if present), points (votes), comments, and submitter, as well as title. - I'm working on classifying titles. The original question prompting my analysis was what US states get the most love from HN (NY, CA, WA*, TX, and CO are the top 5). I'd expanded that US and globally-significant cities, and been doing some tuple-based ngram analysis, though that gets pretty hairy. For 2022 (most recent complete year), the top 40 submitted front-page sites are: 2022: Distinct sites: 6446 Site Stories Points ( mean ) Comments ( mean ) ------------------------------ ------- ------ ---------- -------- ---------- n/a 432 167275 ( 386.32 ) 125304 ( 289.39 ) youtube.com 105 27243 ( 257.01 ) 12489 ( 117.82 ) nature.com 80 17694 ( 218.44 ) 11716 ( 144.64 ) wikipedia.org 68 12258 ( 177.65 ) 5855 ( 84.86 ) nytimes.com 67 21190 ( 311.62 ) 21765 ( 320.07 ) arstechnica.com 63 18319 ( 286.23 ) 12057 ( 188.39 ) ieee.org 53 9432 ( 174.67 ) 5933 ( 109.87 ) reuters.com 53 28360 ( 525.19 ) 29033 ( 537.65 ) theguardian.com 49 12228 ( 244.56 ) 8677 ( 173.54 ) quantamagazine.org 48 11293 ( 230.47 ) 5519 ( 112.63 ) science.org 47 12485 ( 260.10 ) 7655 ( 159.48 ) economist.com 46 12504 ( 266.04 ) 17324 ( 368.60 ) bloomberg.com 43 20037 ( 455.39 ) 20630 ( 468.86 ) lwn.net 43 10566 ( 240.14 ) 5912 ( 134.36 ) theverge.com 43 16313 ( 370.75 ) 14335 ( 325.80 ) arxiv.org 39 7415 ( 185.38 ) 3559 ( 88.97 ) washingtonpost.com 39 15778 ( 394.45 ) 18117 ( 452.93 ) bbc.com 37 11600 ( 305.26 ) 8696 ( 228.84 ) newyorker.com 37 7577 ( 199.39 ) 6549 ( 172.34 ) wsj.com 36 10920 ( 295.14 ) 11646 ( 314.76 ) wired.com 35 9104 ( 252.89 ) 6738 ( 187.17 ) archive.org 32 8011 ( 242.76 ) 4626 ( 140.18 ) gist.github.com 32 10287 ( 311.73 ) 5456 ( 165.33 ) reddit.com 30 12579 ( 405.77 ) 8457 ( 272.81 ) theregister.com 29 8288 ( 276.27 ) 4586 ( 152.87 ) apple.com 28 13245 ( 456.72 ) 12917 ( 445.41 ) github.blog 26 8398 ( 311.04 ) 4242 ( 157.11 ) cnbc.com 23 8568 ( 357.00 ) 10356 ( 431.50 ) phys.org 23 4918 ( 204.92 ) 2380 ( 99.17 ) theatlantic.com 23 7518 ( 313.25 ) 10643 ( 443.46 ) axios.com 22 8903 ( 387.09 ) 8616 ( 374.61 ) news.mit.edu 22 6181 ( 268.74 ) 2887 ( 125.52 ) smithsonianmag.com 22 4964 ( 215.83 ) 2988 ( 129.91 ) stanford.edu 22 8461 ( 367.87 ) 4720 ( 205.22 ) krebsonsecurity.com 21 6299 ( 286.32 ) 3331 ( 151.41 ) microsoft.com 21 7809 ( 354.95 ) 4392 ( 199.64 ) atlasobscura.com 20 2789 ( 132.81 ) 1637 ( 77.95 ) cnn.com 19 4704 ( 235.20 ) 4252 ( 212.60 ) righto.com 19 2568 ( 128.40 ) 795 ( 39.75 ) simonwillison.net 17 4878 ( 271.00 ) 1553 ( 86.28 ) TechCrunch, BTW, lands at #41: techcrunch.com 17 8681 ( 482.28 ) 8224 ( 456.89 ) (The "mean" values are the arithmetic mean of points (votes) and comments by domain.) For 2023, there've only been 10 TechCrunch items (through 21-6-2023), well below trend: Ubuntu 22.04 LTS servers and phased apt updates Twitterrific has been discontinued DuckDB – An in-process SQL OLAP database management system Shane Pitman, leader of the warez group Razor 1911: life after prison (2005) Nearly 40% of software engineers will only work remotely Htmx 1.9.0 has been released Geometry Central: library of data structures, algorithms for geometry processing Google Authenticator now supports Google Account synchronization I Wrote an Activitypub Server in OCaml: Lessons Learnt, Weekends Lost In New Paradox, Black Holes Appear to Evade Heat Death I'll note that breaking stories down by site will tend to obscure categories, as frequently-submitted sites (say, NY Times) will crowd out many individual blogs. I could probably do some manual classification based on sites, including, say, all categories of Twitter (currently broken out by user/account), and might look into that. One of the most surprising facts to jump out to me is how much nytimes.com has fallen since 2019. It had previously been in the top-4 submitted sites pretty consistently, and single top for 2014--2019, but fell to 7th in 2020 and 9th in 2021, recovering to 5 in 2022. I've also paired my own analysis with a 2022 study published by Whaly.io based on the HN API and all content submitted: <https://whaly.io/posts/hacker-news-2021-retrospective https://whaly.io/posts/hacker-news-2021-retrospective> I've been somewhat live-bloogging my analysis on the Fediverse under the #HackerNewsAnalytics hashtag: <https://toot.cat/@dredmorbius/tagged/HackerNewsAnalytics https://toot.cat/@dredmorbius/tagged/HackerNewsAnalytics> That includes a number of findings (and testing/debugging notes), including: mentions of Reddit by year, mentions of the FP-500 companies (top-10: Apple, Microsoft, Amazon, Intel, Tesla, Netflix, IBM, Adobe, Oracle, and AT&T, though Google under various terms (Google, Alphabet, YouTube, Android) nearly doubles the top-ranked Apple, and no, adding in iPhone, iPad, MacBook, etc., doesn't help), trends in votes and comments by story position (interesting IMO), overall submission success rate (a hair under 3%), mentions of the FP Top 100 Global Thinkers in titles (reprising an old study of mine of numerous online sites), a look at the Leaders characteristics, what HN cares about being down, and, well, ... things: <https://toot.cat/@dredmorbius/110454128168815763 https://toot.cat/@dredmorbius/110454128168815763> ________________________________ Notes: * "Washington" can of course designate both a city and a state, amongst other things, and it turns out that the string is dominated by references to the Washington Post, much as "New York" is by the New York Times. But the list gives the naive ranking. Adding in "Silicon Valley" and "San Francisco" put California well on top. Edits: Some in situ updates as I think of things. Sorry!

dredmorbius 3y ago
And for an overall activity summary: Years: 16 Start year: 2007 End year: 2023 Total Stories: 178882 Distinct Sites: 52642 Distinct Submitters: 43648 Year Stories Points ( mean ) Comments ( mean ) Sites Submitters ---- ------- ------ --------- -------- --------- ----- ---------- 2007 9382 92264 ( 9.83 ) 61207 ( 6.52 ) 2644 1163 2008 10980 294775 ( 26.85 ) 186339 ( 16.97 ) 3458 2021 2009 10950 608603 ( 55.58 ) 303962 ( 27.76 ) 4157 3017 2010 10950 1062763 ( 97.06 ) 491718 ( 44.91 ) 4397 3987 2011 10949 1657004 ( 151.34 ) 632724 ( 57.79 ) 4830 4889 2012 10980 1829402 ( 166.61 ) 778634 ( 70.91 ) 5047 5537 2013 10950 2132819 ( 194.78 ) 998387 ( 91.18 ) 5250 5975 2014 10905 2057628 ( 188.69 ) 916438 ( 84.04 ) 5338 5920 2015 10950 2001269 ( 182.76 ) 845719 ( 77.23 ) 5301 5313 2016 10977 2521394 ( 229.70 ) 1137575 ( 103.63 ) 5238 5341 2017 10950 2776064 ( 253.52 ) 1259899 ( 115.06 ) 5236 5274 2018 10950 2762928 ( 252.32 ) 1262654 ( 115.31 ) 4986 5038 2019 10950 3051011 ( 278.63 ) 1447141 ( 132.16 ) 5302 5123 2020 10980 3338150 ( 304.02 ) 1734703 ( 157.99 ) 5938 5564 2021 10950 3376829 ( 308.39 ) 1859933 ( 169.86 ) 6178 5339 2022 10950 3308025 ( 302.10 ) 1986265 ( 181.39 ) 6446 5443 2023 5160 1555335 ( 301.42 ) 879401 ( 170.43 ) 3253 2851 Data through 2023-6-21.
gagejustins 3y ago
Oh man, this is awesome. I learned a lot from collecting this data and one of the big takeaways for me was how diverse the set of news sources on HN is (to your point, very little "traditional" journalism here). Glad you're doing this!
- dredmorbius 3y ago
  Drop me a line if you'd like to discuss this / share w/ reports. username at Protonmail. I'm sort of a Can Haz All The Tables sort of guy, and I'm largely processing via awk (and a few other shell tools). So pasting that here would get a bit tedious... It's also been interesting to look at how HN has, and hasn't, changed over the years. Your categorical analysis would be an interesting filter to look at over time, especially regarding accusations that HN is drifting in various directions. The other bit that stands out to me is how constrained a set the front page is (30 slots per day, 10,950 per year, 10,980 in a leap year), as well as how thin submission titles are for gleaning meaning and context (I'm ... somewhat frustrated by this). Though there is clearly signal that gets through. I don't have time-of-day granularity, but can look at day-of-week (and have) and month-of-year (not yet) looking for seasonality. DoW has been interesting (usually peaks Tue/Wed, starts trailing off on Fri, Sat & Sun are low points, based on votes/comments, but give higher odds of a given submission landing). You might want to look at Whaly's work as well (I'd edited it into my larger top-level comment above: <https://whaly.io/posts/hacker-news-2021-retrospective https://whaly.io/posts/hacker-news-2021-retrospective>).