2 ms·
Dates covered: 2007-2-20 -- 2023-5-25 Total stories: 178,072 (Should match 5,939 days * 30 stories, seems to come a bit short with 98 fewer stories than expe
by dredmorbius 3y ago
Dates covered: 2007-2-20 -- 2023-5-25
Total stories: 178,072 (Should match 5,939 days * 30 stories, seems to come a bit short with 98 fewer stories than expected.)
- States mentioned: 50 (mentions: 1,344)
- Cities mentioned: 109 (checked against list of 330 largest US cities, plus a few additions of my own).
- Sites submitted: 52,687
Top 20 states by raw string-match (this is adjusted below for some overstatements):
1. new york: 316
2. california: 308
3. texas: 98
4. washington: 73
5. colorado: 38
6. florida: 38
7. arizona: 28
8. georgia: 24
9. utah: 24
10. kansas: 23
11. virginia: 23
12. oregon: 22
13. michigan: 19
14. minnesota: 16
15. north carolina: 16
16. ohio: 16
17. alaska: 15
18. indiana: 15
19. hawaii: 14
20. maine: 14
Confounding factors include other terminology, other name matches:
"New York" frequently appears in reference to newspapers or cities:
new york times: 98
new york city: 55
So adjusted NY score: 163
"Washington" is even harder to disambiguate, as it may refer to the state, city (DC), as a topnym for the US / US Government, etc., It's challenging to identify these textually, though the newspaper and monument mentions can be identified:
washington post: 16
washington times: 1
washington monument: 1
Adjusted WA score: 55. And is probably lower.
California is often mentioned by toponym (e.g., "silicon valley", region (bay area), or specific cities (SF, LA, etc.). I'm NOT going to adjust for these but do note the following major occurences:
silicon valley: 399
bay area: 65
california ave: 2
There are a few other more minor examples such as "kansas city" (confounding "kansas"), "oklahoma city", and "iowa city", though those don't affect top results.
Adjusted top-5 state rankings:
1. california: 308
2. new york: 163
3. texas: 98
4. washington: 55
5. colorado: 38
So, answering the original question, Colorado is the fifth-most-loved front-page state.
The 10 least-represented states:
41. new mexico: 6
42. alabama: 5
43. idaho: 5
44. north dakota: 5
45. connecticut: 4
46. nebraska: 4
47. new hampshire: 4
48. rhode island: 3
49. south dakota: 3
50. south carolina: 1
Top-10 US city-name mentions (see confounding factors above):
1. new york: 316
2. san francisco: 192
3. boston: 117
4. seattle: 87
5. hollywood: 76
5. tempe: 69
6. berkeley: 63
7. chicago: 62
8. cambridge: 50
9. detroit: 41
10. los angeles: 30
(This excludes the NYC burough of Manhattan, which gets 36 mentions itself, nudging out LA.)
I've also got a list of most-featured front-page sites, with the ten most frequent being: nytimes.com, techcrunch.com, arstechnica.com, bloomberg.com, wired.com, wsj.com, youtube.com, wikipedia.org, BBC (as both bbc.com and bbc.co.uk), and theguardian.com. The mean votes and comments for each are interesting, with Bloomberg topping by both average vote (255.46) and comments (178.04), from the top-10. (There are obviously sites with much higher mean scores but less frequent appearance, including singletons).
I've broken submitters by freqency, points, and comments as well, with 43,745 submitters accounting for the stories analised. The long-absent nickb tops front-page appearances at 1,322, followed by ingve (1,177) and tosh (679). I've apparently landed 60 front-page submissions averaging 329.00 points and 147.52 comments (both fairly respectable --- I'm apparently not submitting total tosh).
I'm still updating my code and checking data (and found a couple of glitches writing out the above, e.g., whatever I'd done to tabulate city mentions didn't work). Data exist presently in flatfiles, I'm thinking of dumping to sqlite for possibly easier/faster analysis.
A rough and preliminary look at countries represented in front-page titles, again with numerous issues:
1. US: 1350
2. U.S.: 1073
3. China: 634
4. Japan: 526
5. India: 477
6. UK: 288
7. EU: 225
8. Russia: 221
9. Germany: 165
10. Canada: 162
11. Australia: 157
12. Korea: 140
13. France: 116
14. Iran: 91
15. Dutch: 80
16. United States: 75
17. Brazil: 69
18. North Korea: 69
19. Sweden: 68
20. Cuba: 67
21. Egypt: 60
22. Norway: 60
23. USA: 59
24. E.U.: 54
25. Israel: 53
26. Iceland: 52
27. New Zealand: 47
28. Mexico: 45
29. Spain: 43
30. Finland: 39
31. Italy: 38
32. South Korea: 38
33. Switzerland: 38
34. Singapore: 36
35. Mali: 35
36. Chile: 33
37. Turkey: 33
38. Argentina: 27
39. Netherlands: 25
40. Syria: 25
"Cuba" often appears as "Mark Cuban", Jordan is the country most often confounded with a major celebrity and/or IT/tech personality ("Michael Jordan" and "Jordan Hubard" respectively), "Mali" is most likely to be confused with a car or AMD chipset, the United States has multiple possible references (US, U.S., USA, U.S.A., America, American), some of which are confounded (e.g., "North America"). But this provides a rough sense.
Analysis is largely awk, grep, sort, uniq --- standard shell tool stuff. Amazingly powerful though really. Crawl via wget which performed wonderfully as usual. Thanks to HN/YC for not throttling too aggressively.
- dredmorbius 3y agoOne question answered: there are in fact not 30 stories per summary, consistently. 5923 30 3 26 3 27 3 28 2 29 1 0 1 17 1 19 1 20 1 25 Which occur on the following dates: 2007-03-10: 29 2007-03-24: 26 2007-03-25: 25 2007-05-19: 27 2007-05-26: 26 2007-05-28: 29 2007-06-02: 19 2007-06-16: 28 2007-06-23: 17 2007-06-24: 28 2007-06-30: 20 2007-07-01: 28 2007-07-07: 27 2007-07-15: 26 2007-07-28: 27 2014-01-06: 0 The 2014 instance was a site outage, discussed here: <https://news.ycombinator.com/item?id=7015129 https://news.ycombinator.com/item?id=7015129> Spot checks suggest the others were in fact days with fewer-than-normal stories, e.g., <https://news.ycombinator.com/front?day=2007-06-30 https://news.ycombinator.com/front?day=2007-06-30>
- deleted 3y ago[deleted]
- dredmorbius 3y agoI'd long wondered about the day-of-week effect on HN. Turns out that yes, there is one. Looking at just 2022 (most recent full-year data: 2022: DoW Stories Points ( mean ) Comments ( mean ) Monday 1560 481368 ( 308.57 ) 288928 ( 185.21 ) Tuesday 1560 528265 ( 338.63 ) 309799 ( 198.59 ) Wednesday 1560 506977 ( 324.99 ) 310698 ( 199.17 ) Thursday 1560 514046 ( 329.52 ) 320883 ( 205.69 ) Friday 1560 465283 ( 298.26 ) 287805 ( 184.49 ) Saturday 1590 401207 ( 252.33 ) 228966 ( 144.00 ) Sunday 1560 410879 ( 263.38 ) 239186 ( 153.32 ) Tuesday and Wednesday typically see most engagement. Biggest fall-off is Saturday. On the other hand, if you want a chance of your submission making the front page, weekends are probably a better bet. (My data don't show this, mind, as they don't show all submissions for a day, but this tends to support my hunch / general sense in my own submissions.) It's interesting to note that Tuesday and Wednesday have generally captured the top spot, but there was a notable fall-off in Wednesday's standing in the Covid era. Not entirely unprecedented (Wed. lagged Tues. by 10 votes (points) in 2014, for example), but it catches my eye. There's also the long-term trends in votes and comments over the years: Year Stories Points ( mean ) Comments ( mean ) 2007 9382 92264 ( 9.83 ) 61207 ( 6.52 ) 2008 10980 294775 ( 26.85 ) 186339 ( 16.97 ) 2009 10950 608603 ( 55.58 ) 303962 ( 27.76 ) 2010 10950 1062763 ( 97.06 ) 491718 ( 44.91 ) 2011 10949 1657004 ( 151.34 ) 632724 ( 57.79 ) 2012 10980 1829402 ( 166.61 ) 778634 ( 70.91 ) 2013 10950 2132819 ( 194.78 ) 998387 ( 91.18 ) 2014 10905 2057628 ( 188.69 ) 916438 ( 84.04 ) 2015 10950 2001269 ( 182.76 ) 845719 ( 77.23 ) 2016 10977 2521394 ( 229.70 ) 1137575 ( 103.63 ) 2017 10950 2776064 ( 253.52 ) 1259899 ( 115.06 ) 2018 10950 2762928 ( 252.32 ) 1262654 ( 115.31 ) 2019 10950 3051011 ( 278.63 ) 1447141 ( 132.16 ) 2020 10980 3338150 ( 304.02 ) 1734703 ( 157.99 ) 2021 10950 3376829 ( 308.39 ) 1859933 ( 169.86 ) 2022 10950 3308025 ( 302.10 ) 1986265 ( 181.39 ) 2023 4350 1309865 ( 301.12 ) 737425 ( 169.52 )