7 ms·

HN-Index

40 points by alexmolas 3y ago

deleted 3y ago
[deleted]
Waterluvian 3y ago
reads blog and sees own name I wonder if you’d see a bimodal distribution of some sort for posters and commenters. It feels about right that you’ll have people who post a lot and people who comment a lot and then mixes of the two modes. Or is it that people who post a lot also tend to comment a lot too?
montebicyclelo 3y ago
The open Google Bigquery table of Hacker News might possibly be a nice way to get this data. (Although last time I checked it was a few years out of date, so maybe not... Hoping that someone reads this and starts updating it again...)
- zX41ZdbW 3y ago
  There is a constantly updated table in ClickHouse with no login required for querying: https://play.clickhouse.com/play?user=play#U0VMRUNUIGNvdW50KCksIG1heCh0aW1lKSBGUk9NIGhhY2tlcm5ld3NfaGlzdG9yeQ== https://play.clickhouse.com/play?user=play#U0VMRUNUIGNvdW50K...
  - xnx 3y ago
    Does that still work? I'm getting "Code: 60. DB::Exception: Table default.hackernews_history does not exist."
    - codefined 3y ago
      The table name has been changed to "hackernews", although it also hasn't been updated for 7 months.
      - montebicyclelo 3y ago
        It appears to be working for me: SELECT count(), min(time), max(time) FROM hackernews_history 39838828, 1970-01-01 00:00:00, 2024-03-11 13:04:47
    - zX41ZdbW 3y ago
      I created this table on the European server, but forgot to enable replication to the US.
      - xnx 3y ago
        Wow. Awesome resource!
  - deleted 3y ago
    [deleted]
  - montebicyclelo 3y ago
    Nice, here's an attempt to compute the HN-index for all users. (Differs from blog post; not sure if because the data from the two sources is different, or there's a mistake somewhere): WITH Ranking AS ( SELECT by, score, RANK () OVER (PARTITION BY by ORDER BY score DESC) AS rank FROM hackernews_history ) SELECT by, MIN(rank - 1) as h_index FROM Ranking WHERE rank > score GROUP BY by ORDER BY h_index DESC LIMIT 100
naitgacem 3y ago
if OP is reading this, HN also exposes an Algolia search API. Perhaps they could investigate if it would give them an easier time with their analysis.
- SushiHippie 3y ago
  Probably, as it returns directly a list of all stories (without comments) and their vote count, and then quickly paginate it, way fewer requests than using the firebase API For example for the user alexmolas, filtered to only show stories and show 50 items per page (the maximum): https://hn.algolia.com/api/v1/search?tags=story,author_alexmolas&hitsPerPage=50 https://hn.algolia.com/api/v1/search?tags=story,author_alexm... Also adding &numericFilters=points>1 filters out all posts that don't have any votes (all posts have at least one point).
p4bl0 3y ago
The author says he's looking for blogs to follow. So why compute this score based on submitted stories only? People most often submit stories not written by themselves, so it does not really reflect what may be found on their blog (if they have any). If they have a blog, then my guess would be that their blog posts are likely more similar to their comments than their submitted stories, since it's written by them for sure in both cases.
- deleted 3y ago
  [deleted]
aaron695 3y ago
Because there is almost no cost to submitting a story, either on HN or for the calculation of an h-index it doesn't work. Comments are more interesting, but the data is hidden. It's interesting to look at for insights. A low h-index among top users is an imperfect way to see who adds the most value. It works in the opposite way.
outcoldman 3y ago
If people have missed we collected less than a year ago personal blogs of HN users https://github.com/outcoldman/hackernews-personal-blogs https://github.com/outcoldman/hackernews-personal-blogs
abetusk 3y ago
Note I did a scrape four years ago which I saved on archive.org [0] and provided some pedestrian analysis on it [1], along with source code [2]. It looks like a sqlite3 database of all items, including comments and stories, was about 5Gb, so I would expect it to be within the range of 10Gb or so now. I think h-index is an interesting idea but the post only really goes into it for story submission. The author scoffs at doing h-index for everyone but I don't think it would really be that bad. What does an h-index for comments look like? A commentor with h comments that have at least h karma each? Or h replies? Or should it be h comments with `alpha *` h karma, for some value of alpha? It's amazing to me how much progress has happened in 4 years. We now have LLMs. It'd be interesting to try and use the HN data for that type of purpose. Even using it to pick out links for people with high karma (or high h-index comment) could help with blog discovery. [0] https://archive.org/details/HackerNews-Data-2020-06-28 https://archive.org/details/HackerNews-Data-2020-06-28 [1] https://abetusk.github.io/yahnda/ https://abetusk.github.io/yahnda/ [2] https://github.com/abetusk/yahnda https://github.com/abetusk/yahnda

40 points by alexmolas 3y ago

deleted 3y ago

[deleted]

reads blog and sees own name I wonder if you’d see a bimodal distribution of some sort for posters and commenters. It feels about right that you’ll have people who post a lot and people who comment a lot and then mixes of the two modes. Or is it that people who post a lot also tend to comment a lot too?

montebicyclelo 3y ago

The open Google Bigquery table of Hacker News might possibly be a nice way to get this data. (Although last time I checked it was a few years out of date, so maybe not... Hoping that someone reads this and starts updating it again...)

zX41ZdbW 3y ago
There is a constantly updated table in ClickHouse with no login required for querying: https://play.clickhouse.com/play?user=play#U0VMRUNUIGNvdW50KCksIG1heCh0aW1lKSBGUk9NIGhhY2tlcm5ld3NfaGlzdG9yeQ== https://play.clickhouse.com/play?user=play#U0VMRUNUIGNvdW50K...
- xnx 3y ago
  Does that still work? I'm getting "Code: 60. DB::Exception: Table default.hackernews_history does not exist."
  - codefined 3y ago
    The table name has been changed to "hackernews", although it also hasn't been updated for 7 months.
    - montebicyclelo 3y ago
      It appears to be working for me: SELECT count(), min(time), max(time) FROM hackernews_history 39838828, 1970-01-01 00:00:00, 2024-03-11 13:04:47
  - zX41ZdbW 3y ago
    I created this table on the European server, but forgot to enable replication to the US.
    - xnx 3y ago
      Wow. Awesome resource!
- deleted 3y ago
  [deleted]
- montebicyclelo 3y ago
  Nice, here's an attempt to compute the HN-index for all users. (Differs from blog post; not sure if because the data from the two sources is different, or there's a mistake somewhere): WITH Ranking AS ( SELECT by, score, RANK () OVER (PARTITION BY by ORDER BY score DESC) AS rank FROM hackernews_history ) SELECT by, MIN(rank - 1) as h_index FROM Ranking WHERE rank > score GROUP BY by ORDER BY h_index DESC LIMIT 100

naitgacem 3y ago

if OP is reading this, HN also exposes an Algolia search API. Perhaps they could investigate if it would give them an easier time with their analysis.

SushiHippie 3y ago
Probably, as it returns directly a list of all stories (without comments) and their vote count, and then quickly paginate it, way fewer requests than using the firebase API For example for the user alexmolas, filtered to only show stories and show 50 items per page (the maximum): https://hn.algolia.com/api/v1/search?tags=story,author_alexmolas&hitsPerPage=50 https://hn.algolia.com/api/v1/search?tags=story,author_alexm... Also adding &numericFilters=points>1 filters out all posts that don't have any votes (all posts have at least one point).

p4bl0 3y ago

The author says he's looking for blogs to follow. So why compute this score based on submitted stories only? People most often submit stories not written by themselves, so it does not really reflect what may be found on their blog (if they have any). If they have a blog, then my guess would be that their blog posts are likely more similar to their comments than their submitted stories, since it's written by them for sure in both cases.

deleted 3y ago
[deleted]

aaron695 3y ago

Because there is almost no cost to submitting a story, either on HN or for the calculation of an h-index it doesn't work. Comments are more interesting, but the data is hidden. It's interesting to look at for insights. A low h-index among top users is an imperfect way to see who adds the most value. It works in the opposite way.

outcoldman 3y ago

If people have missed we collected less than a year ago personal blogs of HN users https://github.com/outcoldman/hackernews-personal-blogs https://github.com/outcoldman/hackernews-personal-blogs

abetusk 3y ago

Note I did a scrape four years ago which I saved on archive.org [0] and provided some pedestrian analysis on it [1], along with source code [2]. It looks like a sqlite3 database of all items, including comments and stories, was about 5Gb, so I would expect it to be within the range of 10Gb or so now. I think h-index is an interesting idea but the post only really goes into it for story submission. The author scoffs at doing h-index for everyone but I don't think it would really be that bad. What does an h-index for comments look like? A commentor with h comments that have at least h karma each? Or h replies? Or should it be h comments with `alpha *` h karma, for some value of alpha? It's amazing to me how much progress has happened in 4 years. We now have LLMs. It'd be interesting to try and use the HN data for that type of purpose. Even using it to pick out links for people with high karma (or high h-index comment) could help with blog discovery. [0] https://archive.org/details/HackerNews-Data-2020-06-28 https://archive.org/details/HackerNews-Data-2020-06-28 [1] https://abetusk.github.io/yahnda/ https://abetusk.github.io/yahnda/ [2] https://github.com/abetusk/yahnda https://github.com/abetusk/yahnda