Note I did a scrape four years ago which I saved on archive.org [0] and provided some pedestrian analysis on it [1], along with source code [2]. It looks like

Note I did a scrape four years ago which I saved on archive.org [0] and provided some pedestrian analysis on it [1], along with source code [2]. It looks like a sqlite3 database of all items, including comments and stories, was about 5Gb, so I would expect it to be within the range of 10Gb or so now. I think h-index is an interesting idea but the post only really goes into it for story submission. The author scoffs at doing h-index for everyone but I don't think it would really be that bad. What does an h-index for comments look like? A commentor with h comments that have at least h karma each? Or h replies? Or should it be h comments with `alpha *` h karma, for some value of alpha? It's amazing to me how much progress has happened in 4 years. We now have LLMs. It'd be interesting to try and use the HN data for that type of purpose. Even using it to pick out links for people with high karma (or high h-index comment) could help with blog discovery. [0] https://archive.org/details/HackerNews-Data-2020-06-28 https://archive.org/details/HackerNews-Data-2020-06-28 [1] https://abetusk.github.io/yahnda/ https://abetusk.github.io/yahnda/ [2] https://github.com/abetusk/yahnda https://github.com/abetusk/yahnda