3 ms·
Ask HN: Anyone want my realtime-snapshots dataset of Hacker News?
- namiwang 1mo agoAsk HN: Anyone want my realtime-snapshots dataset of Hacker News? so I've been maintaining a project [Track HN](https://track-hacker-news.com https://track-hacker-news.com) since 2023. it continuously takes snapshots of Hacker News, recording how the score/rank/comments of stories/comments change over time. I think this is a unique HN dataset in its own way. I don't have enough time to maintain this project anymore and wondering if anyone would find the data useful and want to take it over and keep it alive. ## the dataset the dataset is over 200M rows in a postgres instance, plus a clickhouse instance to speedup the analytics. - stories/comments/users - ~6.38M Stories, ~43.2M comments, ~1.15M users, since the epoch (2006-10-09) - story score/comment count history - ~21.5M snapshots for ~900K stories, since 2024-03-22 - story rank history - ~145M snapshots for ~178,120 stories, since 2024-03-28 - http status of popular posts ## the product and stack I've build a product around this dataset with a stack: - a web app (with rails) to crawl and serve the data; - dashboards with metabase; - another set of dashboards with retool (retool kindly sponsored a postgres instance back then); - a browser extension; - few data reports; ## ideas things you could look at from the data, for example: - analyze activities of specific HN user - see how stories moving through the front page - find patterns between early score and eventual score - see how long stories survive at different ranks - find posting time/weekday effects, domain/user patterns, etc. - detect/extract topics/keywords from comments for brands/products/languages, etc. Let me know if you're interested, send your proposal/questions/cool ideas to me#nami.land. thanks!
- zX41ZdbW 1mo agoI'm interested. milovidov@clickhouse.com