8 ms·
where can i download the data behind this?
by mrtimo 3y ago
where can i download the data behind this?
- xnx 3y agoYou just need to make 38 million requests to the Firebase API document here https://github.com/HackerNews/API https://github.com/HackerNews/API. :-) I actually made many more than that because I screwed up a few times. Might clean up the compiled data and put it on Kaggle (GitHub? Torrent?), or anywhere else if people have suggestions.
- KomoD 3y agoI'd definitely put it on Kaggle
- xnx 3y agoGood idea. I put it here: https://www.kaggle.com/datasets/nickalt/hacker-news https://www.kaggle.com/datasets/nickalt/hacker-news
- mrtimo 3y agoI thought you were joking about the 38 million requests! wow! Have you heard about .parquet files? Save it as a parquet and share the link.
- xnx 3y agohttps://www.kaggle.com/datasets/nickalt/hacker-news https://www.kaggle.com/datasets/nickalt/hacker-news
- mrtimo 3y agothis is amazing -- thanks. side note: when you want to get more data you can just get the new data, and save it as hacker-news-part-2.parquet - then when you use duckdb / malloy you can read them as a glob (hacker-news*.parquet) - and it is if you were reading one file.
- xnx 3y agoGreat tip. I used this little project as an exercise to some of learn DuckDB and Parquet. Still thinking about the best tradeoff for file format. An extra complicating factor is that items can change after they are posted so to get the most recent data, it's a good idea to re-request at least some range of historical items.