3 ms·
I thought you were joking about the 38 million requests! wow! Have you heard about .parquet files? Save it as a parquet and share the link.
by mrtimo 3y ago
I thought you were joking about the 38 million requests! wow! Have you heard about .parquet files? Save it as a parquet and share the link.
- xnx 3y agohttps://www.kaggle.com/datasets/nickalt/hacker-news https://www.kaggle.com/datasets/nickalt/hacker-news
- mrtimo 3y agothis is amazing -- thanks. side note: when you want to get more data you can just get the new data, and save it as hacker-news-part-2.parquet - then when you use duckdb / malloy you can read them as a glob (hacker-news*.parquet) - and it is if you were reading one file.
- xnx 3y agoGreat tip. I used this little project as an exercise to some of learn DuckDB and Parquet. Still thinking about the best tradeoff for file format. An extra complicating factor is that items can change after they are posted so to get the most recent data, it's a good idea to re-request at least some range of historical items.