5 ms·
They should ship parquet, supports predicate pushdown and is dramatically more compact, while you can't get row level data, it's great for analytical queries.
by theLiminator 3y ago
They should ship parquet, supports predicate pushdown and is dramatically more compact, while you can't get row level data, it's great for analytical queries.
- IanCal 3y agoIMO this is the best simple option right now. For sorted data you only need the relevant row groups which can be tunable to sensible sizes for your data and access pattern.
- RobinL 3y agoTotally agree! So much that I wrote a whole article about it a while back! "Why parquet files are my preferred API for bulk open data" https://www.robinlinacre.com/parquet_api/ https://www.robinlinacre.com/parquet_api/
- swader999 3y agoParquet is a level better than csv but difficult to get customers to adopt and transmit in that format.
- calpaterson 3y agocsvbase does ship parquet! Just add ".parquet" - eg https://csvbase.com/meripaterson/stock-exchanges.parquet https://csvbase.com/meripaterson/stock-exchanges.parquet
- theLiminator 3y agoNice!