5 ms·
One thing that has changed the game with how I work with CSVs is ClickHouse. It is trivially easy to run a local database, import CSV files into a table, and ru
by primitivesuave 2y ago
One thing that has changed the game with how I work with CSVs is ClickHouse. It is trivially easy to run a local database, import CSV files into a table, and run blazing-fast queries on it. If you leave the data there, ClickHouse will gradually optimize the compression. It's pretty magical stuff if you work in data science.
- tgtweak 2y agoI feel the same way about elastic. That being said I noticed .parquet as an export format option on Shopify recently and an hopeful more providers offer the choice.
- emmelaich 2y agoSimon W's https://datasette.io/ https://datasette.io/ is also excellent.
- primitivesuave 2y agoDatasette is a wonderful tool that I've used before, and I have the highest admiration for its creator, but the underlying Sqlite3 database doesn't handle large datasets (i.e. hundreds of millions of rows) nearly as well as ClickHouse does. It's worth noting that I only ran into this limitation when working with huge federal campaign finance datasets [1] and trying to do some compute-intensive querying. For 99% of use cases, datasette is a similarly magical piece of software for quickly exploring some CSV files. 1. https://www.fec.gov/data/browse-data/?tab=bulk-data https://www.fec.gov/data/browse-data/?tab=bulk-data