3 ms·
I mostly ignored Parquet until I realized DuckDB could handle it - now I'm much more likely to poke around with a Parquet file because I've got a solid toolkit
by simonw 4y ago
I mostly ignored Parquet until I realized DuckDB could handle it - now I'm much more likely to poke around with a Parquet file because I've got a solid toolkit available in both the CLI and as a Python library.
A couple of notes I've made about that:
Using DuckDB in Python to access Parquet data: https://til.simonwillison.net/duckdb/parquet https://til.simonwillison.net/duckdb/parquet
Use DuckDB to convert parquet to JSON and then open it in Datasette Lite: https://til.simonwillison.net/duckdb/parquet-to-json https://til.simonwillison.net/duckdb/parquet-to-json
- hackan 4y agoyou can also use pandas in Python to access parquet data, and then i.e. convert it to sqlite
- 0cf8612b2e1e 4y agoNaive pandas is going to limit you to RAM sizes when parquet is deliberately meant for huge datasets.
- FridgeSeal 4y agoAnyone reading who is currently using pandas and running into this issue-polars is generally faster, and now supports (still new though) out-of-core workloads.