5 ms·
It’s 100x faster, works on larger than memory data and let’s you run SQL over csvs, parquet, arrow tables and etc. I mostly use it to extract data from a produ
by m_ke 4y ago
It’s 100x faster, works on larger than memory data and let’s you run SQL over csvs, parquet, arrow tables and etc.
I mostly use it to extract data from a production DB, do basic preprocessing for ML training (which requires joins and filtering that gets ugly in pandas) and typical OLAP queries to compute statistics.
- danielmarkbruce 4y agoGotcha. Out of interest, have you ever done this with sqlite before? I do something similar with sqlite, but speed isn't a factor for me usually because the data is small.
- m_ke 4y agoNo because importing into sqlite can be pretty slow. With duckdb you can query parquet files directly.
- danielmarkbruce 4y agoOk, I see that. Many thanks for the insight.