3 ms·
duckdb is really competing with things like snowflake, where compute is separate from storage. Loading data into postgres and tuning it can be a pain in the ass
by m_ke 4y ago
duckdb is really competing with things like snowflake, where compute is separate from storage. Loading data into postgres and tuning it can be a pain in the ass, with duckdb I can query huge parquet datasets in cloud storage without spinning up a VM.
It already replaced pandas and dask for most of my common data eng tasks.
- ttunguz 4y agoYes, that's right.
- mr_toad 4y ago> I can query huge parquet datasets in cloud storage without spinning up a VM. You’re querying cloud data locally? Wouldn’t egress costs get expensive quickly?
- danielmarkbruce 4y agoYeah, I thought the exact same thing.
- danielmarkbruce 4y ago>> it already replaced pandas and dask for most of my common data eng tasks This is interesting - why did you replace pandas/dask with this? What are you getting out of it?
- m_ke 4y agoIt’s 100x faster, works on larger than memory data and let’s you run SQL over csvs, parquet, arrow tables and etc. I mostly use it to extract data from a production DB, do basic preprocessing for ML training (which requires joins and filtering that gets ugly in pandas) and typical OLAP queries to compute statistics.
- danielmarkbruce 4y agoGotcha. Out of interest, have you ever done this with sqlite before? I do something similar with sqlite, but speed isn't a factor for me usually because the data is small.
- m_ke 4y agoNo because importing into sqlite can be pretty slow. With duckdb you can query parquet files directly.
- danielmarkbruce 4y agoOk, I see that. Many thanks for the insight.