3 ms·
We use DuckDB similarly but productionize by writing pyarrow code. All the modern tools (DuckDB, pyarrow, polars) are fast enough if you store your data well (p
by singhrac 2y ago
We use DuckDB similarly but productionize by writing pyarrow code. All the modern tools (DuckDB, pyarrow, polars) are fast enough if you store your data well (parquet), though we work with not quite “big data” most of the time.
It’s worth remembering that all the modern progress builds on top of years of work by Wes McKinney & co (many, many contributors).
- wenc 2y agoYes Wes McKinney was involved in both Pandas and Parquet and Arrow.
- 7thaccount 2y agoI remember reading something awhile back that when building pandas he was getting a lot of inspiration from things like APL and I assume Kdb+.