4 ms·
Pandas doesn't. Polars I think has some lazy-loading capability, but it's not the default mode of operation and I don't think it supports all features. If DuckD
by nerdponx 4y ago
Pandas doesn't. Polars I think has some lazy-loading capability, but it's not the default mode of operation and I don't think it supports all features. If DuckDB doesn't, then that's a big advantage.
- jcheng 4y agoI think you mean that Pandas does require your entire data to fit in memory? https://pandas.pydata.org/docs/user_guide/scale.html https://pandas.pydata.org/docs/user_guide/scale.html
- nerdponx 4y agoYes, thank you, that was a typo.
- corford 4y agoApparently duckdb paired with arrow lets you work performatively on bigger than memory parquet data (I haven't tried this though)
- riezebos 4y agoArrow seems to be the common denominator for larger than ram. Polars is completely built on arrow, so the same advantages should apply. https://github.com/pola-rs/polars#handles-larger-than-ram-data https://github.com/pola-rs/polars#handles-larger-than-ram-da...