3 ms·
As much as I love to tout the benefits of duckdb, the ability to only pull down the columns that you need is not really functionality that is enabled by or excl
by chrisjc 3y ago
As much as I love to tout the benefits of duckdb, the ability to only pull down the columns that you need is not really functionality that is enabled by or exclusive to duckdb.
It really has to do with the parquet file format. And it gets even more powerful than just being able to download specific columns. To a certain degree, you can also limit rows at the same time.
http://peter-hoffmann.com/2020/understand-predicate-pushdown-on-rowgroup-level-in-parquet-with-pyarrow-and-python.html http://peter-hoffmann.com/2020/understand-predicate-pushdown...
All that being said... yes, as far as I know, polars can do all of that. Probably because it has the necessary parquet dependencies.
Now whether you should use polars instead of duckdb or vise-versa? Is polars "faster"? It depends