3 ms·

Use polars it’s faster

• simonw 3y ago
  Can it run aggregate queries against 8 GiB of remotely hosted data while only pulling down the 287 MiB of columns it needs?
  • chrisjc 3y ago
    As much as I love to tout the benefits of duckdb, the ability to only pull down the columns that you need is not really functionality that is enabled by or exclusive to duckdb. It really has to do with the parquet file format. And it gets even more powerful than just being able to download specific columns. To a certain degree, you can also limit rows at the same time. http://peter-hoffmann.com/2020/understand-predicate-pushdown-on-rowgroup-level-in-parquet-with-pyarrow-and-python.html http://peter-hoffmann.com/2020/understand-predicate-pushdown... All that being said... yes, as far as I know, polars can do all of that. Probably because it has the necessary parquet dependencies. Now whether you should use polars instead of duckdb or vise-versa? Is polars "faster"? It depends
  • cmdlineluser 3y ago
    Looks like somebody requested it after reading your TIL. https://github.com/pola-rs/polars/issues/12493#issuecomment-1814763393 https://github.com/pola-rs/polars/issues/12493#issuecomment-... It will be in the next release. (later today?)
    • simonw 3y ago
      Wow that was fast! https://github.com/pola-rs/polars/pull/12517 https://github.com/pola-rs/polars/pull/12517
      • cmdlineluser 3y ago
        Nice :-D Do you happen to know what happened to this thread? (I recall reading a previous post of yours about tracking HN positions?) I also asked as a top-level comment: https://news.ycombinator.com/item?id=38304938 https://news.ycombinator.com/item?id=38304938
• pjot 3y ago
  DuckDB will work with polars https://duckdb.org/docs/archive/0.9.2/guides/python/polars.html https://duckdb.org/docs/archive/0.9.2/guides/python/polars.h...