3 ms·

Ive found polars quite intuitive, though for python, I lean more towards [ibis](https://ibis-project.org/ https://ibis-project.org/). The interface is nearly id

by 0thgen 3y ago

Ive found polars quite intuitive, though for python, I lean more towards [ibis](https://ibis-project.org/ https://ibis-project.org/). The interface is nearly identical, but ibis has the benefit if building sql queries before pulling any actual data (like dbplyr) — whereas polars requires the data to be in-memory (at least for rdb’s, though correct me if Im wrong) this to me seems like a good argument for only using ibis, but Im happy to be convinced otherwise

oofbey 3y ago
If it all fits in memory, isn't it simpler to just load it first? Sure, you're kinda painting yourself into a corner, but it can be a big corner. e.g. I know that until recently Amazon still ran a bunch of key recommendation algorithms on a single big machine each night, because it's just simpler to load everything into memory and crank on it.
- hn72774 3y ago
  When reading from the dataframe, depending on the columns, filters, and aggregates you want returned, most query engines can skip data that is irrelevant, making the query run much faster. In your amazon example the data is probably optimized for fast lookups on a small number of records, like an in-memory cache.
- theLiminator 3y ago
  Simpler, but you can avoid huge amounts of work just simply by not loading what you need. That's the beauty of the scan_* apis in polars. Most query engines have some level of support for io skipping.
quadrature 3y ago
I haven't used it but polars has a lazy api https://docs.pola.rs/user-guide/lazy/using/#using-the-lazy-api-from-a-dataframe https://docs.pola.rs/user-guide/lazy/using/#using-the-lazy-a...
ritchie46 3y ago
Polars doesn't require all data to be in memory. It has a lazy API, optimizations that prune data at the scan level and parts of the engine can process data in batches. Algorithms like joins, group bys, distinct, etc, are designed for out-of-core processing and can spill to disk if available RAM is low.
theLiminator 3y ago
What backend do you use? The little spin I've given Ibis felt like it's not deeply supported with a lot of the backends.

by 0thgen 3y ago

oofbey 3y ago

If it all fits in memory, isn't it simpler to just load it first? Sure, you're kinda painting yourself into a corner, but it can be a big corner. e.g. I know that until recently Amazon still ran a bunch of key recommendation algorithms on a single big machine each night, because it's just simpler to load everything into memory and crank on it.

hn72774 3y ago
When reading from the dataframe, depending on the columns, filters, and aggregates you want returned, most query engines can skip data that is irrelevant, making the query run much faster. In your amazon example the data is probably optimized for fast lookups on a small number of records, like an in-memory cache.
theLiminator 3y ago
Simpler, but you can avoid huge amounts of work just simply by not loading what you need. That's the beauty of the scan_* apis in polars. Most query engines have some level of support for io skipping.

quadrature 3y ago

I haven't used it but polars has a lazy api https://docs.pola.rs/user-guide/lazy/using/#using-the-lazy-api-from-a-dataframe https://docs.pola.rs/user-guide/lazy/using/#using-the-lazy-a...

ritchie46 3y ago

Polars doesn't require all data to be in memory. It has a lazy API, optimizations that prune data at the scan level and parts of the engine can process data in batches. Algorithms like joins, group bys, distinct, etc, are designed for out-of-core processing and can spill to disk if available RAM is low.

theLiminator 3y ago

What backend do you use? The little spin I've given Ibis felt like it's not deeply supported with a lot of the backends.