2 ms·
Laziness in this context has huge advantages in reducing memory allocation. Many operations can be fused together, so there's less of a need to allocate huge in
by mjhay 3y ago
Laziness in this context has huge advantages in reducing memory allocation. Many operations can be fused together, so there's less of a need to allocate huge intermediate data structures at every step.
- disgruntledphd2 3y agoyeah, totally, I can see that. I think that polars is the first library to do this locally, which is surprising if it has so many advantages.
- mjhay 3y agoIt's been around in R-land for a while with dplyr and its variety of backends (including Arrow, the same as Polars). Pandas is just an incredibly mediocre library in nearly all respects.
- disgruntledphd2 3y ago> It's been around in R-land for a while with dplyr and its variety of backends Only for SQL databases, so not really. Source: have been running dplyr since 2011.
- mjhay 3y agoThe Arrow backend does allow for lazy eval. https://arrow.apache.org/cookbook/r/manipulating-data---tables.html https://arrow.apache.org/cookbook/r/manipulating-data---tabl...