3 ms·
Polars seems nice but in my experience using it, the "lazy" APIs would still immediately materialize a ton of stuff in memory and had very spotty support on wha
by viccis 14d ago
Polars seems nice but in my experience using it, the "lazy" APIs would still immediately materialize a ton of stuff in memory and had very spotty support on what data formats and storage integrations were possible with scan_* functions (though that was half a year ago and the support is slowly improving). It's frustrating, I mean really frustrating, to think I could solve a lot of my "scan through heinous amounts of data without any memory hungry things like window aggregations without blowing out my memory" with Polars and then watch my scan_thisorthat() call result in instant memory usage ballooning.
DuckDB on the other hand is wonderful and truly doesn't use any more memory than it really needs to.
- __mharrison__ 14d agoDid you ever report these issues to polars developers? I reported many bugs to both pandas and polars over the years and both teams tend to address relatively big ones. (Still have some outstanding pandas bugs that I think are a big deal but the devs disagree.)
- viccis 14d agoYeah, was in the Discord for a while
- thijsn 13d agoThat sounds like there's something going wrong, because that definitely shouldn't be the case! Although it sound like you've given up on it: if you have any example to send me in the right direction I can look into that for you. I'd love to know if there's something I can relay to the team. Thijs, DevRel @ Polars