4 ms·
Polars is cool, but man, I really have come to think that dataframes are disastrous for software. The mess of internal state and confusion of writing functions
by spenczar5 3y ago
Polars is cool, but man, I really have come to think that dataframes are disastrous for software. The mess of internal state and confusion of writing functions that take “df” and manipulate it - its all so hard to clean up once you’re deep in the mess.
Quivr (https://github.com/spenczar/quivr https://github.com/spenczar/quivr) is an alternative approach that has been working for me. Maybe types are good!
- radus 3y agoTotally agree with the critique, though it bears mentioning that one way that polars differentiates itself from pandas is the expressions API. See [1] for an example. 1: https://kevinheavey.github.io/modern-polars/method_chaining.html#extract-city-names https://kevinheavey.github.io/modern-polars/method_chaining....
- theLiminator 3y agoPolars is a lot better than pandas at maintaining valid state. Because you ideally describe everything in terms of lazy operations, it actually internally keeps track of all your data types at every step until materialization and execution of the query plan. Because of that, you're not going to have the same kind of data type issues you might have in pandas. There are also libraries based off pydantic built for polars dataframes validation (see patito, though it's not mature yet).
- efxhoy 3y agoYes, the dataframe is very tricky to get right function signature wise. I used to write a lot of pandas heavy software and converged on writing most functions to take a dataframe or series and when possible just return a series, then put that series back into the df at the function call site. Handing dfs off to functions that mutate them gets gnarly very quickly.