4 ms·
While polars is better if you work with predefined data formats, pandas is imo still better as a general purpose table container. I work with chemical datasets
by rdedev 8mo ago
While polars is better if you work with predefined data formats, pandas is imo still better as a general purpose table container.
I work with chemical datasets and this always involves converting SMILES string to Rdkit Molecule objects. Polars cannot do this as simply as calling .map on pandas.
Pandas is also much better to do EDA. So calling it worse in every instance is not true. If you are doing pure data manipulation then go ahead with polars
- data-ottawa 8mo agoMap is one operation pandas does nicely that most other “wrap a fast language” dataframe tools do poorly. When it feels like you’re writing some external udf thats executed in another environment, it does not feel as nice as throwing in a lambda, even if the lambda is not ideal.
- vegabook 8mo agoyou have map_elements in polars which does exactly this. https://docs.pola.rs/api/python/dev/reference/expressions/api/polars.Expr.map_elements.html https://docs.pola.rs/api/python/dev/reference/expressions/ap... You can also iter_rows into a lambda if you really want to. https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.iter_rows.html https://docs.pola.rs/api/python/stable/reference/dataframe/a... Personally I find it extremely rare that I need to do this given Polars expressions are so comprehensive, including when.then.otherwise when all else fails.
- data-ottawa 8mo agoThat one has a bit more friction than pandas because the return schema requirement -- pandas let's you get away with this bad practice. It also does batches when you declare scalar outputs, but you can't control the batch size, which usually isn't an issue, but I've run into situations where it is.
- deleted 8mo ago[deleted]