3 ms·

> As far as I can tell everybody loves it and it'll probably supplant pandas over time. I've been using pandas heavily, everyday, for something like 8 years no

> As far as I can tell everybody loves it and it'll probably supplant pandas over time. I've been using pandas heavily, everyday, for something like 8 years now. I also did contribute to it, as well as wrote numpy extensions. That is to say, I'm fairly familiar with the pandas/numpy ecosystem, strengths and weaknesses. Polars is a breeze of fresh air. The API of pandas is a mess: * overuse of polymorphic parameters and return types (functions that accept lists, ndarrays, dataframes, series or scalar) and return differently shaped dataframes or series. * lots of indirections behind layers of trampoline functions that hide default values behind undocumented "=None" default values. * abuse of half baked immutable APIs, favoring "copy everything" style of coding, mixed with half supported, should-have-been-deprecated, in-place variants. * lots and lots of regressions at every new release, of the worst kind ("oh yeah we changed the behavior of function X when there is more than Y NaNs over the window") * Very hard to actually know what is delegated to numpy, what is Cython/pandas, and what is pure python/pandas. * Overall the beast seemed to have won against its masters, and the maintainers seem lost as to what to fix versus what to keep backward compatible. Polars fixes a lot of these issues, but it has some shortcomings as well. Mainly I found that: * the API is definitely more consistent, but also more rigid than pandas. Some things can be very verbose to write. It will take some years for nicer simpler "shortcuts" and patterns to emerge. * The main issue IMHO is polars' handling of cross sectional (axis=1) computations. Polars is _very_ time series (axis=0) oriented, and most cross sectional computations require to transpose the data frame, which is very slow. Pandas has a lot of dedicated axis=1 implementations that avoid a full transposition.

ritchie46 3y ago
Many axis=1 operations in pandas do a transpose under the hood, mind you. Axis=1 belongs in matrices, not in heterogeneous data. They are a performance footgun. We make the transpose explicit.
- Galanwe 3y ago
  > Many axis=1 operations in pandas do a transpose under the hood, mind you Sure, but many others are natively axis=1-aware and avoid full transposition. > Axis=1 belongs in matrices, not in heterogeneous data. I'm not sure to understand what that means. Care to elaborate? > They are a performance footgun. You don't get to only solve the problems that are efficient to solve... > We make the transpose explicit. Yes, but when you do mixed time series / cross sectional computations, you cannot always untangle both dimensions and transpose once. Sometimes your computation intrinsicely interleaves cross sectional and time series. In these case, which happen a lot in financial computations, then explicitly fully transposing is very slow.

ritchie46 3y ago

Many axis=1 operations in pandas do a transpose under the hood, mind you. Axis=1 belongs in matrices, not in heterogeneous data. They are a performance footgun. We make the transpose explicit.

Galanwe 3y ago
> Many axis=1 operations in pandas do a transpose under the hood, mind you Sure, but many others are natively axis=1-aware and avoid full transposition. > Axis=1 belongs in matrices, not in heterogeneous data. I'm not sure to understand what that means. Care to elaborate? > They are a performance footgun. You don't get to only solve the problems that are efficient to solve... > We make the transpose explicit. Yes, but when you do mixed time series / cross sectional computations, you cannot always untangle both dimensions and transpose once. Sometimes your computation intrinsicely interleaves cross sectional and time series. In these case, which happen a lot in financial computations, then explicitly fully transposing is very slow.