3 ms·

I'm currently getting dragged into "data" stuff, and I get the impression it's a parallel universe, with its own background and culture. A lot of stuff is like

by Joeboy 3y ago

I'm currently getting dragged into "data" stuff, and I get the impression it's a parallel universe, with its own background and culture. A lot of stuff is like "connect to your Antelope or Meringue instances with the usability of Nincompoop and the performance of ARSE2". Anyway, probably the interesting things about polars are that it's like pandas, but uses a more efficient rust "backend" called Arrow (although I think that part's also in pandas now) and something like a "query planner" that makes combining operations more efficient. Typically doing things in polars is much more efficient than pandas, to the extent that things that previously required complicated infrastructure can often be done on a single machine. It's a very friendly competition, created by the main developer of pandas. As far as I can tell everybody loves it and it'll probably supplant pandas over time.

Galanwe 3y ago
> As far as I can tell everybody loves it and it'll probably supplant pandas over time. I've been using pandas heavily, everyday, for something like 8 years now. I also did contribute to it, as well as wrote numpy extensions. That is to say, I'm fairly familiar with the pandas/numpy ecosystem, strengths and weaknesses. Polars is a breeze of fresh air. The API of pandas is a mess: * overuse of polymorphic parameters and return types (functions that accept lists, ndarrays, dataframes, series or scalar) and return differently shaped dataframes or series. * lots of indirections behind layers of trampoline functions that hide default values behind undocumented "=None" default values. * abuse of half baked immutable APIs, favoring "copy everything" style of coding, mixed with half supported, should-have-been-deprecated, in-place variants. * lots and lots of regressions at every new release, of the worst kind ("oh yeah we changed the behavior of function X when there is more than Y NaNs over the window") * Very hard to actually know what is delegated to numpy, what is Cython/pandas, and what is pure python/pandas. * Overall the beast seemed to have won against its masters, and the maintainers seem lost as to what to fix versus what to keep backward compatible. Polars fixes a lot of these issues, but it has some shortcomings as well. Mainly I found that: * the API is definitely more consistent, but also more rigid than pandas. Some things can be very verbose to write. It will take some years for nicer simpler "shortcuts" and patterns to emerge. * The main issue IMHO is polars' handling of cross sectional (axis=1) computations. Polars is _very_ time series (axis=0) oriented, and most cross sectional computations require to transpose the data frame, which is very slow. Pandas has a lot of dedicated axis=1 implementations that avoid a full transposition.
- ritchie46 3y ago
  Many axis=1 operations in pandas do a transpose under the hood, mind you. Axis=1 belongs in matrices, not in heterogeneous data. They are a performance footgun. We make the transpose explicit.
  - Galanwe 3y ago
    > Many axis=1 operations in pandas do a transpose under the hood, mind you Sure, but many others are natively axis=1-aware and avoid full transposition. > Axis=1 belongs in matrices, not in heterogeneous data. I'm not sure to understand what that means. Care to elaborate? > They are a performance footgun. You don't get to only solve the problems that are efficient to solve... > We make the transpose explicit. Yes, but when you do mixed time series / cross sectional computations, you cannot always untangle both dimensions and transpose once. Sometimes your computation intrinsicely interleaves cross sectional and time series. In these case, which happen a lot in financial computations, then explicitly fully transposing is very slow.
btbuildem 3y ago
All domains seem to have this kind of in-group shorthand, regardless of scale of the community.
flexagoon 3y ago
> A lot of stuff is like "connect to your Antelope or Meringue instances with the usability of Nincompoop and the performance of ARSE2". I've been recently getting into DevOps stuff, and this is exactly what that sounds like to me. Same thing whenever I look at any new web framework: Okay, cool, so your product brings the power of "WazzupJS" to "the edge"... But it also has "teleports" like "Blimper" but without all of the downsides of "Vormhole Blues"? I'm sure that's really useful, but I kinda wish I knew what the product actually does
sagarm 3y ago
Arrow is an in-memory data format designed to allow zero-copy interop between languages, including Rust, C++, and Python. It's a bit more sophisticated than just some arrays, but ultimately everything is just arrays. Polars is implemented in Rust and uses Arrow to represent some data in memory. Pandas is often used in a way that results in a lot of processing happening in Python. Data processing in Python is comically inefficient and single-threaded. Beating that sort of pipeline by 10-100x is not that difficult if you have optimization experience and are able to work in a more suitable language.

by Joeboy 3y ago

Galanwe 3y ago

> As far as I can tell everybody loves it and it'll probably supplant pandas over time. I've been using pandas heavily, everyday, for something like 8 years now. I also did contribute to it, as well as wrote numpy extensions. That is to say, I'm fairly familiar with the pandas/numpy ecosystem, strengths and weaknesses. Polars is a breeze of fresh air. The API of pandas is a mess: * overuse of polymorphic parameters and return types (functions that accept lists, ndarrays, dataframes, series or scalar) and return differently shaped dataframes or series. * lots of indirections behind layers of trampoline functions that hide default values behind undocumented "=None" default values. * abuse of half baked immutable APIs, favoring "copy everything" style of coding, mixed with half supported, should-have-been-deprecated, in-place variants. * lots and lots of regressions at every new release, of the worst kind ("oh yeah we changed the behavior of function X when there is more than Y NaNs over the window") * Very hard to actually know what is delegated to numpy, what is Cython/pandas, and what is pure python/pandas. * Overall the beast seemed to have won against its masters, and the maintainers seem lost as to what to fix versus what to keep backward compatible. Polars fixes a lot of these issues, but it has some shortcomings as well. Mainly I found that: * the API is definitely more consistent, but also more rigid than pandas. Some things can be very verbose to write. It will take some years for nicer simpler "shortcuts" and patterns to emerge. * The main issue IMHO is polars' handling of cross sectional (axis=1) computations. Polars is _very_ time series (axis=0) oriented, and most cross sectional computations require to transpose the data frame, which is very slow. Pandas has a lot of dedicated axis=1 implementations that avoid a full transposition.

ritchie46 3y ago
Many axis=1 operations in pandas do a transpose under the hood, mind you. Axis=1 belongs in matrices, not in heterogeneous data. They are a performance footgun. We make the transpose explicit.
- Galanwe 3y ago
  > Many axis=1 operations in pandas do a transpose under the hood, mind you Sure, but many others are natively axis=1-aware and avoid full transposition. > Axis=1 belongs in matrices, not in heterogeneous data. I'm not sure to understand what that means. Care to elaborate? > They are a performance footgun. You don't get to only solve the problems that are efficient to solve... > We make the transpose explicit. Yes, but when you do mixed time series / cross sectional computations, you cannot always untangle both dimensions and transpose once. Sometimes your computation intrinsicely interleaves cross sectional and time series. In these case, which happen a lot in financial computations, then explicitly fully transposing is very slow.

btbuildem 3y ago

All domains seem to have this kind of in-group shorthand, regardless of scale of the community.

flexagoon 3y ago

> A lot of stuff is like "connect to your Antelope or Meringue instances with the usability of Nincompoop and the performance of ARSE2". I've been recently getting into DevOps stuff, and this is exactly what that sounds like to me. Same thing whenever I look at any new web framework: Okay, cool, so your product brings the power of "WazzupJS" to "the edge"... But it also has "teleports" like "Blimper" but without all of the downsides of "Vormhole Blues"? I'm sure that's really useful, but I kinda wish I knew what the product actually does

sagarm 3y ago

Arrow is an in-memory data format designed to allow zero-copy interop between languages, including Rust, C++, and Python. It's a bit more sophisticated than just some arrays, but ultimately everything is just arrays. Polars is implemented in Rust and uses Arrow to represent some data in memory. Pandas is often used in a way that results in a lot of processing happening in Python. Data processing in Python is comically inefficient and single-threaded. Beating that sort of pipeline by 10-100x is not that difficult if you have optimization experience and are able to work in a more suitable language.