3 ms·

My data science team evaluated Polars and came back with a mixed bag of results. If there was any performance-critical section, then we would consider employing

My data science team evaluated Polars and came back with a mixed bag of results. If there was any performance-critical section, then we would consider employing it, but otherwise it was marginal negative given the overhead of replacing Pandas across dozens of projects.

gcarvalho 3y ago
I think that's the right call. IMHO now is the time to experiment with it, not to replace pandas where it's already working. The API is still seeing some (expected) breaking changes, and could become a maintenance burden across multiple projects. But the API already feels more consistent, and overall seems to be going in the right direction.
- asqueella 3y ago
  You made me curious to look up the recent breaking releases https://github.com/pola-rs/polars/releases/tag/py-0.19.0 https://github.com/pola-rs/polars/releases/tag/py-0.19.0 https://github.com/pola-rs/polars/releases/tag/py-0.20.0 https://github.com/pola-rs/polars/releases/tag/py-0.20.0 And their policy about it: https://docs.pola.rs/development/versioning/#deprecation-warnings https://docs.pola.rs/development/versioning/#deprecation-war... Looks like you ought to set aside some time to do updates each quarter, but I do wonder how much breakage can there be in practice, most changes seem pretty niche.
choppaface 3y ago
does your team use pandasql or mainly the direct pandas api? curious
theLiminator 3y ago
Definitely don't rewrite all the code, I think it's worth adopting or evaluating in new code though. Especially with the very cheap pandas interop, you can zero-copy to pandas pretty easily if you use the arrow backend.
esafak 3y ago
Going forward: https://github.com/fugue-project/fugue https://github.com/fugue-project/fugue
- nerdponx 3y ago
  Do you know if this project is collaborating with the people behind the "standard data frame API"? https://data-apis.org/dataframe-api/draft/index.html https://data-apis.org/dataframe-api/draft/index.html
  - esafak 3y ago
    I do not know but it does not appear so, based on a quick investigation. I had not heard of this API effort myself.

gcarvalho 3y ago

I think that's the right call. IMHO now is the time to experiment with it, not to replace pandas where it's already working. The API is still seeing some (expected) breaking changes, and could become a maintenance burden across multiple projects. But the API already feels more consistent, and overall seems to be going in the right direction.

asqueella 3y ago
You made me curious to look up the recent breaking releases https://github.com/pola-rs/polars/releases/tag/py-0.19.0 https://github.com/pola-rs/polars/releases/tag/py-0.19.0 https://github.com/pola-rs/polars/releases/tag/py-0.20.0 https://github.com/pola-rs/polars/releases/tag/py-0.20.0 And their policy about it: https://docs.pola.rs/development/versioning/#deprecation-warnings https://docs.pola.rs/development/versioning/#deprecation-war... Looks like you ought to set aside some time to do updates each quarter, but I do wonder how much breakage can there be in practice, most changes seem pretty niche.

choppaface 3y ago

does your team use pandasql or mainly the direct pandas api? curious

theLiminator 3y ago

Definitely don't rewrite all the code, I think it's worth adopting or evaluating in new code though. Especially with the very cheap pandas interop, you can zero-copy to pandas pretty easily if you use the arrow backend.

esafak 3y ago

Going forward: https://github.com/fugue-project/fugue https://github.com/fugue-project/fugue

nerdponx 3y ago
Do you know if this project is collaborating with the people behind the "standard data frame API"? https://data-apis.org/dataframe-api/draft/index.html https://data-apis.org/dataframe-api/draft/index.html
- esafak 3y ago
  I do not know but it does not appear so, based on a quick investigation. I had not heard of this API effort myself.