4 ms·

Curious if you considered Polars. That's become the defacto standard in my group as we all dislike pandas.

cpcloud 2y ago
We support Polars as a backend. DuckDB was chosen as the default because Polars was too unstable to be the default backend two years ago.
amelius 2y ago
Good riddance. Pandas turns 10x programmers into 1x programmers.
- data-ottawa 2y ago
  I've been using pandas for over a decade and I'm very efficient with it. The Modern Pandas blog series from Tom Augsburger is excellent, and the old Wes McKinney book (creator of pandas) gave a good glimpse into how to effectively use it. I don't think many people learn it correctly so it gets used in a very inefficient and spaghetti script style. It's definitely a tool that's showing it's age, but it's still a very effective swiss army knife for data processing. I am glad we're getting better alternatives, Hadley Wickams tidyverse really showed what data processing can be.
  - codyvoda 2y ago
    fun fact if you weren't aware, Ibis was also originally created by Wes McKinney and takes a lot of inspiration from the tidyverse
    - data-ottawa 2y ago
      I was not aware of that, the influence from the tidyverse is very clear I’ve tried to make mockups of similarly inspired API and I think they did a great job with clever tricks like the `_` reference (does that clash with IPython/Jupyter’s use?). I do wonder how they manage compatibility with so many backends, it seems like many features will be directly tied to your backend (e.g. Trino using Java Pattern for regex vs BigQuery using Re2) in hard to explain ways. But maybe that’s not a big concern, because very often you’re only going to be using a couple of backends. I’ll have to try Ibis out for myself, it looks like it can unify a lot of the work I have to do. I’ve moved away from pandas for all but the last mile of computation, so this might be a direct good SQL replacement.
      - cpcloud 2y ago
        `_` definitely clashes with IPython and Jupyter's use of it. However, if you import it from Ibis then it ceases to be used as "most recent result" and remains the ibis underscore object (unless of course you explicitly assign it to something else). Regarding backend compatibility, there are definitely a few kinds of things that we don't currently abstract over. One is regular expression syntax and another is floating point math (e.g., various algebraic properties that are violated that result in slightly different outputs). Hope you give it a go, and please report issues at https://github.com/ibis-project/ibis https://github.com/ibis-project/ibis.
        data-ottawa 2y ago
        Thanks for the reply! I think that’s the right policy to take, and I did notice the support matrix on your website which addresses my earlier question: https://ibis-project.org/backends/support/matrix https://ibis-project.org/backends/support/matrix
  - throwway_278314 2y ago
    Just imagine how much more efficient you would be if you were using R's DataTable. Look, I applaud your skill, but at some point even a master craftsman realizes that the swiss army knife may not be the best tool, and a leatherman offers certain advantages.
    - cpcloud 2y ago
      Sometimes I write data analysis code in R to make myself aware of how it is occasionally possible to have nice things.
    - data-ottawa 2y ago
      I really like R’s library and I’ll use them any chance I get (libraries like lmer are still orders of magnitude more efficient than the same model in Statsmodels). From my experience the biggest impediment to using R in production is many orgs don’t have a blessed way to run it. R is my favourite language for data processing, the manual section Computing on the Language[1]is why R is such an ergonomic tool. I had hoped Julia would catch up, but Julia’s macros are not comparable in their depth. I think pandas is probably the data equivalent of editing files using default vim or processing data with awk. [1] https://rstudio.github.io/r-manuals/r-lang/Computing-on-the-language.html https://rstudio.github.io/r-manuals/r-lang/Computing-on-the-...
      - cpcloud 2y ago
        As a joke, I wrote an Ibis backend (https://github.com/cpcloud/ibish https://github.com/cpcloud/ibish) that processes expressions using shell commands strung together with named pipes. It supports joins using the coreutils join command, projections, filters and some aggregations with awk. It's faster than pandas in some cases and folks should put it into production immediately!
  - scrlk 2y ago
    I also recommend Matt Harrison's talks ("Idiomatic Pandas") and book ("Effective Pandas"). Once I adopted method chaining, a lot of the issues that I had with pandas in the past due to poor style (e.g. SettingWithCopyWarning) pretty much disappeared.
- __mharrison__ 2y ago
  Curious if you could expand on this?
- Kalanos 2y ago
  Lol. As if different tools for filtering rows/columns determines a programmer's capability.
fifilura 2y ago
What is the difference, other than parallel execution?
- magnio 2y ago
  - Much faster and more memory efficient. - Consistent Expression and SQL-like API. - Lazy execution mode, where queries are compiled and optimized before running. - Sane NaN and null handling. - Much faster. - Much more memory efficient.
  - cpcloud 2y ago
    Did it get more memory efficient during the time between authoring point 1 and authoring the last point?
    - dgfitz 2y ago
      You may think this was clever, but it is a common literary technique to emphasize a point. I'd suggest less snark, you're not doing yourself any favors.
      - itsoktocry 2y ago
        >I'd suggest less snark, you're not doing yourself any favors. Why do people feel the need to jump in and police tone like this? Who are you? You're not doing yourself any favours, either. >common literary technique to emphasize a point. "Common" is a stetch, and who cares.
        temp_praneshp 2y ago
        > Why do people feel the need to jump in and police tone like this? From the community guidelines(https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html): "Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes." > Who are you? A member of the community.
        2y ago
codyvoda 2y ago
Polars is a great choice but, like pandas, locks you into its execution engine(s). Polars is a supported backend for Ibis, though depending on your use case DuckDB or DataFusion may scale better. we did some benchmarking recently: https://ibis-project.org/posts/1tbc/ https://ibis-project.org/posts/1tbc/
- marcogorelli 2y ago
  Also depending on your use case, Ibis may not support the part of the Polars API you actually need (to be clear, I'm a fan of "DuckDB via Ibis", I'm much less a fan of statements to the effect of "use Polars via Ibis instead of Polars directly", as I think they miss a lot :wink:)
ang_cire 2y ago
I wouldn't switch to Polars as a replacement to Pandas, given that Polars also seems likely to die soon. I'd also recommend avoiding Manatees.
- highfrequency 2y ago
  Why do you think polars is likely to die soon?
  - dr_mee6 2y ago
    Global warming...
    - ang_cire 2y ago
      Sadly, I don't think many caught the joke :)
- cpcloud 2y ago
  On the flip side, Ibises (Ibisii ??) have been around for years and aren't going anywhere.

We support Polars as a backend. DuckDB was chosen as the default because Polars was too unstable to be the default backend two years ago.

amelius 2y ago

Good riddance. Pandas turns 10x programmers into 1x programmers.

data-ottawa 2y ago
I've been using pandas for over a decade and I'm very efficient with it. The Modern Pandas blog series from Tom Augsburger is excellent, and the old Wes McKinney book (creator of pandas) gave a good glimpse into how to effectively use it. I don't think many people learn it correctly so it gets used in a very inefficient and spaghetti script style. It's definitely a tool that's showing it's age, but it's still a very effective swiss army knife for data processing. I am glad we're getting better alternatives, Hadley Wickams tidyverse really showed what data processing can be.
- codyvoda 2y ago
  fun fact if you weren't aware, Ibis was also originally created by Wes McKinney and takes a lot of inspiration from the tidyverse
  - data-ottawa 2y ago
    I was not aware of that, the influence from the tidyverse is very clear I’ve tried to make mockups of similarly inspired API and I think they did a great job with clever tricks like the `_` reference (does that clash with IPython/Jupyter’s use?). I do wonder how they manage compatibility with so many backends, it seems like many features will be directly tied to your backend (e.g. Trino using Java Pattern for regex vs BigQuery using Re2) in hard to explain ways. But maybe that’s not a big concern, because very often you’re only going to be using a couple of backends. I’ll have to try Ibis out for myself, it looks like it can unify a lot of the work I have to do. I’ve moved away from pandas for all but the last mile of computation, so this might be a direct good SQL replacement.
    - cpcloud 2y ago
      `_` definitely clashes with IPython and Jupyter's use of it. However, if you import it from Ibis then it ceases to be used as "most recent result" and remains the ibis underscore object (unless of course you explicitly assign it to something else). Regarding backend compatibility, there are definitely a few kinds of things that we don't currently abstract over. One is regular expression syntax and another is floating point math (e.g., various algebraic properties that are violated that result in slightly different outputs). Hope you give it a go, and please report issues at https://github.com/ibis-project/ibis https://github.com/ibis-project/ibis.
      - data-ottawa 2y ago
        Thanks for the reply! I think that’s the right policy to take, and I did notice the support matrix on your website which addresses my earlier question: https://ibis-project.org/backends/support/matrix https://ibis-project.org/backends/support/matrix
- throwway_278314 2y ago
  Just imagine how much more efficient you would be if you were using R's DataTable. Look, I applaud your skill, but at some point even a master craftsman realizes that the swiss army knife may not be the best tool, and a leatherman offers certain advantages.
  - cpcloud 2y ago
    Sometimes I write data analysis code in R to make myself aware of how it is occasionally possible to have nice things.
  - data-ottawa 2y ago
    I really like R’s library and I’ll use them any chance I get (libraries like lmer are still orders of magnitude more efficient than the same model in Statsmodels). From my experience the biggest impediment to using R in production is many orgs don’t have a blessed way to run it. R is my favourite language for data processing, the manual section Computing on the Language[1]is why R is such an ergonomic tool. I had hoped Julia would catch up, but Julia’s macros are not comparable in their depth. I think pandas is probably the data equivalent of editing files using default vim or processing data with awk. [1] https://rstudio.github.io/r-manuals/r-lang/Computing-on-the-language.html https://rstudio.github.io/r-manuals/r-lang/Computing-on-the-...
    - cpcloud 2y ago
      As a joke, I wrote an Ibis backend (https://github.com/cpcloud/ibish https://github.com/cpcloud/ibish) that processes expressions using shell commands strung together with named pipes. It supports joins using the coreutils join command, projections, filters and some aggregations with awk. It's faster than pandas in some cases and folks should put it into production immediately!
- scrlk 2y ago
  I also recommend Matt Harrison's talks ("Idiomatic Pandas") and book ("Effective Pandas"). Once I adopted method chaining, a lot of the issues that I had with pandas in the past due to poor style (e.g. SettingWithCopyWarning) pretty much disappeared.
__mharrison__ 2y ago
Curious if you could expand on this?
Kalanos 2y ago
Lol. As if different tools for filtering rows/columns determines a programmer's capability.

fifilura 2y ago

What is the difference, other than parallel execution?

magnio 2y ago
- Much faster and more memory efficient. - Consistent Expression and SQL-like API. - Lazy execution mode, where queries are compiled and optimized before running. - Sane NaN and null handling. - Much faster. - Much more memory efficient.
- cpcloud 2y ago
  Did it get more memory efficient during the time between authoring point 1 and authoring the last point?
  - dgfitz 2y ago
    You may think this was clever, but it is a common literary technique to emphasize a point. I'd suggest less snark, you're not doing yourself any favors.
    - itsoktocry 2y ago
      >I'd suggest less snark, you're not doing yourself any favors. Why do people feel the need to jump in and police tone like this? Who are you? You're not doing yourself any favours, either. >common literary technique to emphasize a point. "Common" is a stetch, and who cares.
      - temp_praneshp 2y ago
        > Why do people feel the need to jump in and police tone like this? From the community guidelines(https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html): "Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes." > Who are you? A member of the community.
      - 2y ago

codyvoda 2y ago

Polars is a great choice but, like pandas, locks you into its execution engine(s). Polars is a supported backend for Ibis, though depending on your use case DuckDB or DataFusion may scale better. we did some benchmarking recently: https://ibis-project.org/posts/1tbc/ https://ibis-project.org/posts/1tbc/

marcogorelli 2y ago
Also depending on your use case, Ibis may not support the part of the Polars API you actually need (to be clear, I'm a fan of "DuckDB via Ibis", I'm much less a fan of statements to the effect of "use Polars via Ibis instead of Polars directly", as I think they miss a lot :wink:)

ang_cire 2y ago

I wouldn't switch to Polars as a replacement to Pandas, given that Polars also seems likely to die soon. I'd also recommend avoiding Manatees.

highfrequency 2y ago
Why do you think polars is likely to die soon?
- dr_mee6 2y ago
  Global warming...
  - ang_cire 2y ago
    Sadly, I don't think many caught the joke :)
cpcloud 2y ago
On the flip side, Ibises (Ibisii ??) have been around for years and aren't going anywhere.