4 ms·

I've always found pandas really hard to use or reason about. I eventually get there but I don't like the code. Obviously subjective. I've never used another "da

I've always found pandas really hard to use or reason about. I eventually get there but I don't like the code. Obviously subjective. I've never used another "data science" language though so I've no experience beyond it.

mjburgess 5y ago
Everyone does. Perhaps if python would add more support for FP (rather than its present hostility), we'd be able to phrase data transformations more naturally in the language.
- zasdffaa 5y ago
  My python is rusty but IIRC it allows functional stuff (except for expression-only lambdas, boo). From https://docs.python.org/3/howto/functional.html https://docs.python.org/3/howto/functional.html it's got map/filter/currying and plenty more, what's misting in your view?
  - mjburgess 5y ago
    pattern matching expressions, syntax for partial application and composition, a typing system which can express structural types, a generalised list comprehension
    - zasdffaa 5y ago
      > pattern matching expressions, nothing intrinsically 'functional' about this, though it's nice > syntax for partial application <https://docs.python.org/3/library/functools.html#functools.partial https://docs.python.org/3/library/functools.html#functools.p...> > and composition, Hmm, to my surprise I can't find this but TBH it's trivial to write, here's an eg <https://stackoverflow.com/questions/16739290/composing-functions-in-python https://stackoverflow.com/questions/16739290/composing-funct...> > a typing system which can express structural types, typing is orthogonal to FP (though very nice to have) > a generalised list comprehension IDK what this means, what's ungeneral about comprehensions currently?
      - mjburgess 5y ago
        expressing generic data transformations naturally in syntax, requires: being able to wrap/unwrap data types (ie., pattern matching), composing operations, partially applying them, and providing syntax to support filter/map/flatMap on arbitrary data structures Pythons failure to provide this, indeed, outright hostility to doing so is half the reason pandas is a mess of incomprehensible syntax By prioritising assignment expressions and implementing pattern "matching" as a statement, they're clearly showing a lot of hostility to one of the major use cases of their language (incidentally, recall that C# introduced LINQ in 2008, a generalised comprehension! That's C#.) There isnt the syntax to reimplement to support natural phrasings of data transformation, in lieu of this, pandas exploits weird operators such as `.loc[a,b]`. At some point the dam is going to have to break, and python is going to have to introduce something to resolve this mess. However, i'd bet it'll be a decade of tooth-and-nail fighting about it. It isnt a software engineering language for education any more, and i'm not seeing this reality being acknowledged
    - biellls 5y ago
      Pattern matching is supported in 3.10 and can match classes structurally, as well as other types (edit: I see that you mean you don't like that it's a statement, which I agree with). The typing system supports structural types with Protocol. Personally what I miss most are multi-line lambdas.
      - mjburgess 5y ago
        I'm not asking for "support" i'm asking for syntax. The question is why the syntax of data processing in python is illegible, not "are things technically possible in python". Commenters here are keen to say "but dont you know!" -- and, yes, I do.
        zasdffaa 5y ago
        To be fair the confusion of FP vs FP syntax is from your comment: "Perhaps if python would add more support for FP (rather than its present hostility)" Python does FP fine and FP via an ugly library or via elegant inbuilt syntax is still FP. I get you want nicer syntax but it wasn't clear. I dunno, write a PEP and see if it gets support. I hope you succeed!
    - ok_dad 5y ago
      > pattern matching expressions Available in 3.10: https://peps.python.org/pep-0636/ https://peps.python.org/pep-0636/ > syntax for partial application... from functools import partial somefunc_arg1_arg2 = partial(somefunc, arg1, arg2) > ...and composition A native compositional syntax would be nice. > a typing system which can express structural types # mypy will typecheck this code and see `MyString()` has a `.read()` method, so it's a `Readable` even though it doesn't sub-class `Readable` from typing import Any, Protocol class Readable(Protocol): def read(self) -> Any: ... def read_something(something: Readable) -> Any: return something.read() class MyString: a_string: str = "something" def read(self) -> str: return self.a_string read_something(MyString()) > a generalised list comprehension I agree this would be nice.
      - mjburgess 5y ago
        they arent expressions, defeating the whole point that isnt syntax for partial appication, and the protocol system for structrural typing is syntactically and practically absurd the question is "why do data processing libs in python look syntactically illegible" and the answer is largely, as youve posted above, that python doesnt support useful syntax
tomtom1337 5y ago
I'd recommend checking out polars as an alternative to pandas - https://github.com/pola-rs/polars https://github.com/pola-rs/polars It has a rather different api, and is significantly faster. Highly recommend it.
- riyadparvez 5y ago
  The issue is there's a huge ecosystem developed on top of Pandas. So if it's different API, now you lose access to all the ecosystem.
  - nojito 5y ago
    Polars can export to arrow which you can then load into pandas. Or you can convert it directly into a pandas Df
  - ritchie46 5y ago
    Should not be a real blocker. There is good interop with arrow, numpy and pandas. Where arrow and numpy mostly is zero-copy. So you are one `df.to_numpy()/df.to_pandas()` away to `X` libary you want to use.
animal_spirits 5y ago
Pandas makes it really hard to do easy things, and really easy to do hard things. But its more useful to the people who need it to do the hard things.
biellls 5y ago
A lot of people hate on SQL. I used to be one of them but I've come to think that for data transformations it's hard to beat. My current favorite is DuckDB, which is like a SQLite but columnar. It has great performance and it's easy to call it from python and even run SQL on pandas dataframes.
otsaloma 5y ago
Here's another alternative. I wrote Dataiter specifically as I too was frustrated with Pandas. In my experience if you design a new API from scratch (and don't try to reimplement the Pandas API as many projects have done!) and have some vision and consistent principles, it's well possible to get a good intuitive API as a result. Two relevant issues remain: You're limited by NumPy's datatypes and their problems, such as memory-hogging strings and a lack of a proper missing value (NA), and secondly, limited by the Python language, so compared to e.g. dplyr's non-standard evaluation, you'll need to use lambda functions, which are unfortunately clumsy and verbose. https://github.com/otsaloma/dataiter https://github.com/otsaloma/dataiter Here's a comparison of dplyr vs. Dataiter vs. Pandas, which should give quick overview of the similarieties and differences. https://dataiter.readthedocs.io/en/latest/_static/comparison.html https://dataiter.readthedocs.io/en/latest/_static/comparison...
smartmic 5y ago
If you are ready to stretch your mind and have time and energy to learn a new skill, I recommend this as "data science" language: APL Background: I first gained some experience with J, where I first learned to appreciate the advantages of array languages. The main advantage is actually having your own way of thinking about processing multidimensional data. Now, recently, I've gotten into the notation of APL, and it's really even cooler than the ASCII "noise" of J. The symbols make it easier for me to both write and read programs. Admittedly, for more complex operations with data it takes a lot of learning that you usually don't have. But for simple transformation APL is already quite fast usable and convinces beyond the "mainstream".
- account-5 5y ago
  APL and J are on my list of programming languages I'd like to learn. I've been listening to the Array Cast podcast for a while. I just need a place to start. I don't really learn a language until I use it for something, at the moment I have nothing to use APL for. Any recommendations of where to start would be welcome.

mjburgess 5y ago

Everyone does. Perhaps if python would add more support for FP (rather than its present hostility), we'd be able to phrase data transformations more naturally in the language.

zasdffaa 5y ago
My python is rusty but IIRC it allows functional stuff (except for expression-only lambdas, boo). From https://docs.python.org/3/howto/functional.html https://docs.python.org/3/howto/functional.html it's got map/filter/currying and plenty more, what's misting in your view?
- mjburgess 5y ago
  pattern matching expressions, syntax for partial application and composition, a typing system which can express structural types, a generalised list comprehension
  - zasdffaa 5y ago
    > pattern matching expressions, nothing intrinsically 'functional' about this, though it's nice > syntax for partial application <https://docs.python.org/3/library/functools.html#functools.partial https://docs.python.org/3/library/functools.html#functools.p...> > and composition, Hmm, to my surprise I can't find this but TBH it's trivial to write, here's an eg <https://stackoverflow.com/questions/16739290/composing-functions-in-python https://stackoverflow.com/questions/16739290/composing-funct...> > a typing system which can express structural types, typing is orthogonal to FP (though very nice to have) > a generalised list comprehension IDK what this means, what's ungeneral about comprehensions currently?
    - mjburgess 5y ago
      expressing generic data transformations naturally in syntax, requires: being able to wrap/unwrap data types (ie., pattern matching), composing operations, partially applying them, and providing syntax to support filter/map/flatMap on arbitrary data structures Pythons failure to provide this, indeed, outright hostility to doing so is half the reason pandas is a mess of incomprehensible syntax By prioritising assignment expressions and implementing pattern "matching" as a statement, they're clearly showing a lot of hostility to one of the major use cases of their language (incidentally, recall that C# introduced LINQ in 2008, a generalised comprehension! That's C#.) There isnt the syntax to reimplement to support natural phrasings of data transformation, in lieu of this, pandas exploits weird operators such as `.loc[a,b]`. At some point the dam is going to have to break, and python is going to have to introduce something to resolve this mess. However, i'd bet it'll be a decade of tooth-and-nail fighting about it. It isnt a software engineering language for education any more, and i'm not seeing this reality being acknowledged
  - biellls 5y ago
    Pattern matching is supported in 3.10 and can match classes structurally, as well as other types (edit: I see that you mean you don't like that it's a statement, which I agree with). The typing system supports structural types with Protocol. Personally what I miss most are multi-line lambdas.
    - mjburgess 5y ago
      I'm not asking for "support" i'm asking for syntax. The question is why the syntax of data processing in python is illegible, not "are things technically possible in python". Commenters here are keen to say "but dont you know!" -- and, yes, I do.
      - zasdffaa 5y ago
        To be fair the confusion of FP vs FP syntax is from your comment: "Perhaps if python would add more support for FP (rather than its present hostility)" Python does FP fine and FP via an ugly library or via elegant inbuilt syntax is still FP. I get you want nicer syntax but it wasn't clear. I dunno, write a PEP and see if it gets support. I hope you succeed!
  - ok_dad 5y ago
    > pattern matching expressions Available in 3.10: https://peps.python.org/pep-0636/ https://peps.python.org/pep-0636/ > syntax for partial application... from functools import partial somefunc_arg1_arg2 = partial(somefunc, arg1, arg2) > ...and composition A native compositional syntax would be nice. > a typing system which can express structural types # mypy will typecheck this code and see `MyString()` has a `.read()` method, so it's a `Readable` even though it doesn't sub-class `Readable` from typing import Any, Protocol class Readable(Protocol): def read(self) -> Any: ... def read_something(something: Readable) -> Any: return something.read() class MyString: a_string: str = "something" def read(self) -> str: return self.a_string read_something(MyString()) > a generalised list comprehension I agree this would be nice.
    - mjburgess 5y ago
      they arent expressions, defeating the whole point that isnt syntax for partial appication, and the protocol system for structrural typing is syntactically and practically absurd the question is "why do data processing libs in python look syntactically illegible" and the answer is largely, as youve posted above, that python doesnt support useful syntax

tomtom1337 5y ago

I'd recommend checking out polars as an alternative to pandas - https://github.com/pola-rs/polars https://github.com/pola-rs/polars It has a rather different api, and is significantly faster. Highly recommend it.

riyadparvez 5y ago
The issue is there's a huge ecosystem developed on top of Pandas. So if it's different API, now you lose access to all the ecosystem.
- nojito 5y ago
  Polars can export to arrow which you can then load into pandas. Or you can convert it directly into a pandas Df
- ritchie46 5y ago
  Should not be a real blocker. There is good interop with arrow, numpy and pandas. Where arrow and numpy mostly is zero-copy. So you are one `df.to_numpy()/df.to_pandas()` away to `X` libary you want to use.

animal_spirits 5y ago

Pandas makes it really hard to do easy things, and really easy to do hard things. But its more useful to the people who need it to do the hard things.

biellls 5y ago

A lot of people hate on SQL. I used to be one of them but I've come to think that for data transformations it's hard to beat. My current favorite is DuckDB, which is like a SQLite but columnar. It has great performance and it's easy to call it from python and even run SQL on pandas dataframes.

otsaloma 5y ago

Here's another alternative. I wrote Dataiter specifically as I too was frustrated with Pandas. In my experience if you design a new API from scratch (and don't try to reimplement the Pandas API as many projects have done!) and have some vision and consistent principles, it's well possible to get a good intuitive API as a result. Two relevant issues remain: You're limited by NumPy's datatypes and their problems, such as memory-hogging strings and a lack of a proper missing value (NA), and secondly, limited by the Python language, so compared to e.g. dplyr's non-standard evaluation, you'll need to use lambda functions, which are unfortunately clumsy and verbose. https://github.com/otsaloma/dataiter https://github.com/otsaloma/dataiter Here's a comparison of dplyr vs. Dataiter vs. Pandas, which should give quick overview of the similarieties and differences. https://dataiter.readthedocs.io/en/latest/_static/comparison.html https://dataiter.readthedocs.io/en/latest/_static/comparison...

smartmic 5y ago

If you are ready to stretch your mind and have time and energy to learn a new skill, I recommend this as "data science" language: APL Background: I first gained some experience with J, where I first learned to appreciate the advantages of array languages. The main advantage is actually having your own way of thinking about processing multidimensional data. Now, recently, I've gotten into the notation of APL, and it's really even cooler than the ASCII "noise" of J. The symbols make it easier for me to both write and read programs. Admittedly, for more complex operations with data it takes a lot of learning that you usually don't have. But for simple transformation APL is already quite fast usable and convinces beyond the "mainstream".

account-5 5y ago
APL and J are on my list of programming languages I'd like to learn. I've been listening to the Array Cast podcast for a while. I just need a place to start. I don't really learn a language until I use it for something, at the moment I have nothing to use APL for. Any recommendations of where to start would be welcome.