4 ms·

I really, really dislike all the dtype wrangling and how those choices resonate throughout the API. I understand that a lot of work has been done to make that A

by tel 7y ago

I really, really dislike all the dtype wrangling and how those choices resonate throughout the API. I understand that a lot of work has been done to make that API "work", but in practice it feels like that effort would have been better avoided by changing expectations and interfaces. Now, to be clear, that's a hard problem. Heterogenous named bags of homogenous columns with a variety of data types, storage patterns, and ideas about missingness isn't an easy domain... but instead of just trying to make everything work through hammering 6+ semi-coherent interfaces (indices, databases, mutability, immutability/chaining, numpy, dataframes) together, I'd be willing to pay a lot more in verbosity and explicitness for something simple. pd.Series(str, [np.nan, "a"]) => ["nan", "a"] # or even an exception! pd.Series(nullable(str), [np.nan, "a"]) => [nan, "a"] Indexing is vastly over-designed. GroupBy is a very common API and is poorly documented and just weird in no small part due to attempts at dtype inference. Foundational useful concepts like categories feel bolted on. There's join, merge, pivot, pivot_table. I'd chalk this all up to just being "hard", but at the same time I can go pick up R's dplyr library and get a very nice existence proof of how a nice interface could work. Not to say dplyr has it all figured out, but it's a night-and-day improvement to Pandas. Pandas is great. It makes doing data science in Python so vastly much less of a chore than working with straight Numpy. It steals some great ideas and tries out a few interesting ones of its own... but it is far from a joy to work with.

closed 7y ago
Agreed RE GroupBy being challenging, especially compared to dplyr. As I've worked on a port of dplyr to python over the past year, though, I've realized the dtype issue (like you said), indexes, and GroupBy being difficult are likely connected. Basically, * dplyr can chop up a dataframe into 50,000 groups and apply arbitrary functions to it--no problem. * custom pandas grouped applies are very slow There are basically three reasons for slow pandas apply methods... 1. creating an index for each subgroup is slow (will not be a RangeIndex) 2. initializing a series for each group is slow (mostly due to type inference being re-run; could be avoided) 3. AFAIK more type inference is run when concatenating results This leads to a world where grouped calculations can't be run using arbitrary expressions (e.g. lambdas), but have to go through specific SeriesGroupBy methods. I wrote a bit on how I tried to work around that, to enable fast dplyr-like syntax over grouped data in python. Would definitely be interested in your take! There are other libraries, like ibis that do a good job with it, too! https://siuba.readthedocs.io/en/latest/developer/pandas-group-ops.html https://siuba.readthedocs.io/en/latest/developer/pandas-grou...
- has2k1 7y ago
  I have been through this path of trying to figure out why pandas Groupby is slow. It is a big bottleneck when building packages on top of pandas.
  - closed 7y ago
    Well, I'm a big fan of plotnine, and plydata was part of the inspiration for siuba! I think at its core, the groupby issue is a really big problem, and am devoting most of this year to working on it. So if you ever want to pair to work on pandas / pandas wrapping libraries send me an email (link in profile)!
ImaCake 7y ago
>GroupBy is a very common API and is poorly documented My god is it poorly documented. A lot of us newbies trying to do dataframe stuff for the first time get stuck on GroupBy, trying to figure out how it works. Dplyr's functions are miles ahead on useability and documentation.
antipaul 7y ago
Hadley had the discipline to let go and start from scratch 2 if not 3 times before getting it perfect with `dplyr`. plyr came before and I think there was something else That’s how we got the amazing `dplyr` I think pandas is well liked by those who move from C++ or Java, but is disliked by those who move from R
- dm319 7y ago
  I agree - R has performant and robust dataframe functions. dplyr is great for small-medium sized datasets, data.table seems to be really performant for larger sets.
  - bart_spoon 7y ago
    And now there is dtplyr, which simply creates a data.table back end with dplyr syntax on the front end.
- tel 7y ago
  I think this is key: there was a massive learning effort that went into the current dplyr interface. This risked fragmenting the community, but Hadley and his collaborators navigated those waters effectively. That's super tough work and R benefits significantly from it. If I had a magic wand for Pandas, I feel giving that team the opportunity to rework interfaces without killing momentum and community would go so far.
maest 7y ago
> hammering 6+ semi-coherent interfaces (indices, databases, mutability, immutability/chaining, numpy, dataframes) together, That's a pretty succinct way of expressing one of panda's biggest pain points. The API surface is huge because it heroically tries to handle all possible use cases. I do think it would've been significantly easier to develop, maintain and _use_ if pandas was more opinionated and offered a smaller set of very focused functions.
IceDane 7y ago
I've just spent the past 3 months reengineering what is essentially a spreadsheet calculator to use pandas. Pandas was chosen before I was there but it seemed like it was perfect for the job. Aaand.. well, I guess it kind of is. But at the same time it isn't. Our biggest issues haven't been the complexity of what we've been implementing, but rather the insanity of the pandas interface. We've tried to keep it in mind that we're all inexperienced wrt. pandas, and it has definitely gotten better as we have gained experience, but that doesn't give us the hundreds of man-hours back we've spent trying to please pandas. All the insane overloading of everything is awful and stupid. Give me 3 different ways to do 3 different things, not 9 ways to use 1 feature to do 1 thing.

by tel 7y ago

closed 7y ago

Agreed RE GroupBy being challenging, especially compared to dplyr. As I've worked on a port of dplyr to python over the past year, though, I've realized the dtype issue (like you said), indexes, and GroupBy being difficult are likely connected. Basically, * dplyr can chop up a dataframe into 50,000 groups and apply arbitrary functions to it--no problem. * custom pandas grouped applies are very slow There are basically three reasons for slow pandas apply methods... 1. creating an index for each subgroup is slow (will not be a RangeIndex) 2. initializing a series for each group is slow (mostly due to type inference being re-run; could be avoided) 3. AFAIK more type inference is run when concatenating results This leads to a world where grouped calculations can't be run using arbitrary expressions (e.g. lambdas), but have to go through specific SeriesGroupBy methods. I wrote a bit on how I tried to work around that, to enable fast dplyr-like syntax over grouped data in python. Would definitely be interested in your take! There are other libraries, like ibis that do a good job with it, too! https://siuba.readthedocs.io/en/latest/developer/pandas-group-ops.html https://siuba.readthedocs.io/en/latest/developer/pandas-grou...

has2k1 7y ago
I have been through this path of trying to figure out why pandas Groupby is slow. It is a big bottleneck when building packages on top of pandas.
- closed 7y ago
  Well, I'm a big fan of plotnine, and plydata was part of the inspiration for siuba! I think at its core, the groupby issue is a really big problem, and am devoting most of this year to working on it. So if you ever want to pair to work on pandas / pandas wrapping libraries send me an email (link in profile)!

ImaCake 7y ago

>GroupBy is a very common API and is poorly documented My god is it poorly documented. A lot of us newbies trying to do dataframe stuff for the first time get stuck on GroupBy, trying to figure out how it works. Dplyr's functions are miles ahead on useability and documentation.

antipaul 7y ago

Hadley had the discipline to let go and start from scratch 2 if not 3 times before getting it perfect with `dplyr`. plyr came before and I think there was something else That’s how we got the amazing `dplyr` I think pandas is well liked by those who move from C++ or Java, but is disliked by those who move from R

dm319 7y ago
I agree - R has performant and robust dataframe functions. dplyr is great for small-medium sized datasets, data.table seems to be really performant for larger sets.
- bart_spoon 7y ago
  And now there is dtplyr, which simply creates a data.table back end with dplyr syntax on the front end.
tel 7y ago
I think this is key: there was a massive learning effort that went into the current dplyr interface. This risked fragmenting the community, but Hadley and his collaborators navigated those waters effectively. That's super tough work and R benefits significantly from it. If I had a magic wand for Pandas, I feel giving that team the opportunity to rework interfaces without killing momentum and community would go so far.

maest 7y ago

> hammering 6+ semi-coherent interfaces (indices, databases, mutability, immutability/chaining, numpy, dataframes) together, That's a pretty succinct way of expressing one of panda's biggest pain points. The API surface is huge because it heroically tries to handle all possible use cases. I do think it would've been significantly easier to develop, maintain and _use_ if pandas was more opinionated and offered a smaller set of very focused functions.

IceDane 7y ago

I've just spent the past 3 months reengineering what is essentially a spreadsheet calculator to use pandas. Pandas was chosen before I was there but it seemed like it was perfect for the job. Aaand.. well, I guess it kind of is. But at the same time it isn't. Our biggest issues haven't been the complexity of what we've been implementing, but rather the insanity of the pandas interface. We've tried to keep it in mind that we're all inexperienced wrt. pandas, and it has definitely gotten better as we have gained experience, but that doesn't give us the hundreds of man-hours back we've spent trying to please pandas. All the insane overloading of everything is awful and stupid. Give me 3 different ways to do 3 different things, not 9 ways to use 1 feature to do 1 thing.