3 ms·
In my opinion a better argument to stop using pandas is the very unintuitive API pandas have. Additionally, a slight change in the query can force you to restru
by latent-person 24d ago
In my opinion a better argument to stop using pandas is the very unintuitive API pandas have. Additionally, a slight change in the query can force you to restructure the whole query (change all lines), while in Polars (and tidyverse in R) it's just a simple one-line change.
- __mharrison__ 24d agoIf you learn to write pandas correctly you end up writing it very similar to polars (or tidyverse). I agree that the API has warts, though typically it is more concise than polars.
- latent-person 24d agoThe problem is a small change in the question can force you to make huge changes in the code in pandas. I recently gave some examples in my blog [1]. E.g. compare the last two code blocks, where the small change is just that the median is taken within countries. This requires several line changes in pandas. [1]https://bjarkehautop.github.io/Website/blog/data-wrangling-tour.html https://bjarkehautop.github.io/Website/blog/data-wrangling-t...
- anakaine 24d agoPersonally, I think the inclusion of lambda functions in the pandas code shows that the approach isnt ideal to begin with. Fine on smaller datasets, but typically becomes a slow and cumbersome overhead when interacting with larger datasets. More verbose per line vector operations are generally orders of magnitude faster, and become more so as the data grows. They also tend to be easier to read / identify what is going on or what the intent is for people unfamiliar with the code base.
- __mharrison__ 24d agoThis is a common complaint I get all the time (heard it this week while teaching pandas). I compare this to whitespace indentation in Python. Lots of folks complain about it before using it. After they use it it is a non issue. If it really is an issue (and it generally isn't a cause of vectorization removal when used correctly) and you can't get over the syntactic noise if the lambda, pandas 3 introduced pd.col (that work in most (I filed a big about some exceptions) places when you'd use lambda).
- anakaine 24d agoIve seen it in such a wide variety of scenarios where it causes vectorisation removal because of its implementation that I dont reach for it by default any more. I've a number of devs around me who have similar opinions. Each know how to use it, but you dont always write the code you're reviewing or optimising.
- __mharrison__ 23d agoWould love to see examples.
- __mharrison__ 24d agoI looked at your code very quickly, but it looks like you need to use .filter after a .groupby...
- latent-person 24d agoThe text right above the code says why you can't... edit: Let me clarify. From the blog-post: > since a `DataFrameGroupBy` object doesn’t have a `.query()` or boolean-indexing shortcut of its own, so filtering within groups needs `.apply()` again, and the surrounding pipeline has to be rebuilt around it: Hence you really do need one of the versions of the code I gave. You can't do the naive approach with just `.groupby().filter(lambda: )`, since you need a row-wise decision.
- __mharrison__ 24d agoI'm confused, you can use filter after a groupby in pandas... It's late here, I'm going to bed, perhaps I'll write the code tomorrow when I'm at my laptop and not on my phone.
- deleted 24d ago[deleted]
- __mharrison__ 23d agoI misspoke, you need to use .groupby/.transform to add a new filtering column: (sales .assign(country_median=lambda df_: ( df_.groupby("country")["amount"].transform("median") )) .query("amount <= country_median * 10") .assign(net=pd.col('amount') - pd.col('discount')) .groupby("country", as_index=False) .agg(total=("net", "sum")) )
- latent-person 23d agoYes, so basically equivalent to the code I showed in the blog.
- d2ou 22d agoAnd for a similar dplyr-like API on duckdb I recommand the excellent https://ibis-project.org/ https://ibis-project.org/ !