4 ms·
Hey regarding groupby operations in pandas. I have been moving more and more (for too complicated groupbys) to running them something like: out_rec = []
by lysecret 5y ago
Hey regarding groupby operations in pandas. I have been moving more and more (for too complicated groupbys) to running them something like:
out_rec = []
for id, group in data_frame.groupby("id"):
ladidida....
result = f(group)
out_rec.append(result)
in my experience it isn't much slower than a groupby.apply.
- ellisv 5y agoPerformance aside, I think one of the main criticism of `groupby` in pandas is that they are not composable and become verbose. I'd argue that's the case with your example too.
- sweezyjeezy 5y agogroupby.apply is often the thing that is already way too slow though based on use-case - because I think it pretty much does what you wrote rather than something that makes stronger optimisation assumptions.
- closed 5y agoAuthor here. My concern in the article isn't that you can't do groupby fast, but that the approach above is not composable. * If f() converts grouped data to something ungrouped, then you can't use a similar function f2(f(group)) * If f() returns a grouped object, then you can't do basic operations like f(grouped) + 1, because DataFrameGroupBy, SeriesGroupBy do not define basic operators like addition. Let alone operations against other grouped data. A lot of this is worked out in siuba now, and this doc explains a bit more: https://siuba.readthedocs.io/en/latest/developer/pandas-group-ops.html https://siuba.readthedocs.io/en/latest/developer/pandas-grou...
- lysecret 5y agoYea makes sense. I actually come from R and the deplyr world. Was really happy to read your article :). I might give siuba a try.