5 ms·
> I have a python/pandas script that take 3 days to run. Pandas .map() and .apply() get real slow on big datasets. I found it quicker to solve a problem with a
by ImaCake 7y ago
> I have a python/pandas script that take 3 days to run.
Pandas .map() and .apply() get real slow on big datasets. I found it quicker to solve a problem with a million line dataset by just using base python iterables instead, so nothing needed to fit into my RAM and I didn't have to work with slow pandas mapping.
- appleiigs 7y agoYes, i was thinking about removing the Pandas code and using python iterables. The issue is I did a lot exploration with pandas, which it was good at. If I started with Julia, i wouldn’t need to refactor seeking performance by removing pandas, or for numba, dask, etc. For the existing project i’m thinking the switch to Julia + DataFrames library (despite it being a completely different language) is more of a 1:1 port. In contrast would need to use brain power to build arrays or dicts to mimic Pandas (and probably get it wrong and introduce bugs)
- mirekrusin 7y agoI hear you can use python from Julia, have you tried this as, possibly temporary/transition, approach?
- appleiigs 7y agoYes, thanks. That ability definitely does give me a bit of comfort in case Julia is missing something. I plan not to use it though. When you use it, it’s actually running python, so same speed and such.