4 ms·
I use Python extensively for my analysis projects - and while Pandas is my go-to library for many things, I feel it's just very slow. I know that just doing stu
by TrackerFF 4y ago
I use Python extensively for my analysis projects - and while Pandas is my go-to library for many things, I feel it's just very slow. I know that just doing stuff in numpy instead speeds things up considerably, and that's way before doing other optimization, but are there any other libraries out there similar to Pandas, but made for higher performance?
- HFguy 4y agoPolars and Modin support multiple cores and have optimizations. Modin has most of the Pandas api supported. Pyspark api for pandas supports multiple cores and clusters.
- kzrdude 4y agopolars, but it's still developing.
- sireat 4y agoCurious where do you experience slowness with Pandas? Usually slowness in Pandas will be if you start doing non vector things. Whenever you catch yourself writing a loop in Pandas you know you've gone the wrong way. I too use Python extensively (with Pandas among other things) and usually Pandas is perfectly fine. (havent gone over 64GB memory usage yet) I consider Pandas to be Numpy with benefits (methods) - since Pandas dataframe is basically just a collection of Numpy arrays. Those are about as close to C type arrays you can get in Python. The only practical problems have been with regressions among supporting libraries such as pyarrow not playing nice with numpy when working with parquet files.
- anothernewdude 4y agoSlowness from pandas mostly comes when people forget to use df.values before running their algorithm.