5 ms·
A Beginner’s Guide to Optimizing Pandas Code for Speed
- L_226 9y agostep 1 - don't use pandas
- bunderbunder 9y agoPandas isn't perfect, but for small- to medium-size datasets, I haven't seen much that matches its performance, and I haven't seen anything that matches its combination of performance and ease of use.
- closed 9y agoAs a person who deeply enjoys developing with python, I'd have to reluctantly say R's tidyverse is a delight to use and often faster than pandas in my experience.
- bunderbunder 9y agoAh, good to know. I haven't touched R in a while. Does tidyverse fix up the mess that is string handling in R?
- closed 9y agoSorry for the late reply. stringr does a pretty good job (though I like how python handles strings better).
- davidcbc 9y agoBut they work for bamboo shoots. Tough to beat that
- deepsun 9y agoDoesn't NumPy use SIMD instructions? There's no mention of that in the article.
- VHRanger 9y agoIt does under certain installations. Anaconda tends to favor intel MKL on most x86 systems
- VHRanger 9y agoMain performance tip is to find ways not to copy data generally: - use .loc[] - use inplace=True
- audiometry 9y agocan you elaborate on `using .loc[]` -- what is the defective approach it replaces?
- stevesimmons 9y agoThis is a good point to plug my talks on 'Pandas from the Inside' (A), 'Big Pandas' (B) and 'Pandas 2.0' (C) that I presented at various PyData conferences over the last 18 months: - PyData London 2016 - (A), 60 mins, videos online - PyData Washington DC 2016 - (A), 90 mins, videos online - PyData Amsterdam 2017 - (A) and (B), 3 hours - PyData Berlin 2017 - (A) and (B), compressed 90 mins, video probably online - PyCon UK 2017 - (A), (B) and (C), 2 hours PDF slides for (A) are here: https://github.com/stevesimmons/pydata-ams2017-pandas-and-dask-from-the-inside/raw/master/slides-1-pandas-from-the-inside.pdf https://github.com/stevesimmons/pydata-ams2017-pandas-and-da... Others are in my other repos on github: https://github.com/stevesimmons https://github.com/stevesimmons
- setr 9y agoThere's something really offensive about attributing "premature optimization..." to xkcd, even as a joke