6 ms·

Data Science: Performance of Python vs. Pandas vs. Numpy

118 points by lukasz_km 9y ago

autokad 9y ago
I usually do all my feature engineering and data exploration in pandas, and then I change my data from pandas to numpy just before i throw it to the classifiers / regressors
- homarp 9y ago
  have you looked at PyTorch ? can be seen as a GPU acceleratd numpy see https://vwrideout.github.io/QR-Blog/index.html https://vwrideout.github.io/QR-Blog/index.html
- nerdponx 9y ago
  Likewise. There's also sklearn_pandas which helps you plug a DataFrame directly into a Transformer or Classifier.
apl 9y ago
I only skimmed the notebook, but the code looks fairly inadequate. The pandas portion, for instance, treats the data frame as a dumb array and critically ignores grouping functionality which should offer a tremendous speedup. Moreover, for the exact same task pandas should never be slower than NumPy. Benchmarks are hard to get right but this one falls way short of saying anything at all about performance penalties incurred by various libraries and abstractions.
em500 9y ago
I left the following comment at the blog: For this example, idiomatic Pandas would look like this: def gen_stats_pandas2(dataset_pandas): grouped = dataset_pandas.groupby('product_id') stats = grouped.aggregate(['count', 'sum', 'mean']) return stats[[('product_order_num','count'), ('quantity','sum'), ('price','mean')]] which is simpler and about 500x faster than gen_stats_pandas() in the example: %%time res1 = gen_stats_pandas(dataset_pandas) CPU times: user 8.48 s, sys: 32 ms, total: 8.51 s Wall time: 8.58 s %%time res2 = gen_stats_pandas2(dataset_pandas) CPU times: user 16 ms, sys: 4 ms, total: 20 ms Wall time: 17.6 ms
- apl 9y ago
  Thanks for coding this up. The code in the blog post is, at best, incredibly misleading.
- minimaxir 9y ago
  It's especially weird in this case because the value-proposition of pandas is the ability to do things like aggregation in 3 lines of code and not have to iterate through every row.
  - Radim 9y ago
    Well pandas iterates through every row anyway, just in native-land rather than pure Python. That's not the difference. The trouble with this code was the choice of the quadratic algorithm (which the idiomatic pandas version avoids). You could achieve a similar speed-up as OP for the pure Python code too, by dropping the doubly-nested loop. The point stands as usual: better algo > clever tricks :) Personally, I find the code brevity that pandas offers does not outweigh the added complexity and cognitive load of its (IMO hard to remember and reason about) API. EDIT: pure Python version def gen_stats_python2(dataset_python): aggr = {} for product_id, product_order_num, quantity, price in dataset_python: item = aggr.get(product_id, [0, []]) item[0] += quantity item[1].append(price) aggr[product_id] = item return [ (int(product_id), len(prices), int(quantity), round(float(sum(prices)) / len(prices), 2)) for product_id, (quantity, prices) in aggr.items() ] 761x speedup! %timeit gen_stats_python(dataset_python) 1 loops, best of 3: 36.3 s per loop %timeit gen_stats_python2(dataset_python) 10 loops, best of 3: 47.7 ms per loop
    - iClaudiusX 9y ago
      It's a lot faster if you use python's itertools and operator itemgetter: def gen_stats(data): start = time.time() stats = [] data.sort(key=itemgetter(0)) for p_id, p_items in itertools.groupby(data, key=itemgetter(0)): num_orders, _, total_quantity, total_price = map(sum, zip(*p_items)) p_id, num_orders, total_quantity = map(int, [p_id, num_orders/p_id, total_quantity]) stats.append([p_id, num_orders, total_quantity, round(float(total_price/num_orders),2)]) end = time.time() working_time = end-start return stats, working_time Results: Dataset size 55096 records [[1, 8, 340, 22.55], [2, 4, 94, 15.83], [3, 2, 97, 3.89]] [[1, 8, 340, 22.55], [2, 4, 94, 15.83], [3, 2, 97, 3.89]] All results equal: True Runtime (s) gen_stats_slow: 28.489468574523926 gen_stats_fast: 0.06000018119812012
Radim 9y ago
No comment on the poor implementations (why quadratic complexity, in all three cases?), but some general notes: For reporting benchmarks, you probably want the minimum, rather than average. While keeping an eye on the distribution/variation of the individual timings as well (to mitigate caching effects). Check out Python's timeit module or the %timeit magic command in IPython. With more numeric-heavy workflows, the gap between Python/C grows even larger. The more you can push to native-compiled-land, away from Python loops, the better. As another data point, when optimizing the word2vec algorithm, the numbers came out something like [0]: 1x (pure Python) - ~120x (NumPy) - ~2880x (C/Cython) [0] https://rare-technologies.com/word2vec-in-python-part-two-optimizing/ https://rare-technologies.com/word2vec-in-python-part-two-op...
wcr3 9y ago
Much of pandas is actually built on numpy...
aleyan 9y ago
There are a couple of things wrong with this article. The first problem, as pointed out by apl and em500, is the non-idiomatic pandas code. Pandas is fast not when you rewrite your algorithms using one for one with pandas functions, but when you use functions provided uniquely by pandas, then you start thinking with pandas. My rewrite[0] with groupby is 47 times faster than pure python; it would be 1848 times faster if I didn't convert the result to list of lists from DataFrame as in the other cases. Secondly, if your analysis is taking too long on your MacBook Air, you should switch up to faster hardware before investing your time in time consuming optimizations. Presumably this is work done for an employer and for them the data scientists time is much more expensive than a powerful AWS instance. Of course if you made big Big O mistakes, no amount of hardware is going to fix that. [0] https://github.com/aleyan/machinelearningexp/blob/master/DataScience_Performance_Python_Pandas_Numpy.ipynb https://github.com/aleyan/machinelearningexp/blob/master/Dat...
petters 9y ago
The data consists of 50k records. The fastest code in the post takes 3 seconds. Have many programmers completely lost the ability to know roughly how fast code "should" be?
- RipRap 9y ago
  Additionally, how many programmers do you know who write code as efficiently as possible? Or even know how to write "Efficient Code"?
- Johnny_Brahms 9y ago
  We use stupid, simple maths problems (like project Euler 1-30) when hiring for junior positions. We are not looking for someone to do the maths tricks that lets you solve most of the problems with pen and paper, but for someone that recognizes a 100x speed increase using memoization. You'd think people with some programming experience would be able to reason about how to speed up a simple brute force approach, but those parts of the hiring process usually weed out 75%, and I don't expect someone to give me a perfect solution. Heck, not even a working piece of code. Just being able to say "hmm, we could maybe cut half of the computation by just using the relevant parts of the alphabet" is all I want. Then we have people like my colleague that just wrote print(103) which was the correct answer. (He could also give several fast approaches to solving the problem, one which we hadn't thought about before).
Stasis5001 9y ago
This comparison isn't so great because of the following: for product_id in unique_products: product_items = [x for x in dataset_python if x[0]==product_id ] This is O(unique_products * observations), and it looks like O(unique_products) = O(observations). Thus we have a quadratic scan when a linear one would suffice. You'll get the best performance using whichever solution lets you code this to linear the fastest. E.g. pure python, make a dict from product_id to observations and iterate over that, or for pandas, use groupby.
0xfaded 9y ago
As other comments point out, performance mostly depends on correct implementation. Once the scripting overhead is under control, you're now measuring the performance of your blas library. A much more useful comparison, although 2.5 years old: http://blog.nguyenvq.com/blog/2014/11/10/optimized-r-and-python-standard-blas-vs-atlas-vs-openblas-vs-mkl/ http://blog.nguyenvq.com/blog/2014/11/10/optimized-r-and-pyt...
akilat90 9y ago
While we're on the subject, I'd like to see what this community has to offer on the efficiency of the pandas groupby-apply operations, which I believe is an idiomatic pandas way. The question [0] on SO is a demonstration where the groupby-apply falls to complete iteration of the dataset(also using some pandas, not sure if that's idiomatic pandas though) Thanks! [0] https://stackoverflow.com/questions/45012503/conditionally-populating-elements-in-a-pandas-groupby-object-a-vectorized-solu https://stackoverflow.com/questions/45012503/conditionally-p...
gravypod 9y ago
Maybe someone can tell me the benefit but why use Pandas rather then a real database? Pandas seems like a non-standard, in memory, python specific database system that has no indexing, caching, or persistence layer aside from dumping to flat files. It's an RDBMS without the Relational bit.
- eesmith 9y ago
  I don't think you have looked at Pandas that closely. For example, Pandas has multiple ways to persist besides a flat file, including to SQLite and HDF5. - http://pandas.pydata.org/pandas-docs/stable/io.html http://pandas.pydata.org/pandas-docs/stable/io.html It's also aware of the relational data model. The 10 minute intro at http://pandas.pydata.org/pandas-docs/stable/10min.html?highlight=relational http://pandas.pydata.org/pandas-docs/stable/10min.html?highl... says: "pandas provides various facilities for easily combining together Series, DataFrame, and Panel objects with various kinds of set logic for the indexes and relational algebra functionality in the case of join / merge-type operations." and http://pandas.pydata.org/pandas-docs/stable/comparison_with_sql.html http://pandas.pydata.org/pandas-docs/stable/comparison_with_... gives a mapping of basic SQL operations to Pandas. There is also some caching. The documentation mentions things like "A large range of dates for various offsets are pre-computed and cached under the hood in order to make generating subsequent date ranges very fast (just have to grab a slice)" and "Bug in Series update where the parent frame is not updating its cache based on changes". Perhaps you can think of it as an in-memory relational database without SQL or a declarative language but instead has an imperative API for Python, containing a large number of helper functions not in most RDMSes but which are important in data analysis. Think of it as a database engine where the user has control over the internal memory layout so C extension and other code (a "cartridge" or "datablade", if you will) can work with it directly. Supposed you want to read a CSV into a data frame, compute the difference between column "X" and "Y", and show those differences as a histogram. That's one line of Pandas. How do you do that in a RDBMS?
karuth 9y ago
This post is rather naive. Pandas performance improves when one uses in built functions. Also the performance of pandas is better over numpy for larger sizes [1]. [1] http://gouthamanbalaraman.com/blog/numpy-vs-pandas-comparison.html http://gouthamanbalaraman.com/blog/numpy-vs-pandas-compariso...
lukasz_km 9y ago
There is a lot of great Python and Pandas code snippets, but I am not sure anynone posted a Numpy based solution. Below is mine. It gives 100x speed up over the base (quadratic Python) solution. Still, suggested Pure Python solutions and idiomatic Numpy solutions are much faster. I suspect Numpy has more power than that. def gen_stats_numpy_l(dataset_numpy): start = time.time() unique_products,unique_indices = np.unique(dataset_numpy[:,0],return_index = True) product_stats = [] split = np.split(dataset_numpy,unique_indices)[1:] for item in split: length = len(item) product_stats.append([int(item[0,0]),int(length),int(np.sum(item[:,2])),float(np.round(np.sum(item[:,3])/length,2))]) end = time.time() working_time = end-start return product_stats,working_time

118 points by lukasz_km 9y ago

I usually do all my feature engineering and data exploration in pandas, and then I change my data from pandas to numpy just before i throw it to the classifiers / regressors

homarp 9y ago
have you looked at PyTorch ? can be seen as a GPU acceleratd numpy see https://vwrideout.github.io/QR-Blog/index.html https://vwrideout.github.io/QR-Blog/index.html
nerdponx 9y ago
Likewise. There's also sklearn_pandas which helps you plug a DataFrame directly into a Transformer or Classifier.

apl 9y ago

I only skimmed the notebook, but the code looks fairly inadequate. The pandas portion, for instance, treats the data frame as a dumb array and critically ignores grouping functionality which should offer a tremendous speedup. Moreover, for the exact same task pandas should never be slower than NumPy. Benchmarks are hard to get right but this one falls way short of saying anything at all about performance penalties incurred by various libraries and abstractions.

em500 9y ago

I left the following comment at the blog: For this example, idiomatic Pandas would look like this: def gen_stats_pandas2(dataset_pandas): grouped = dataset_pandas.groupby('product_id') stats = grouped.aggregate(['count', 'sum', 'mean']) return stats[[('product_order_num','count'), ('quantity','sum'), ('price','mean')]] which is simpler and about 500x faster than gen_stats_pandas() in the example: %%time res1 = gen_stats_pandas(dataset_pandas) CPU times: user 8.48 s, sys: 32 ms, total: 8.51 s Wall time: 8.58 s %%time res2 = gen_stats_pandas2(dataset_pandas) CPU times: user 16 ms, sys: 4 ms, total: 20 ms Wall time: 17.6 ms

apl 9y ago
Thanks for coding this up. The code in the blog post is, at best, incredibly misleading.
minimaxir 9y ago
It's especially weird in this case because the value-proposition of pandas is the ability to do things like aggregation in 3 lines of code and not have to iterate through every row.
- Radim 9y ago
  Well pandas iterates through every row anyway, just in native-land rather than pure Python. That's not the difference. The trouble with this code was the choice of the quadratic algorithm (which the idiomatic pandas version avoids). You could achieve a similar speed-up as OP for the pure Python code too, by dropping the doubly-nested loop. The point stands as usual: better algo > clever tricks :) Personally, I find the code brevity that pandas offers does not outweigh the added complexity and cognitive load of its (IMO hard to remember and reason about) API. EDIT: pure Python version def gen_stats_python2(dataset_python): aggr = {} for product_id, product_order_num, quantity, price in dataset_python: item = aggr.get(product_id, [0, []]) item[0] += quantity item[1].append(price) aggr[product_id] = item return [ (int(product_id), len(prices), int(quantity), round(float(sum(prices)) / len(prices), 2)) for product_id, (quantity, prices) in aggr.items() ] 761x speedup! %timeit gen_stats_python(dataset_python) 1 loops, best of 3: 36.3 s per loop %timeit gen_stats_python2(dataset_python) 10 loops, best of 3: 47.7 ms per loop
  - iClaudiusX 9y ago
    It's a lot faster if you use python's itertools and operator itemgetter: def gen_stats(data): start = time.time() stats = [] data.sort(key=itemgetter(0)) for p_id, p_items in itertools.groupby(data, key=itemgetter(0)): num_orders, _, total_quantity, total_price = map(sum, zip(*p_items)) p_id, num_orders, total_quantity = map(int, [p_id, num_orders/p_id, total_quantity]) stats.append([p_id, num_orders, total_quantity, round(float(total_price/num_orders),2)]) end = time.time() working_time = end-start return stats, working_time Results: Dataset size 55096 records [[1, 8, 340, 22.55], [2, 4, 94, 15.83], [3, 2, 97, 3.89]] [[1, 8, 340, 22.55], [2, 4, 94, 15.83], [3, 2, 97, 3.89]] All results equal: True Runtime (s) gen_stats_slow: 28.489468574523926 gen_stats_fast: 0.06000018119812012

Radim 9y ago

No comment on the poor implementations (why quadratic complexity, in all three cases?), but some general notes: For reporting benchmarks, you probably want the minimum, rather than average. While keeping an eye on the distribution/variation of the individual timings as well (to mitigate caching effects). Check out Python's timeit module or the %timeit magic command in IPython. With more numeric-heavy workflows, the gap between Python/C grows even larger. The more you can push to native-compiled-land, away from Python loops, the better. As another data point, when optimizing the word2vec algorithm, the numbers came out something like [0]: 1x (pure Python) - ~120x (NumPy) - ~2880x (C/Cython) [0] https://rare-technologies.com/word2vec-in-python-part-two-optimizing/ https://rare-technologies.com/word2vec-in-python-part-two-op...

wcr3 9y ago

Much of pandas is actually built on numpy...

aleyan 9y ago

There are a couple of things wrong with this article. The first problem, as pointed out by apl and em500, is the non-idiomatic pandas code. Pandas is fast not when you rewrite your algorithms using one for one with pandas functions, but when you use functions provided uniquely by pandas, then you start thinking with pandas. My rewrite[0] with groupby is 47 times faster than pure python; it would be 1848 times faster if I didn't convert the result to list of lists from DataFrame as in the other cases. Secondly, if your analysis is taking too long on your MacBook Air, you should switch up to faster hardware before investing your time in time consuming optimizations. Presumably this is work done for an employer and for them the data scientists time is much more expensive than a powerful AWS instance. Of course if you made big Big O mistakes, no amount of hardware is going to fix that. [0] https://github.com/aleyan/machinelearningexp/blob/master/DataScience_Performance_Python_Pandas_Numpy.ipynb https://github.com/aleyan/machinelearningexp/blob/master/Dat...

petters 9y ago

The data consists of 50k records. The fastest code in the post takes 3 seconds. Have many programmers completely lost the ability to know roughly how fast code "should" be?

RipRap 9y ago
Additionally, how many programmers do you know who write code as efficiently as possible? Or even know how to write "Efficient Code"?
Johnny_Brahms 9y ago
We use stupid, simple maths problems (like project Euler 1-30) when hiring for junior positions. We are not looking for someone to do the maths tricks that lets you solve most of the problems with pen and paper, but for someone that recognizes a 100x speed increase using memoization. You'd think people with some programming experience would be able to reason about how to speed up a simple brute force approach, but those parts of the hiring process usually weed out 75%, and I don't expect someone to give me a perfect solution. Heck, not even a working piece of code. Just being able to say "hmm, we could maybe cut half of the computation by just using the relevant parts of the alphabet" is all I want. Then we have people like my colleague that just wrote print(103) which was the correct answer. (He could also give several fast approaches to solving the problem, one which we hadn't thought about before).

Stasis5001 9y ago

This comparison isn't so great because of the following: for product_id in unique_products: product_items = [x for x in dataset_python if x[0]==product_id ] This is O(unique_products * observations), and it looks like O(unique_products) = O(observations). Thus we have a quadratic scan when a linear one would suffice. You'll get the best performance using whichever solution lets you code this to linear the fastest. E.g. pure python, make a dict from product_id to observations and iterate over that, or for pandas, use groupby.

0xfaded 9y ago

As other comments point out, performance mostly depends on correct implementation. Once the scripting overhead is under control, you're now measuring the performance of your blas library. A much more useful comparison, although 2.5 years old: http://blog.nguyenvq.com/blog/2014/11/10/optimized-r-and-python-standard-blas-vs-atlas-vs-openblas-vs-mkl/ http://blog.nguyenvq.com/blog/2014/11/10/optimized-r-and-pyt...

akilat90 9y ago

While we're on the subject, I'd like to see what this community has to offer on the efficiency of the pandas groupby-apply operations, which I believe is an idiomatic pandas way. The question [0] on SO is a demonstration where the groupby-apply falls to complete iteration of the dataset(also using some pandas, not sure if that's idiomatic pandas though) Thanks! [0] https://stackoverflow.com/questions/45012503/conditionally-populating-elements-in-a-pandas-groupby-object-a-vectorized-solu https://stackoverflow.com/questions/45012503/conditionally-p...

gravypod 9y ago

Maybe someone can tell me the benefit but why use Pandas rather then a real database? Pandas seems like a non-standard, in memory, python specific database system that has no indexing, caching, or persistence layer aside from dumping to flat files. It's an RDBMS without the Relational bit.

eesmith 9y ago
I don't think you have looked at Pandas that closely. For example, Pandas has multiple ways to persist besides a flat file, including to SQLite and HDF5. - http://pandas.pydata.org/pandas-docs/stable/io.html http://pandas.pydata.org/pandas-docs/stable/io.html It's also aware of the relational data model. The 10 minute intro at http://pandas.pydata.org/pandas-docs/stable/10min.html?highlight=relational http://pandas.pydata.org/pandas-docs/stable/10min.html?highl... says: "pandas provides various facilities for easily combining together Series, DataFrame, and Panel objects with various kinds of set logic for the indexes and relational algebra functionality in the case of join / merge-type operations." and http://pandas.pydata.org/pandas-docs/stable/comparison_with_sql.html http://pandas.pydata.org/pandas-docs/stable/comparison_with_... gives a mapping of basic SQL operations to Pandas. There is also some caching. The documentation mentions things like "A large range of dates for various offsets are pre-computed and cached under the hood in order to make generating subsequent date ranges very fast (just have to grab a slice)" and "Bug in Series update where the parent frame is not updating its cache based on changes". Perhaps you can think of it as an in-memory relational database without SQL or a declarative language but instead has an imperative API for Python, containing a large number of helper functions not in most RDMSes but which are important in data analysis. Think of it as a database engine where the user has control over the internal memory layout so C extension and other code (a "cartridge" or "datablade", if you will) can work with it directly. Supposed you want to read a CSV into a data frame, compute the difference between column "X" and "Y", and show those differences as a histogram. That's one line of Pandas. How do you do that in a RDBMS?

karuth 9y ago

This post is rather naive. Pandas performance improves when one uses in built functions. Also the performance of pandas is better over numpy for larger sizes [1]. [1] http://gouthamanbalaraman.com/blog/numpy-vs-pandas-comparison.html http://gouthamanbalaraman.com/blog/numpy-vs-pandas-compariso...

lukasz_km 9y ago

There is a lot of great Python and Pandas code snippets, but I am not sure anynone posted a Numpy based solution. Below is mine. It gives 100x speed up over the base (quadratic Python) solution. Still, suggested Pure Python solutions and idiomatic Numpy solutions are much faster. I suspect Numpy has more power than that. def gen_stats_numpy_l(dataset_numpy): start = time.time() unique_products,unique_indices = np.unique(dataset_numpy[:,0],return_index = True) product_stats = [] split = np.split(dataset_numpy,unique_indices)[1:] for item in split: length = len(item) product_stats.append([int(item[0,0]),int(length),int(np.sum(item[:,2])),float(np.round(np.sum(item[:,3])/length,2))]) end = time.time() working_time = end-start return product_stats,working_time