4 ms·

One of the things that most pisses me off about numpy, pandas, and the whole scipy ecosystem is that everything is "immediate mode": all operations evaluate _in

One of the things that most pisses me off about numpy, pandas, and the whole scipy ecosystem is that everything is "immediate mode": all operations evaluate _instantly_ to new arrays or dataframes. There's no opportunity for any component to evaluate a whole expression tree and optimize it, e.g., by loop hoisting. The right way to design a data analysis DSL is to do it the way Python's dask does it: build an operation graph and execute it as the last step. The trouble is that Dask doesn't get it right either because, as part of its graph formation, it computes sizes of operands, and computing operand sizes can involve huge amounts of computation, and so dask, in effect, is also an "immediate mode" system. What I really want to see is something lazy that can do sane query planning and that can work within limited system resources. Maybe one day I'll open source the work I've done in this space. Query languages are infinitely nicer for analytics than data processing libraries.

bobbylarrybobby 7y ago
It really is remarkable that Python's numerical computing libraries have such poor performance. When doing chains of elementwise operations on large arrays (such as toy example `elementwise cos(sqrt(sin(pow(array, 2))))`), Julia appears to outperform Python by a factor 2! Numpy cannot avoid computing each intermediate array, which means it has to allocate a ton of wasteful memory. Meanwhile Julia does the smart thing and coalesces all operations into one and applies that single operation elementwise, allocating only a single new array. Pandas does not also defer computations, which means computing Boolean functions that include the same data multiple times must make multiple passes over said data. Absurd.
- deleted 7y ago
  [deleted]
- quotemstr 7y ago
  > Julia appears to outperform Python by a factor 2 Now I have some more reading to do. Thanks.
  - timClicks 7y ago
    Julia is very, very good. I say this as a person who has invested about a decade in the Python data science ecosystem. Numpy will probably always be copy-heavy, whereas Julia's JIT can optimize as aggressively as the impressive type system allows.
- nmca 7y ago
  JAX supports this; as do other performance-targeting numerical libraries. There are hoops of course.
- a_t48 7y ago
  numpy appears to have optional arguments for the storage location of outputs: https://docs.scipy.org/doc/numpy/reference/generated/numpy.cos.html https://docs.scipy.org/doc/numpy/reference/generated/numpy.c... - Could you elaborate a little further? The syntax might not be as nice as another language or framework, but it's not "unavoidable". Disclaimer: have never used numpy, have used python fairly extensively.
  - bobbylarrybobby 7y ago
    You are right about the `out` argument, I'd forgotten about that. But even avoiding the wasteful memory allocations, numpy still is about 60% slower than Julia, as it makes multiple passes over the input data. (If there's a way to get numpy to just make a single pass over the data and remain performant, I'd love to know.)
    - a_t48 7y ago
      The first thing I'd reach for is to refactor into a list comprehension. Looks like this is the proper way to do it: for x in np.nditer(a, op_flags = ['readwrite']): x[...] = cos(sqrt(sin(pow(x, 2)))) That's some knarly syntax though. I've never seen that ellipsis operator? Edit: just read about `Ellipsis`. I'm a fan, even if it's sort of nonstandard across libraries. Those readwrite flags are a travesty though, but at least you can paper over it with a helper function. Something like: def np_apply(a, f): for x in np.nditer(a, op_flags = ['readwrite']): x[...] = f(x) np_apply(x, lambda x: return cos(sqrt(sin(pow(x, 2)))))) or def np_apply(a, *argv): for x in np.nditer(a, op_flags = ['readwrite']): for f in argv: x = f(x) x[...] = x np_apply(lambda x: return pow(x, 2), sin, sqrt, cos) Edit3: There's a way to turn this into "pythonic" list comprehension code, but it would probably only make it look prettier rather than more performant.
  - quotemstr 7y ago
    Yes, you can use the out arguments, but doing so 1) complicates your code, and 2) isn't necessarily a win. You have to do the same number of memory write bus cycles whether you write to a specified output array or to some new array.
adamnemecek 7y ago
Do you have an opinion on julia?
fulafel 7y ago
Isn't the alternative in the end a computation DSL and optimizing compiler? GPU HLLs like Halide, Futhark, etc show the way here.
nevi-me 7y ago
Apache Spark has a great 'lazy' story, and I recommend it as a compute supplement for pandas and numpy. I'm always fascinated by the lengths the authors go through to optimise queries, from the cluster to one's machine. Also, the integration with Apache Arrow over the years has bore much fruit, to the extent that a pyspark user can cross over to Pandas with little serde cost. Fun story, about 18 months ago I was consulting for a telco. One of my tasks was to run a segmentation model on their subscriber base. They wanted me to use SAS in their cluster, but because I had 'brought my infra/laptop', I told them that I could do it with Python instead. I got the gist of it working on Pandas, but it was 'slow'. So they asked me if I was going to run it on a sample, but I said I'll port it to Spark in the evening. The following morning I presented them with results, and the porting to Spark took about 2 hours (I hadn't used SparkML before), and the thing took an hour to run on the whole population.
pangoraw 7y ago
Have you heard about Weld[1] ? It is an ongoing effort to built an intermediary representation that numerical library could share to produce an optimized instructions tree. [1] https://github.com/weld-project/weld https://github.com/weld-project/weld
muldvarp 7y ago
I never understood why we limit ourselves to sending SQL to databases. Instead of sending SQL we should be sending actual logical plans to the database. These logical plans could then be optimized by the database and executed. This would solve two issues I have with SQL: 1. Without CTEs SQL isn't that expressive and with CTEs queries can be hard to understand. Logical query plans on the other hand are rather easy to understand and can express more queries than SQL can. 2. SQL is hard to modify programmatically. To most software, SQL is simply a string without any inherent meaning. A logical query plan on the hand is a tree structure that can be introspected and modified. That's basically the same thing you want, except that you want it implemented on dataframes while I want it implemented for database systems.
logicchains 7y ago
>What I really want to see is something lazy that can do sane query planning and that can work within limited system resources. Maybe one day I'll open source the work I've done in this space. Eigen does this via expression templates; it essentially optimises the computation graph at compile time.
- quotemstr 7y ago
  > Eigen does this via expression templates; it essentially optimises the computation graph at compile time. That's wrong too. The right query execution plan can depend heavily on the sizes and statistical distributions of the operands. Eigen doesn't have enough information at compile time to make good decisions in all cases. This kind of computation really calls for a JIT.

bobbylarrybobby 7y ago

It really is remarkable that Python's numerical computing libraries have such poor performance. When doing chains of elementwise operations on large arrays (such as toy example `elementwise cos(sqrt(sin(pow(array, 2))))`), Julia appears to outperform Python by a factor 2! Numpy cannot avoid computing each intermediate array, which means it has to allocate a ton of wasteful memory. Meanwhile Julia does the smart thing and coalesces all operations into one and applies that single operation elementwise, allocating only a single new array. Pandas does not also defer computations, which means computing Boolean functions that include the same data multiple times must make multiple passes over said data. Absurd.

deleted 7y ago
[deleted]
quotemstr 7y ago
> Julia appears to outperform Python by a factor 2 Now I have some more reading to do. Thanks.
- timClicks 7y ago
  Julia is very, very good. I say this as a person who has invested about a decade in the Python data science ecosystem. Numpy will probably always be copy-heavy, whereas Julia's JIT can optimize as aggressively as the impressive type system allows.
nmca 7y ago
JAX supports this; as do other performance-targeting numerical libraries. There are hoops of course.
a_t48 7y ago
numpy appears to have optional arguments for the storage location of outputs: https://docs.scipy.org/doc/numpy/reference/generated/numpy.cos.html https://docs.scipy.org/doc/numpy/reference/generated/numpy.c... - Could you elaborate a little further? The syntax might not be as nice as another language or framework, but it's not "unavoidable". Disclaimer: have never used numpy, have used python fairly extensively.
- bobbylarrybobby 7y ago
  You are right about the `out` argument, I'd forgotten about that. But even avoiding the wasteful memory allocations, numpy still is about 60% slower than Julia, as it makes multiple passes over the input data. (If there's a way to get numpy to just make a single pass over the data and remain performant, I'd love to know.)
  - a_t48 7y ago
    The first thing I'd reach for is to refactor into a list comprehension. Looks like this is the proper way to do it: for x in np.nditer(a, op_flags = ['readwrite']): x[...] = cos(sqrt(sin(pow(x, 2)))) That's some knarly syntax though. I've never seen that ellipsis operator? Edit: just read about `Ellipsis`. I'm a fan, even if it's sort of nonstandard across libraries. Those readwrite flags are a travesty though, but at least you can paper over it with a helper function. Something like: def np_apply(a, f): for x in np.nditer(a, op_flags = ['readwrite']): x[...] = f(x) np_apply(x, lambda x: return cos(sqrt(sin(pow(x, 2)))))) or def np_apply(a, *argv): for x in np.nditer(a, op_flags = ['readwrite']): for f in argv: x = f(x) x[...] = x np_apply(lambda x: return pow(x, 2), sin, sqrt, cos) Edit3: There's a way to turn this into "pythonic" list comprehension code, but it would probably only make it look prettier rather than more performant.
- quotemstr 7y ago
  Yes, you can use the out arguments, but doing so 1) complicates your code, and 2) isn't necessarily a win. You have to do the same number of memory write bus cycles whether you write to a specified output array or to some new array.

adamnemecek 7y ago

Do you have an opinion on julia?

fulafel 7y ago

Isn't the alternative in the end a computation DSL and optimizing compiler? GPU HLLs like Halide, Futhark, etc show the way here.

nevi-me 7y ago

Apache Spark has a great 'lazy' story, and I recommend it as a compute supplement for pandas and numpy. I'm always fascinated by the lengths the authors go through to optimise queries, from the cluster to one's machine. Also, the integration with Apache Arrow over the years has bore much fruit, to the extent that a pyspark user can cross over to Pandas with little serde cost. Fun story, about 18 months ago I was consulting for a telco. One of my tasks was to run a segmentation model on their subscriber base. They wanted me to use SAS in their cluster, but because I had 'brought my infra/laptop', I told them that I could do it with Python instead. I got the gist of it working on Pandas, but it was 'slow'. So they asked me if I was going to run it on a sample, but I said I'll port it to Spark in the evening. The following morning I presented them with results, and the porting to Spark took about 2 hours (I hadn't used SparkML before), and the thing took an hour to run on the whole population.

pangoraw 7y ago

Have you heard about Weld[1] ? It is an ongoing effort to built an intermediary representation that numerical library could share to produce an optimized instructions tree. [1] https://github.com/weld-project/weld https://github.com/weld-project/weld

muldvarp 7y ago

I never understood why we limit ourselves to sending SQL to databases. Instead of sending SQL we should be sending actual logical plans to the database. These logical plans could then be optimized by the database and executed. This would solve two issues I have with SQL: 1. Without CTEs SQL isn't that expressive and with CTEs queries can be hard to understand. Logical query plans on the other hand are rather easy to understand and can express more queries than SQL can. 2. SQL is hard to modify programmatically. To most software, SQL is simply a string without any inherent meaning. A logical query plan on the hand is a tree structure that can be introspected and modified. That's basically the same thing you want, except that you want it implemented on dataframes while I want it implemented for database systems.

logicchains 7y ago

>What I really want to see is something lazy that can do sane query planning and that can work within limited system resources. Maybe one day I'll open source the work I've done in this space. Eigen does this via expression templates; it essentially optimises the computation graph at compile time.

quotemstr 7y ago
> Eigen does this via expression templates; it essentially optimises the computation graph at compile time. That's wrong too. The right query execution plan can depend heavily on the sizes and statistical distributions of the operands. Eigen doesn't have enough information at compile time to make good decisions in all cases. This kind of computation really calls for a JIT.