3 ms·

The array focused style enables efficient interaction with the C-written backend of numpy. You need to adapt your style (in Python) to get these performance ga

by carlmr 4y ago

The array focused style enables efficient interaction with the C-written backend of numpy. You need to adapt your style (in Python) to get these performance gains. In a language like Rust you can write normal, readable code and get the same performance, plus you don't incur the penalty of copying data between the language boundaries.

jrochkind1 4y ago
That makes sense, and still to me suggests the OP is inaccurate talking about where numpy's performance advantage came from: > NumPy. NumPy takes a similar approach: it cannot itself do anything to speed up Python’s fundamental interpreted execution speed3, but it can move the intensive numerical operations that it implements into operations on arrays, rather than operations on individual objects, whether numbers or not. The move to operations on arrays would do nothing if the logic wasn't also happening in the C-written backend -- in fact numpy _did_ do something about Python's fundamental interpreted execution speed, it moved the logic to the C backend. You point out that _also_ needed the move to array-based things to take full advantage of it, to minimize trips across the python/C boundary if I understand right. But the gain is not simply from changing from operations on individual objects to operations on arrays. This is not really a demonstration of the array "style" having amazing performance advantages (5000%!) in general, generalizable to any platform as the OP does. It is instead in the numpy case specific to the enviroment of Python with C bindings, and wanting to minimize trips across the python/C boundary.
- carlmr 4y ago
  >You point out that _also_ needed the move to array-based things to take full advantage of it, to minimize trips across the python/C boundary if I understand right. So yes and no. Moving to array-based operations removes the need for the for-loops in python that are slow. So you can do more on the C backend than in the slow Python frontend, by changing to an array-based style. If you did this for single numbers, the boundary operations would be too much to make this useful. By moving the array into the backend we have a lot of computations in C, so it's useful to incur the penalty at the boundary.

by carlmr 4y ago

jrochkind1 4y ago

That makes sense, and still to me suggests the OP is inaccurate talking about where numpy's performance advantage came from: > NumPy. NumPy takes a similar approach: it cannot itself do anything to speed up Python’s fundamental interpreted execution speed3, but it can move the intensive numerical operations that it implements into operations on arrays, rather than operations on individual objects, whether numbers or not. The move to operations on arrays would do nothing if the logic wasn't also happening in the C-written backend -- in fact numpy _did_ do something about Python's fundamental interpreted execution speed, it moved the logic to the C backend. You point out that _also_ needed the move to array-based things to take full advantage of it, to minimize trips across the python/C boundary if I understand right. But the gain is not simply from changing from operations on individual objects to operations on arrays. This is not really a demonstration of the array "style" having amazing performance advantages (5000%!) in general, generalizable to any platform as the OP does. It is instead in the numpy case specific to the enviroment of Python with C bindings, and wanting to minimize trips across the python/C boundary.

carlmr 4y ago
>You point out that _also_ needed the move to array-based things to take full advantage of it, to minimize trips across the python/C boundary if I understand right. So yes and no. Moving to array-based operations removes the need for the for-loops in python that are slow. So you can do more on the C backend than in the slow Python frontend, by changing to an array-based style. If you did this for single numbers, the boundary operations would be too much to make this useful. By moving the array into the backend we have a lot of computations in C, so it's useful to incur the penalty at the boundary.