4 ms·

The idea of using Python for high performance numerical work is daft. The whole scientific Python thing revolves around the fact that Python is an excellent low

The idea of using Python for high performance numerical work is daft. The whole scientific Python thing revolves around the fact that Python is an excellent low boilerplate research and prototyping language with possible production uses where performance isn't critical. Unfortunately this lead to a lot of software in this area being developed for it. You wouldn't be doing performance critical production stuff with R or Matlab? I'm afraid that for serious numbers crunching nothing can truly replace compiled, statically typed languages and additionally you can't rely on GPGPU abstractions without fully understanding the underlying mechanisms and I'm pretty sure that the same is true for NumPy. Your problem are your expectations.

kevin_thibedeau 9y ago
You just switch critical sections to Cython. It knows how to handle NumPy objects and you get compiled, statically typed code when it is needed for performance.
- curiousgal 9y ago
  Numba is a solid option as well.
yongjik 9y ago
There's "not for performance-critical use", and then there's "why is it gobbling up all the RAM (~10GB) and make it impossible for me to do anything, when the entire data I fed it is several hundred MB." Python frequently saunters into the second territory. Well, I guess there are tools available to profile memory usage and stuff, but if I had to spend that much effort tracking down memory issues, I might as well rewrite it in C++. (It doesn't help (or it helps?) that I'm much more comfortable with C++ than Python. YMMV.)
- cookiecaper 9y ago
  My experience is that Python is much better about this than many other languages in its class, Ruby in particular. It's going to be hard to beat manual pointer/memory management in terms of raw performance, but Python is pretty respectable for an untyped, memory-managed language.
- jacquesm 9y ago
  There are some easy to avoid pathological cases such as concatenating numpy arrays in a loop. Take this gem from a well known course: np.concatenate([x.next() for i in range(x.nb)]) That looks pretty innocent but it can eat up your memory in an eyeblink if the input is large enough. That's the sort of pitfall that a lot of python code suffers from because the abstractions are just nice enough to make you believe this will work without penalty and without knowing how it is implemented under the hood you're suddenly out a few gigs of ram.
  - ofek 9y ago
    If np.concatenate accepts any iterable, removing the brackets is what you want to do there.
    - jzwinck 9y ago
      That won't help. Concatenate needs to know the total length up front. It will evaluate all the input items at once.
breatheoften 9y ago
Lots of good replies to this -- and some good links which I am going to follow. I appreciate the comments! I appreciate this comment as well -- and indeed I will agree that my expectations are part of the problem. I'm not actually doing 'performance critical' work -- but rather prototyping tasks -- the speed of my feedback cycle is the only thing that's performance sensitive within the context of my rant -- and this includes time to run the code and the time to debug it. I should also acknowledge other (self-induced) problems including the need to run code on a remote server for extra hardware capability vs my pitiful laptop. I should acknowledge that part of my frustration could come from my development process as from the inherent nature of the underlying tools themselves ... I've experimented with my process -- trying to find something that works for me and I've ended up with this hobbled kludge of tools: - a script that re-runs rsync on file changes - atom + hydrogen extension connected via ssh to a jupyter kernel_gateway - some ssh terminals where I run longer running code from command line - an sshfs mount of a remote directory on the server for viewing some output artifacts ... Avoidable runtime errors after longish-running processing tasks are a very frustrating time-sync ...
- abhirag 9y ago
  Now that I have a better idea of your dev setup I think I can be of a bit more help :) 1. If you are prototyping and prefer working on a remote server, have a look at(https://notebooks.azure.com/ https://notebooks.azure.com/), memory is limited to 4Gb but its free and jupyter notebooks are great for prototyping. 2. For prototyping you should try and do REPL driven development, what I mean is that you should be ok with just playing around with the library API before you write a longish-running processing task, that would reduce(not remove) the chances of a runtime error. Jupyter notebooks excel in this too as you can just try out code in a cell, learn from it, rinse and repeat. You can also use your IDE to send code fragments to REPL and get immediate feedback on them. This way of iterative development would make sure that the speed of your feedback cycle isn't slow. If python's feedback cycle seems slower to you than C++, you are definitely not using the REPL enough :) 3. If debugging is a pain point definitely give an IDE a try, I prefer Visual Studio as I am on Windows. You can very well go with Pycharm or vscode(not technically an IDE but has a debugger so that's that). I personally prefer Jupyter notebook or Emacs with my code and REPL in split windows for prototyping, different strokes for different folks. I personally love working with python, I even write my blog posts in a Jupyter Notebook so although it isn't perfect, it doesn't have to be frustrating either :) Hope my suggestions can be of some help!
jacquesm 9y ago
I'm doing some pretty heavy image recognition while stitching the output of a camera to image objects larger than the camera can see in one go from two different angles in real time. Python works fine for high performance numerical work, the only thing you have to keep an eye on is to use python as much as possible as the glue and the various libraries written in C or C++ to do the heavy lifting for you (and there are tons of such libraries). The reason why python is the right tool for this job is because it has such libraries for just about everything, from serial communications to a bunch of hardware driving stepper motors, relays and reading inputs to imaging and number crunching and all the other bits that went into this. I'd have a much harder time achieving the same effect in any other language and likely it would have cost me a lot more time. Python is not perfect (far from it) but it gets the job done.

kevin_thibedeau 9y ago

You just switch critical sections to Cython. It knows how to handle NumPy objects and you get compiled, statically typed code when it is needed for performance.

curiousgal 9y ago
Numba is a solid option as well.

yongjik 9y ago

There's "not for performance-critical use", and then there's "why is it gobbling up all the RAM (~10GB) and make it impossible for me to do anything, when the entire data I fed it is several hundred MB." Python frequently saunters into the second territory. Well, I guess there are tools available to profile memory usage and stuff, but if I had to spend that much effort tracking down memory issues, I might as well rewrite it in C++. (It doesn't help (or it helps?) that I'm much more comfortable with C++ than Python. YMMV.)

cookiecaper 9y ago
My experience is that Python is much better about this than many other languages in its class, Ruby in particular. It's going to be hard to beat manual pointer/memory management in terms of raw performance, but Python is pretty respectable for an untyped, memory-managed language.
jacquesm 9y ago
There are some easy to avoid pathological cases such as concatenating numpy arrays in a loop. Take this gem from a well known course: np.concatenate([x.next() for i in range(x.nb)]) That looks pretty innocent but it can eat up your memory in an eyeblink if the input is large enough. That's the sort of pitfall that a lot of python code suffers from because the abstractions are just nice enough to make you believe this will work without penalty and without knowing how it is implemented under the hood you're suddenly out a few gigs of ram.
- ofek 9y ago
  If np.concatenate accepts any iterable, removing the brackets is what you want to do there.
  - jzwinck 9y ago
    That won't help. Concatenate needs to know the total length up front. It will evaluate all the input items at once.

breatheoften 9y ago

Lots of good replies to this -- and some good links which I am going to follow. I appreciate the comments! I appreciate this comment as well -- and indeed I will agree that my expectations are part of the problem. I'm not actually doing 'performance critical' work -- but rather prototyping tasks -- the speed of my feedback cycle is the only thing that's performance sensitive within the context of my rant -- and this includes time to run the code and the time to debug it. I should also acknowledge other (self-induced) problems including the need to run code on a remote server for extra hardware capability vs my pitiful laptop. I should acknowledge that part of my frustration could come from my development process as from the inherent nature of the underlying tools themselves ... I've experimented with my process -- trying to find something that works for me and I've ended up with this hobbled kludge of tools: - a script that re-runs rsync on file changes - atom + hydrogen extension connected via ssh to a jupyter kernel_gateway - some ssh terminals where I run longer running code from command line - an sshfs mount of a remote directory on the server for viewing some output artifacts ... Avoidable runtime errors after longish-running processing tasks are a very frustrating time-sync ...

abhirag 9y ago
Now that I have a better idea of your dev setup I think I can be of a bit more help :) 1. If you are prototyping and prefer working on a remote server, have a look at(https://notebooks.azure.com/ https://notebooks.azure.com/), memory is limited to 4Gb but its free and jupyter notebooks are great for prototyping. 2. For prototyping you should try and do REPL driven development, what I mean is that you should be ok with just playing around with the library API before you write a longish-running processing task, that would reduce(not remove) the chances of a runtime error. Jupyter notebooks excel in this too as you can just try out code in a cell, learn from it, rinse and repeat. You can also use your IDE to send code fragments to REPL and get immediate feedback on them. This way of iterative development would make sure that the speed of your feedback cycle isn't slow. If python's feedback cycle seems slower to you than C++, you are definitely not using the REPL enough :) 3. If debugging is a pain point definitely give an IDE a try, I prefer Visual Studio as I am on Windows. You can very well go with Pycharm or vscode(not technically an IDE but has a debugger so that's that). I personally prefer Jupyter notebook or Emacs with my code and REPL in split windows for prototyping, different strokes for different folks. I personally love working with python, I even write my blog posts in a Jupyter Notebook so although it isn't perfect, it doesn't have to be frustrating either :) Hope my suggestions can be of some help!

jacquesm 9y ago

I'm doing some pretty heavy image recognition while stitching the output of a camera to image objects larger than the camera can see in one go from two different angles in real time. Python works fine for high performance numerical work, the only thing you have to keep an eye on is to use python as much as possible as the glue and the various libraries written in C or C++ to do the heavy lifting for you (and there are tons of such libraries). The reason why python is the right tool for this job is because it has such libraries for just about everything, from serial communications to a bunch of hardware driving stepper motors, relays and reading inputs to imaging and number crunching and all the other bits that went into this. I'd have a much harder time achieving the same effect in any other language and likely it would have cost me a lot more time. Python is not perfect (far from it) but it gets the job done.