4 ms·

> once I had a decent algorithm, I could turn to Cython to tighten up the bottlenecks and make it fast. What are your preferred ways to profile Python code? Co

by pbowyer 11y ago

> once I had a decent algorithm, I could turn to Cython to tighten up the bottlenecks and make it fast. What are your preferred ways to profile Python code? Coming recently from PHP, where we have XDebug/KCachegrind, the excellent Facebook-sponsored Xhprof, https://blackfire.io https://blackfire.io and https://tideways.io https://tideways.io, it's felt a step backwards. I've tried line_profiler, and used memory_profiler and cProfile with pyprof2calltree and KCachegrind. I've found the cProfile output confusing when it crosses the Python-C barrier for numpy, sklearn etc.

lqdc13 11y ago
cProfile takes a while to learn how to use well. What didn't you like about line_profiler? Here's a good guide on how to write fast(ish) code in Python: https://wiki.python.org/moin/PythonSpeed/PerformanceTips https://wiki.python.org/moin/PythonSpeed/PerformanceTips Generally, the best strategy for me has been to use NumPy wherever possible and to avoid creating many complex objects. Best to use built in dicts or tuples for things that store data. Thus the only time I run into issues is when implementing algorithms in which case I usually isolate the slow function and turn it into a Cython module. Recently have been playing around with https://github.com/jboy/nim-pymod https://github.com/jboy/nim-pymod which seems like a much better solution.
- pbowyer 11y ago
  > cProfile takes a while to learn how to use well. I bet :) I have only scratched the surface, that's for sure. > What didn't you like about line_profiler? In this case I was trying to profile the overall simulation codebase to find the slow spots (rather than guess) - a simulation that takes 60 minutes to run. line_profiler wasn't great at giving digestible results from that - and I haven't worked out how to write all output (across multiple modules) to file, without specifying the file in each decorator. I've started to break the codebase down into 'tests' to measure each algorithm separately, and will give line_profiler another go then. Re memory_profiler, while the mprof command showed all peaks, the line by line output only showed the result after executing each line - so when 4GB of RAM disappeared in a skimage call, only to be released at the end - it wasn't reflected in the output. Which is tricy when trying to reduce overall memory usage.
echlebek 11y ago
I've optimized quite a bit of code with Cython. I do something like: python -m cProfile module.py | gprof2dot -f pstats | python -m xdot -
syllogism 11y ago
After using the profiler to look at the right piece of code, an important part of the process when optimising with Cython is: cython -a module.pyx && open module.html the -a command produces an annotated html file, which lets you see how Cython is compiling your Python into C. There's a colour coding to give you an indication of which bits are fully typed and being translated directly, and which parts are hitting the CPython API at run-time (and hence will be slow).

by pbowyer 11y ago

lqdc13 11y ago

cProfile takes a while to learn how to use well. What didn't you like about line_profiler? Here's a good guide on how to write fast(ish) code in Python: https://wiki.python.org/moin/PythonSpeed/PerformanceTips https://wiki.python.org/moin/PythonSpeed/PerformanceTips Generally, the best strategy for me has been to use NumPy wherever possible and to avoid creating many complex objects. Best to use built in dicts or tuples for things that store data. Thus the only time I run into issues is when implementing algorithms in which case I usually isolate the slow function and turn it into a Cython module. Recently have been playing around with https://github.com/jboy/nim-pymod https://github.com/jboy/nim-pymod which seems like a much better solution.

pbowyer 11y ago
> cProfile takes a while to learn how to use well. I bet :) I have only scratched the surface, that's for sure. > What didn't you like about line_profiler? In this case I was trying to profile the overall simulation codebase to find the slow spots (rather than guess) - a simulation that takes 60 minutes to run. line_profiler wasn't great at giving digestible results from that - and I haven't worked out how to write all output (across multiple modules) to file, without specifying the file in each decorator. I've started to break the codebase down into 'tests' to measure each algorithm separately, and will give line_profiler another go then. Re memory_profiler, while the mprof command showed all peaks, the line by line output only showed the result after executing each line - so when 4GB of RAM disappeared in a skimage call, only to be released at the end - it wasn't reflected in the output. Which is tricy when trying to reduce overall memory usage.

echlebek 11y ago

I've optimized quite a bit of code with Cython. I do something like: python -m cProfile module.py | gprof2dot -f pstats | python -m xdot -

syllogism 11y ago

After using the profiler to look at the right piece of code, an important part of the process when optimising with Cython is: cython -a module.pyx && open module.html the -a command produces an annotated html file, which lets you see how Cython is compiling your Python into C. There's a colour coding to give you an indication of which bits are fully typed and being translated directly, and which parts are hitting the CPython API at run-time (and hence will be slow).