4 ms·
The thing I have to do most often is find out where a function spends its time. It's a trivial problem. But there's no good out-of-the-box solution. I ended up
by mherrmann 9y ago
The thing I have to do most often is find out where a function spends its time. It's a trivial problem. But there's no good out-of-the-box solution. I ended up creating one [1]. Example:
with Timer('Task') as timer:
with timer.child('First step'):
sleep(1)
for _ in range(5):
with timer.child('Baby steps'):
sleep(.5)
Outputs:
Task: 3.520s
Baby steps: 2.518s (71%)
First step: 1.001s (28%)
This way, you know where most of the time is spent and what you need to optimise.
It's on PyPI as well:
pip install timer_cm
The code is a grand 42 lines. But once you've had to do it a few times it becomes so tedious to code up from scratch that it's much nicer to have a ready-made solution.
[1]: https://github.com/mherrmann/timer-cm https://github.com/mherrmann/timer-cm
- tgb 9y agoI really like the simplicity of your solution, but you might be interested in kernprof: https://github.com/rkern/line_profiler https://github.com/rkern/line_profiler It seems to be the nearly-canonical line-by-line profiler for Python, or was some years ago.
- mherrmann 9y agoInteresting, but much more complex. I suppose it's more powerful as well as a reward. I've never had the need for such complexity though.
- tempay 9y agoIts implimentation is much more complex but I'd recommend trying it as it's trivial to use in practice: - Find a suspiciously slow function using cProfile - pip install line_profiler - Edit function to start with @profile - Run kernprof - Take a look at the output and act accordigly
- mattmanser 9y agoI hope I'm not teaching you to suck eggs, but you can use a profiler to do that rather than changing your code with what are basically print statements. I don't use Python any more and have never profiled it, but there seems to be good options: https://docs.python.org/2/library/profile.html https://docs.python.org/2/library/profile.html There are reasons to do what you're doing, e.g. Stack Exchange have a mini-profiler to watch production, but to identify problems in a development environment, profilers are usually quicker/easier once you've learnt to use them. It also helps you identify problems you didn't even know about.
- mherrmann 9y agoInteresting point, though again much more complex. [edit] Anyone reading this: Do not discard cProfile like I did. It just let me shave 900 ms off the startup time of my file manager (mentioned in a child comment). Wow!
- mattmanser 9y agoI admit they take time up front to learn to use and interpret the results, but it's a valuable skill and worth doing every year or so even on smaller projects to see if there are any easy wins.
- mherrmann 9y agoYes, you're right. Mine is just the simplest and probably least powerful solution. cProfile is definitely more powerful.
- rrauenza 9y agoI think yours still has a valid use in production logging...
- mherrmann 9y agoOmg, cProfile just let me shave 900ms off the startup-time of my PyQt-based file manager [1]. I would not have found this that quickly without your hint. Thank you! [1]: https://fman.io https://fman.io
- mattmanser 9y agoNo problem! This was on the front page today, but flagged as a dupe, also an interesting read as it explains about flame graphs too: https://news.ycombinator.com/item?id=14384370 https://news.ycombinator.com/item?id=14384370
- dom0 9y agoCouple notes about cProfile - The slowdown is difficult to predict and somewhat unstable; never compare timings from runs with/without cProfile - It's a deterministic, wall-clock profiler - pyprof2calltree & kcachegrind. - Loading a profile file = arbitrary code execution. Don't ever load a profile file you did not generate yourself.