4 ms·

Two other profilers that also let you spy on a running Python program: - py-spy: https://github.com/benfred/py-spy https://github.com/benfred/py-spy (written i

by jvns 6y ago

Two other profilers that also let you spy on a running Python program: - py-spy: https://github.com/benfred/py-spy https://github.com/benfred/py-spy (written in Rust) - pyflame: https://github.com/uber-archive/pyflame https://github.com/uber-archive/pyflame (C++, seems to be not maintained anymore) The "no performance cost" thing is interesting: my experience writing a similar profiler is that there are a couple of things that can affect performance a little bit: 1. You have to make a lot of system calls to read the memory of the target process, and if you want to sample at a high rate then that does use some CPU. This can be an issue if you only have 1 CPU. 2. you have two choices when reading memory from a process: you can either race with the program and hope that you read its memory to get the function stack before it changes what function it's running (and you're likely to win the race, because C is faster than Python), or you can pause the program briefly while taking a sample. py-spy has an option to choose which one you want to do: https://github.com/benfred/py-spy#how-can-i-avoid-pausing-the-python-program https://github.com/benfred/py-spy#how-can-i-avoid-pausing-th... Definitely this method is a lot lower overhead than a tracing profiler that instruments every single function call, and in practice it works well. One thing I think is nice about this kind of profiler is that reading memory from the target process sounds like a complicated thing, but it's not: you can see austin's code for reading memory here, and it's implemented for 3 platforms in just 130 lines of C: https://github.com/P403n1x87/austin/blob/877e2ff946ea5313e4773b4ee1d01da853b5b31d/src/mem.h#L98 https://github.com/P403n1x87/austin/blob/877e2ff946ea5313e47...

p403n1x87 6y ago
All good points. The "no performance cost" is indeed more like "negligible performance costs". That's because these days multicore architectures are quite ubiquitous and standard Python applications are single process. For multi-process Python applications, a busy profiler would certainly steal a good chunk of a core, so the impact might be noticeable in that case. As for the race conditions, Austin does not introduce any pauses. Even if it did, there would be no guarantee that it paused at a "good" point, so there are no real benefits in terms of accuracy in pausing. Error rates are quite low anyway, so the actual benefit comes from not pausing at all.
jvns 6y ago
Also, this kind of profiler is great because you can use it on any running Python program, which is pretty magical and very useful. (especially when it's an application you didn't write) But it's not right for every use case: by design austin/py-spy can only really profile the whole program, and if you want to profile a specific function or endpoint in your program, something like PyInstrument https://github.com/joerick/pyinstrument https://github.com/joerick/pyinstrument (which includes Django middlewares & Flask decorators) is a lot more useful.
- p403n1x87 6y ago
  For these kind of things one could use https://github.com/P403n1x87/austin-python https://github.com/P403n1x87/austin-python An early attempt at an APM that was specifically designed to measure endpoint response times was here https://github.com/P403n1x87/austin/blob/apm/austin/apm.py https://github.com/P403n1x87/austin/blob/apm/austin/apm.py The problem with statistical profilers is that they might fail to catch every single call, especially when they are fast.
dTal 6y ago
>you're likely to win the race, because C is faster than Python Heh, so the only reason there's "no performance cost" is the enormous performance cost of using Python in the first place?
- p403n1x87 6y ago
  Running Python is not a performance cost. The meaning behind "no performance cost" is that a tool like this is unlikely going to impact the performance of the application that is being profiled. The fact that Python is not a "fast" programming language is then a different matter.
  - dTal 6y ago
    The point is that it's only possible in the first place because Python leaves so much performance on the table. You can't snoop inside a C++ program in the same way - unless you throw a bunch of sleep() calls everywhere to slow it down, and then hey presto you can! Or, to put it another way, taking Python from 500x SlowerThanCee to 501x is "negligible", but taking C from 1x to 2x slower isn't.
    - p403n1x87 6y ago
      true that
- Demiurge 6y ago
  Yes, enormous performance cost in CPU hours and enormous performance gain in human life hours. hehe
  - junon 6y ago
    I seriously, seriously beg to differ.
- junon 6y ago
  Correct.
benfrederickson 6y ago
> you can either race with the program and hope that you read its memory to get the function stack before it changes what function it's running (and you're likely to win the race, because C is faster than Python), or you can pause the program briefly while taking a sample. Py-spy defaults to blocking because the results can be pretty wrong otherwise: https://github.com/benfred/py-spy/issues/56 https://github.com/benfred/py-spy/issues/56 . You can see this problem profiling a program like https://github.com/benfred/py-spy/blob/master/tests/scripts/recursive.py https://github.com/benfred/py-spy/blob/master/tests/scripts/... with or without the nonblocking flag in py-spy - the nonblocking version produces garbage output. Somewhat interestingly, this problem doesn't seem to occur with Ruby - and rbspy can get away without pausing the target program with only minor errors seen when profiling a similar function. I suspect this is because of differences between how the Ruby and Python interpreters store call stack information, but haven't had a chance to dig into the specifics.
teh_matt 6y ago
Note that tracing profilers don't need to be high overhead - Python is slow enough that efficient tracing can be mostly hidden. For example, https://functiontrace.com https://functiontrace.com tends to have <10% overhead when tracing.

by jvns 6y ago

p403n1x87 6y ago

All good points. The "no performance cost" is indeed more like "negligible performance costs". That's because these days multicore architectures are quite ubiquitous and standard Python applications are single process. For multi-process Python applications, a busy profiler would certainly steal a good chunk of a core, so the impact might be noticeable in that case. As for the race conditions, Austin does not introduce any pauses. Even if it did, there would be no guarantee that it paused at a "good" point, so there are no real benefits in terms of accuracy in pausing. Error rates are quite low anyway, so the actual benefit comes from not pausing at all.

jvns 6y ago

Also, this kind of profiler is great because you can use it on any running Python program, which is pretty magical and very useful. (especially when it's an application you didn't write) But it's not right for every use case: by design austin/py-spy can only really profile the whole program, and if you want to profile a specific function or endpoint in your program, something like PyInstrument https://github.com/joerick/pyinstrument https://github.com/joerick/pyinstrument (which includes Django middlewares & Flask decorators) is a lot more useful.

p403n1x87 6y ago
For these kind of things one could use https://github.com/P403n1x87/austin-python https://github.com/P403n1x87/austin-python An early attempt at an APM that was specifically designed to measure endpoint response times was here https://github.com/P403n1x87/austin/blob/apm/austin/apm.py https://github.com/P403n1x87/austin/blob/apm/austin/apm.py The problem with statistical profilers is that they might fail to catch every single call, especially when they are fast.

dTal 6y ago

>you're likely to win the race, because C is faster than Python Heh, so the only reason there's "no performance cost" is the enormous performance cost of using Python in the first place?

p403n1x87 6y ago
Running Python is not a performance cost. The meaning behind "no performance cost" is that a tool like this is unlikely going to impact the performance of the application that is being profiled. The fact that Python is not a "fast" programming language is then a different matter.
- dTal 6y ago
  The point is that it's only possible in the first place because Python leaves so much performance on the table. You can't snoop inside a C++ program in the same way - unless you throw a bunch of sleep() calls everywhere to slow it down, and then hey presto you can! Or, to put it another way, taking Python from 500x SlowerThanCee to 501x is "negligible", but taking C from 1x to 2x slower isn't.
  - p403n1x87 6y ago
    true that
Demiurge 6y ago
Yes, enormous performance cost in CPU hours and enormous performance gain in human life hours. hehe
- junon 6y ago
  I seriously, seriously beg to differ.
junon 6y ago
Correct.

benfrederickson 6y ago

> you can either race with the program and hope that you read its memory to get the function stack before it changes what function it's running (and you're likely to win the race, because C is faster than Python), or you can pause the program briefly while taking a sample. Py-spy defaults to blocking because the results can be pretty wrong otherwise: https://github.com/benfred/py-spy/issues/56 https://github.com/benfred/py-spy/issues/56 . You can see this problem profiling a program like https://github.com/benfred/py-spy/blob/master/tests/scripts/recursive.py https://github.com/benfred/py-spy/blob/master/tests/scripts/... with or without the nonblocking flag in py-spy - the nonblocking version produces garbage output. Somewhat interestingly, this problem doesn't seem to occur with Ruby - and rbspy can get away without pausing the target program with only minor errors seen when profiling a similar function. I suspect this is because of differences between how the Ruby and Python interpreters store call stack information, but haven't had a chance to dig into the specifics.

teh_matt 6y ago

Note that tracing profilers don't need to be high overhead - Python is slow enough that efficient tracing can be mostly hidden. For example, https://functiontrace.com https://functiontrace.com tends to have <10% overhead when tracing.