22 ms·

Show HN: Austin-Tui – Spy inside a running Python program at no performance cost

256 points by p403n1x87 6y ago

tr33house 6y ago
looks awesome. how easy is it to use this to profile a webapp written in a framework like django? Open-metrics? just curious
- p403n1x87 6y ago
  as easy as profiling with Austin itself. Just pass the pid of the web server sudo austin-tui -Cp <webserverpid> See https://github.com/P403n1x87/austin#examples https://github.com/P403n1x87/austin#examples for more details. To avoid using sudo on Linux you can do sudo setcap cap_sys_ptrace+ep `which austin` and then simply austin-tui -Cp <webserverpid>
- galacticdessert 6y ago
  Pyinstrument might be easier to profile web requests: https://github.com/joerick/pyinstrument#profile-a-web-request-in-django https://github.com/joerick/pyinstrument#profile-a-web-reques...
  - p403n1x87 6y ago
    One benefit of Austin-based tools is that they don't require any instrumentation/extra configuration. The low overhead means that you can just attach to an application that is running in production.
jvns 6y ago
Two other profilers that also let you spy on a running Python program: - py-spy: https://github.com/benfred/py-spy https://github.com/benfred/py-spy (written in Rust) - pyflame: https://github.com/uber-archive/pyflame https://github.com/uber-archive/pyflame (C++, seems to be not maintained anymore) The "no performance cost" thing is interesting: my experience writing a similar profiler is that there are a couple of things that can affect performance a little bit: 1. You have to make a lot of system calls to read the memory of the target process, and if you want to sample at a high rate then that does use some CPU. This can be an issue if you only have 1 CPU. 2. you have two choices when reading memory from a process: you can either race with the program and hope that you read its memory to get the function stack before it changes what function it's running (and you're likely to win the race, because C is faster than Python), or you can pause the program briefly while taking a sample. py-spy has an option to choose which one you want to do: https://github.com/benfred/py-spy#how-can-i-avoid-pausing-the-python-program https://github.com/benfred/py-spy#how-can-i-avoid-pausing-th... Definitely this method is a lot lower overhead than a tracing profiler that instruments every single function call, and in practice it works well. One thing I think is nice about this kind of profiler is that reading memory from the target process sounds like a complicated thing, but it's not: you can see austin's code for reading memory here, and it's implemented for 3 platforms in just 130 lines of C: https://github.com/P403n1x87/austin/blob/877e2ff946ea5313e4773b4ee1d01da853b5b31d/src/mem.h#L98 https://github.com/P403n1x87/austin/blob/877e2ff946ea5313e47...
- p403n1x87 6y ago
  All good points. The "no performance cost" is indeed more like "negligible performance costs". That's because these days multicore architectures are quite ubiquitous and standard Python applications are single process. For multi-process Python applications, a busy profiler would certainly steal a good chunk of a core, so the impact might be noticeable in that case. As for the race conditions, Austin does not introduce any pauses. Even if it did, there would be no guarantee that it paused at a "good" point, so there are no real benefits in terms of accuracy in pausing. Error rates are quite low anyway, so the actual benefit comes from not pausing at all.
- jvns 6y ago
  Also, this kind of profiler is great because you can use it on any running Python program, which is pretty magical and very useful. (especially when it's an application you didn't write) But it's not right for every use case: by design austin/py-spy can only really profile the whole program, and if you want to profile a specific function or endpoint in your program, something like PyInstrument https://github.com/joerick/pyinstrument https://github.com/joerick/pyinstrument (which includes Django middlewares & Flask decorators) is a lot more useful.
  - p403n1x87 6y ago
    For these kind of things one could use https://github.com/P403n1x87/austin-python https://github.com/P403n1x87/austin-python An early attempt at an APM that was specifically designed to measure endpoint response times was here https://github.com/P403n1x87/austin/blob/apm/austin/apm.py https://github.com/P403n1x87/austin/blob/apm/austin/apm.py The problem with statistical profilers is that they might fail to catch every single call, especially when they are fast.
- dTal 6y ago
  >you're likely to win the race, because C is faster than Python Heh, so the only reason there's "no performance cost" is the enormous performance cost of using Python in the first place?
  - 6y ago
emddudley 6y ago
This is a TUI for the profiler Austin, whose README has a bit more detail on what it can do: https://github.com/P403n1x87/austin https://github.com/P403n1x87/austin
shepardrtc 6y ago
Love the TUI! Which library did you use for it?
- p403n1x87 6y ago
  I wrote a blog post about the technology behind the TUI https://p403n1x87.github.io/the-austin-tui-way-to-resourceful-text-based-user-interfaces.html https://p403n1x87.github.io/the-austin-tui-way-to-resourcefu... It's a custom resource-based framework that uses curses as back-end.
  - kingosticks 6y ago
    This looks really interesting - thanks! One question: in the first XML example (minimal-view.xml), why is the root element a <aui:MinimalView ...> but then the last line ends it with </aui:MiniTop> ? Is it a typo or is there something else going on?
    - p403n1x87 6y ago
      Ah! Totally a typo, thanks for that :)
Pulcinella 6y ago
What’s the difference between a TUI and a CLI? (Also my preferred TUI acronym is tactile user interface).
- emddudley 6y ago
  I would say that a CLI is primarily for entering commands, while a TUI is primarily for viewing and interacting with data.
- m4r35n357 6y ago
  TOP User interface (from TFA).
- frou_dh 6y ago
  If a CLI program ceases to make sense when its standard-output is a physical line-printer onto a roll of paper, then it's a TUI.
- ktpsns 6y ago
  Another example: https://en.wikipedia.org/wiki/Ed_(text_editor) https://en.wikipedia.org/wiki/Ed_(text_editor) has a CLI, but https://en.wikipedia.org/wiki/Vim_(text_editor) https://en.wikipedia.org/wiki/Vim_(text_editor) has a TUI.
- MaxBarraclough 6y ago
  Both use the terminal, but in different ways. A CLI interface uses the basic terminal functionality of writing and perhaps reading lines, or even simply accepting flags and running non-interactively. A TUI uses the terminal's advanced features, using the terminal as the basis for a kind of GUI, generally occupying the full area of the terminal. Vim, Midnight Commander, and htop have TUIs. They rely heavily on the terminal's 'control character' features to accomplish this. apt-get has a CLI, as its interactive IO is handled with printing lines and having the user submit lines (even if it's just the letter y). Somewhere in the middle are interfaces like bash and zsh which make light use of the terminal's advanced features for things like auto-complete, but which don't take over the whole terminal area. I'd count non-interactive applications like gcc and sort as command-line applications, although strictly speaking you could just as well use a graphical interface to configure their flags and run commands. See also https://en.wikipedia.org/wiki/Text-based_user_interface https://en.wikipedia.org/wiki/Text-based_user_interface
iwebdevfromhome 6y ago
Is it possible to spy from a python script running in a virtual python 2 environment?
- p403n1x87 6y ago
  yep :)
  - iwebdevfromhome 6y ago
    Great news, can't wait to try it on some long running reports.
RantyDave 6y ago
Wasn't there lots of work around using dtrace for this?
vvladymyrov 6y ago
Can Austin attach to python process running in the docker container from the host system? `sudo austin-tui -Cp <pid>` I used pyflame for that at some point.
- p403n1x87 6y ago
  That should work

256 points by p403n1x87 6y ago

looks awesome. how easy is it to use this to profile a webapp written in a framework like django? Open-metrics? just curious

p403n1x87 6y ago
as easy as profiling with Austin itself. Just pass the pid of the web server sudo austin-tui -Cp <webserverpid> See https://github.com/P403n1x87/austin#examples https://github.com/P403n1x87/austin#examples for more details. To avoid using sudo on Linux you can do sudo setcap cap_sys_ptrace+ep `which austin` and then simply austin-tui -Cp <webserverpid>
galacticdessert 6y ago
Pyinstrument might be easier to profile web requests: https://github.com/joerick/pyinstrument#profile-a-web-request-in-django https://github.com/joerick/pyinstrument#profile-a-web-reques...
- p403n1x87 6y ago
  One benefit of Austin-based tools is that they don't require any instrumentation/extra configuration. The low overhead means that you can just attach to an application that is running in production.

jvns 6y ago

Two other profilers that also let you spy on a running Python program: - py-spy: https://github.com/benfred/py-spy https://github.com/benfred/py-spy (written in Rust) - pyflame: https://github.com/uber-archive/pyflame https://github.com/uber-archive/pyflame (C++, seems to be not maintained anymore) The "no performance cost" thing is interesting: my experience writing a similar profiler is that there are a couple of things that can affect performance a little bit: 1. You have to make a lot of system calls to read the memory of the target process, and if you want to sample at a high rate then that does use some CPU. This can be an issue if you only have 1 CPU. 2. you have two choices when reading memory from a process: you can either race with the program and hope that you read its memory to get the function stack before it changes what function it's running (and you're likely to win the race, because C is faster than Python), or you can pause the program briefly while taking a sample. py-spy has an option to choose which one you want to do: https://github.com/benfred/py-spy#how-can-i-avoid-pausing-the-python-program https://github.com/benfred/py-spy#how-can-i-avoid-pausing-th... Definitely this method is a lot lower overhead than a tracing profiler that instruments every single function call, and in practice it works well. One thing I think is nice about this kind of profiler is that reading memory from the target process sounds like a complicated thing, but it's not: you can see austin's code for reading memory here, and it's implemented for 3 platforms in just 130 lines of C: https://github.com/P403n1x87/austin/blob/877e2ff946ea5313e4773b4ee1d01da853b5b31d/src/mem.h#L98 https://github.com/P403n1x87/austin/blob/877e2ff946ea5313e47...

p403n1x87 6y ago
All good points. The "no performance cost" is indeed more like "negligible performance costs". That's because these days multicore architectures are quite ubiquitous and standard Python applications are single process. For multi-process Python applications, a busy profiler would certainly steal a good chunk of a core, so the impact might be noticeable in that case. As for the race conditions, Austin does not introduce any pauses. Even if it did, there would be no guarantee that it paused at a "good" point, so there are no real benefits in terms of accuracy in pausing. Error rates are quite low anyway, so the actual benefit comes from not pausing at all.
jvns 6y ago
Also, this kind of profiler is great because you can use it on any running Python program, which is pretty magical and very useful. (especially when it's an application you didn't write) But it's not right for every use case: by design austin/py-spy can only really profile the whole program, and if you want to profile a specific function or endpoint in your program, something like PyInstrument https://github.com/joerick/pyinstrument https://github.com/joerick/pyinstrument (which includes Django middlewares & Flask decorators) is a lot more useful.
- p403n1x87 6y ago
  For these kind of things one could use https://github.com/P403n1x87/austin-python https://github.com/P403n1x87/austin-python An early attempt at an APM that was specifically designed to measure endpoint response times was here https://github.com/P403n1x87/austin/blob/apm/austin/apm.py https://github.com/P403n1x87/austin/blob/apm/austin/apm.py The problem with statistical profilers is that they might fail to catch every single call, especially when they are fast.
dTal 6y ago
>you're likely to win the race, because C is faster than Python Heh, so the only reason there's "no performance cost" is the enormous performance cost of using Python in the first place?
- 6y ago

emddudley 6y ago

This is a TUI for the profiler Austin, whose README has a bit more detail on what it can do: https://github.com/P403n1x87/austin https://github.com/P403n1x87/austin

shepardrtc 6y ago

Love the TUI! Which library did you use for it?

p403n1x87 6y ago
I wrote a blog post about the technology behind the TUI https://p403n1x87.github.io/the-austin-tui-way-to-resourceful-text-based-user-interfaces.html https://p403n1x87.github.io/the-austin-tui-way-to-resourcefu... It's a custom resource-based framework that uses curses as back-end.
- kingosticks 6y ago
  This looks really interesting - thanks! One question: in the first XML example (minimal-view.xml), why is the root element a <aui:MinimalView ...> but then the last line ends it with </aui:MiniTop> ? Is it a typo or is there something else going on?
  - p403n1x87 6y ago
    Ah! Totally a typo, thanks for that :)

Pulcinella 6y ago

What’s the difference between a TUI and a CLI? (Also my preferred TUI acronym is tactile user interface).

emddudley 6y ago
I would say that a CLI is primarily for entering commands, while a TUI is primarily for viewing and interacting with data.
m4r35n357 6y ago
TOP User interface (from TFA).
frou_dh 6y ago
If a CLI program ceases to make sense when its standard-output is a physical line-printer onto a roll of paper, then it's a TUI.
ktpsns 6y ago
Another example: https://en.wikipedia.org/wiki/Ed_(text_editor) https://en.wikipedia.org/wiki/Ed_(text_editor) has a CLI, but https://en.wikipedia.org/wiki/Vim_(text_editor) https://en.wikipedia.org/wiki/Vim_(text_editor) has a TUI.
MaxBarraclough 6y ago
Both use the terminal, but in different ways. A CLI interface uses the basic terminal functionality of writing and perhaps reading lines, or even simply accepting flags and running non-interactively. A TUI uses the terminal's advanced features, using the terminal as the basis for a kind of GUI, generally occupying the full area of the terminal. Vim, Midnight Commander, and htop have TUIs. They rely heavily on the terminal's 'control character' features to accomplish this. apt-get has a CLI, as its interactive IO is handled with printing lines and having the user submit lines (even if it's just the letter y). Somewhere in the middle are interfaces like bash and zsh which make light use of the terminal's advanced features for things like auto-complete, but which don't take over the whole terminal area. I'd count non-interactive applications like gcc and sort as command-line applications, although strictly speaking you could just as well use a graphical interface to configure their flags and run commands. See also https://en.wikipedia.org/wiki/Text-based_user_interface https://en.wikipedia.org/wiki/Text-based_user_interface

iwebdevfromhome 6y ago

Is it possible to spy from a python script running in a virtual python 2 environment?

p403n1x87 6y ago
yep :)
- iwebdevfromhome 6y ago
  Great news, can't wait to try it on some long running reports.

RantyDave 6y ago

Wasn't there lots of work around using dtrace for this?

vvladymyrov 6y ago

Can Austin attach to python process running in the docker container from the host system? `sudo austin-tui -Cp <pid>` I used pyflame for that at some point.

p403n1x87 6y ago
That should work