4 ms·

Pyrasite: Tools for injecting code into running Python processes

89 points by tombenner 12y ago

AnkhMorporkian 12y ago
I've used this a few times to debug running processes that were having small issues that weren't bad enough to restart the system to put in a proper manhole, but bad enough to be affecting users. Worked perfectly to figure out what was going on, and to patch the naughty method. It's one of those tools that you'll be glad exists when the need arises, but you'll feel a little dirty using it.
- tenfingers 12y ago
  Why "dirty"? Attaching and inspecting a live process without cooperation is something that should be expected. pdb is quite limited in what it can debug if you have a stuck process which is very hard to reproduce. I've resorted to gdb + python extensions too many times to count (which basically allows you do what pyrasite does, though at a lower level).
tbarbugli 12y ago
Would be nice to know how this actually works; docs dont say much about how the tool injects the code into the running process.
- matthewmacleod 12y ago
  It basically uses GDB to call eval inside a running Python VM. Pretty simple, but quite useful from time to time.
  - BozeWolf 12y ago
    But how does it get into that running Python VM?
    - coolj 12y ago
      GDB uses a system call(s) to attach to the running process[1], and python exposes an API that lets you evaluate code in the current VM context[2]. Specifically, https://github.com/lmacken/pyrasite/blob/develop/pyrasite/injector.py#L22-42 https://github.com/lmacken/pyrasite/blob/develop/pyrasite/in... [1] http://aosabook.org/en/gdb.html http://aosabook.org/en/gdb.html [2] https://docs.python.org/2/c-api/veryhigh.html https://docs.python.org/2/c-api/veryhigh.html
      - BozeWolf 12y ago
        Excellent answer to my question. Thanks!
- klibertp 12y ago
  Just read this file: https://github.com/lmacken/pyrasite/blob/develop/pyrasite/injector.py https://github.com/lmacken/pyrasite/blob/develop/pyrasite/in... it's rather straightforward.
- ionelm 12y ago
  It's basically this: attach with GDB, get GIL, run commands (your session), release GIL. This is pretty dangerous as the GIL release is not guaranteed. This is why I wrote https://pypi.python.org/pypi/manhole https://pypi.python.org/pypi/manhole - I couldn't afford hosing up (deadlocking) my processes, and it happen pretty often with pyrasite in my usecases. Have in mind that manhole is not that flexible as it uses different technique.
mau 12y ago
I had some "memory leak" issues a few months ago with the api hosts of the service I'm working on. As the code changes made during the days before didn't seem to have affected anything that could be related to this leak I realized it was something about the requests we were receiving but I wasn't able to replicate the same issues in my dev host nor in staging. Eventually I dumped the memory of a running process using Pyrasite and after hacking a little bit with meliae (https://launchpad.net/meliae https://launchpad.net/meliae) I was able to find the issue and solve it. Very useful. By the way is worth noticing that you won't probably be able to attach the interactive console to a running web server as the output is usually handled by the supervisor process, at least I wasn't able to do that in my first tentative and the memory dump was good enough for me. Check this sample to dump the memory out of a running process: http://pyrasite.readthedocs.org/en/latest/Payloads.html#dumping-out-object-memory-usage-statistics http://pyrasite.readthedocs.org/en/latest/Payloads.html#dump...
klibertp 12y ago
Wow, live inspection of running code without prior instrumentation is a great feature of Erlang which I always missed in Python. I used `rconsole` from rfoo package (https://code.google.com/p/rfoo/ https://code.google.com/p/rfoo/) before, but this looks like a cleaner solution because you don't have to include server spawning code in the code you want to inspect. Very nice.

89 points by tombenner 12y ago

I've used this a few times to debug running processes that were having small issues that weren't bad enough to restart the system to put in a proper manhole, but bad enough to be affecting users. Worked perfectly to figure out what was going on, and to patch the naughty method. It's one of those tools that you'll be glad exists when the need arises, but you'll feel a little dirty using it.

tenfingers 12y ago
Why "dirty"? Attaching and inspecting a live process without cooperation is something that should be expected. pdb is quite limited in what it can debug if you have a stuck process which is very hard to reproduce. I've resorted to gdb + python extensions too many times to count (which basically allows you do what pyrasite does, though at a lower level).

tbarbugli 12y ago

Would be nice to know how this actually works; docs dont say much about how the tool injects the code into the running process.

matthewmacleod 12y ago
It basically uses GDB to call eval inside a running Python VM. Pretty simple, but quite useful from time to time.
- BozeWolf 12y ago
  But how does it get into that running Python VM?
  - coolj 12y ago
    GDB uses a system call(s) to attach to the running process[1], and python exposes an API that lets you evaluate code in the current VM context[2]. Specifically, https://github.com/lmacken/pyrasite/blob/develop/pyrasite/injector.py#L22-42 https://github.com/lmacken/pyrasite/blob/develop/pyrasite/in... [1] http://aosabook.org/en/gdb.html http://aosabook.org/en/gdb.html [2] https://docs.python.org/2/c-api/veryhigh.html https://docs.python.org/2/c-api/veryhigh.html
    - BozeWolf 12y ago
      Excellent answer to my question. Thanks!
klibertp 12y ago
Just read this file: https://github.com/lmacken/pyrasite/blob/develop/pyrasite/injector.py https://github.com/lmacken/pyrasite/blob/develop/pyrasite/in... it's rather straightforward.
ionelm 12y ago
It's basically this: attach with GDB, get GIL, run commands (your session), release GIL. This is pretty dangerous as the GIL release is not guaranteed. This is why I wrote https://pypi.python.org/pypi/manhole https://pypi.python.org/pypi/manhole - I couldn't afford hosing up (deadlocking) my processes, and it happen pretty often with pyrasite in my usecases. Have in mind that manhole is not that flexible as it uses different technique.

mau 12y ago

I had some "memory leak" issues a few months ago with the api hosts of the service I'm working on. As the code changes made during the days before didn't seem to have affected anything that could be related to this leak I realized it was something about the requests we were receiving but I wasn't able to replicate the same issues in my dev host nor in staging. Eventually I dumped the memory of a running process using Pyrasite and after hacking a little bit with meliae (https://launchpad.net/meliae https://launchpad.net/meliae) I was able to find the issue and solve it. Very useful. By the way is worth noticing that you won't probably be able to attach the interactive console to a running web server as the output is usually handled by the supervisor process, at least I wasn't able to do that in my first tentative and the memory dump was good enough for me. Check this sample to dump the memory out of a running process: http://pyrasite.readthedocs.org/en/latest/Payloads.html#dumping-out-object-memory-usage-statistics http://pyrasite.readthedocs.org/en/latest/Payloads.html#dump...

klibertp 12y ago

Wow, live inspection of running code without prior instrumentation is a great feature of Erlang which I always missed in Python. I used `rconsole` from rfoo package (https://code.google.com/p/rfoo/ https://code.google.com/p/rfoo/) before, but this looks like a cleaner solution because you don't have to include server spawning code in the code you want to inspect. Very nice.