4 ms·
I'm a big fan of pyinstrument. Many of the newer profilers, (e.g. py-spy) attach to a process externally via SYS_PTRACE and though that seems great in many ways
by jrichardshaw 6y ago
I'm a big fan of pyinstrument. Many of the newer profilers, (e.g. py-spy) attach to a process externally via SYS_PTRACE and though that seems great in many ways, it is very much a no-go when you're running code on an HPC cluster and you don't have root access.
- kapilvt 6y agoptrace is quite slow, py-spy reads memory and reconstructs stack frames which is much lower overhead for the profiled app. from their github readme: py-spy works by directly reading the memory of the python program using the process_vm_readv system call on Linux, the vm_read call on OSX or the ReadProcessMemory call on Windows.
- gnufx 6y agoOn an HPC system, why wouldn't you use normal HPC-type performance tools? It will probably give you better analysis tools and allow you to profile/trace the libraries which are probably providing the performance. At least Score-p, Extrae, and TAU have Python support and have sophisticated viewers with support for call trees and inclusive/exclusive views. When I looked (maybe a couple of years ago) I couldn't find anything in the Python ecosystem to do that, though there may be now.