4 ms·

Building FunctionTrace, a graphical Python profiler

53 points by whocanfly 6y ago

JackC 6y ago
"Having a Python C module gives significantly more power and performance, but comes with costs. it requires more code, it’s harder to find good documentation; and few features are easily accessible. While C modules appear to be an under-utilized tool for writing high performance Python modules (based on some FunctionTrace profiles I’ve seen), we’d recommend a balance. Write most of the non-performance critical code in Python and call into inner loops or setup code in C, for the pieces where Python doesn’t shine." I've been wondering recently about the maintainability/community aspect of C vs. pure-python packages. I was choosing between a pure-python simhash library and 5-times-as-fast library with a C++ core, and ultimately had to choose pure-python because the C++ one doesn't have recent releases, can't be reliably installed from source, and is harder for me to fix if it breaks. In other words it suffers from a lower bus facter, which I'd predict is generally true of Python packages with a fast non-Python core. This got me wondering, what is the way to write a fast core for a Python package that is most ... likely to survive, I guess, once the original maintainer loses interest? Would it be some form of C (I think necessary in this case), Rust (once the tooling gels a little), lua, wasm, something else?
- LittlePeter 6y ago
  Would wheels be an option to avoid compiling C/C++ source. The package author would then need to build wheels for the target platforms though.
cynik_ 6y ago
I wrote and open sourced something very similar recently: https://github.com/kunalb/panopticon https://github.com/kunalb/panopticon. I don't think it's as sophisticated as function trace, but it generates chrome/catapult compatible traces instead and has some additions for async coroutines/generators. Firefox's profiler looks pretty cool, I wish it could also accept catapult traces as is -- it doesn't seem to be the case. Now I know of: - chrome://tracing - Firefox's profiler - Perfetto: https://ui.perfetto.dev/#!/ https://ui.perfetto.dev/#!/ - Speedscope: https://www.speedscope.app/ https://www.speedscope.app/ - Brendan Gregg's flamegraph generator to help visualize traces. Are there other good ones?
person_of_color 6y ago
How does this compare to py-spy?

53 points by whocanfly 6y ago

"Having a Python C module gives significantly more power and performance, but comes with costs. it requires more code, it’s harder to find good documentation; and few features are easily accessible. While C modules appear to be an under-utilized tool for writing high performance Python modules (based on some FunctionTrace profiles I’ve seen), we’d recommend a balance. Write most of the non-performance critical code in Python and call into inner loops or setup code in C, for the pieces where Python doesn’t shine." I've been wondering recently about the maintainability/community aspect of C vs. pure-python packages. I was choosing between a pure-python simhash library and 5-times-as-fast library with a C++ core, and ultimately had to choose pure-python because the C++ one doesn't have recent releases, can't be reliably installed from source, and is harder for me to fix if it breaks. In other words it suffers from a lower bus facter, which I'd predict is generally true of Python packages with a fast non-Python core. This got me wondering, what is the way to write a fast core for a Python package that is most ... likely to survive, I guess, once the original maintainer loses interest? Would it be some form of C (I think necessary in this case), Rust (once the tooling gels a little), lua, wasm, something else?

LittlePeter 6y ago
Would wheels be an option to avoid compiling C/C++ source. The package author would then need to build wheels for the target platforms though.

cynik_ 6y ago

I wrote and open sourced something very similar recently: https://github.com/kunalb/panopticon https://github.com/kunalb/panopticon. I don't think it's as sophisticated as function trace, but it generates chrome/catapult compatible traces instead and has some additions for async coroutines/generators. Firefox's profiler looks pretty cool, I wish it could also accept catapult traces as is -- it doesn't seem to be the case. Now I know of: - chrome://tracing - Firefox's profiler - Perfetto: https://ui.perfetto.dev/#!/ https://ui.perfetto.dev/#!/ - Speedscope: https://www.speedscope.app/ https://www.speedscope.app/ - Brendan Gregg's flamegraph generator to help visualize traces. Are there other good ones?

person_of_color 6y ago

How does this compare to py-spy?