5 ms·
Tangential question: Does anybody have a good recommendation for a profiler that works well with massively async codebases? My experience has been that the con
by uniqueuid 4y ago
Tangential question: Does anybody have a good recommendation for a profiler that works well with massively async codebases?
My experience has been that the concurrent nature of coroutines can make it hard to reason about what's going on at a particular point in time. If you don't know how many things you're awaiting on at a specific moment (and what potential external stuff they may be interacting with), it's not exactly easy to identify memory usage of codepaths.
- mjsir911 4y agoI've had good luck with yappi for debugging asynchronous code, not really focused on memory usage but walltime & such. https://pypi.org/project/yappi/ https://pypi.org/project/yappi/
- uniqueuid 4y agoThanks, that looks useful!
- zackangelo 4y agoDo you mean specifically for Python? There's this for async Rust: https://tokio.rs/blog/2021-12-announcing-tokio-console https://tokio.rs/blog/2021-12-announcing-tokio-console
- erwincoumans 4y agoI like open source Perfetto UI (formerly Google Chrome about::/tracing) Wrote my own timing trace json export for Python and C++, across threads and processes. The docs have some pointers to creating your own traces with the Tracing SDK. See https://ui.perfetto.dev https://ui.perfetto.dev
- ogrisel 4y agoviztracer can create traces for python programs. Not sure about async-io awareness though.
- ogrisel 4y agoActually it has explicit support for async task based reporting: https://github.com/gaogaotiantian/viztracer#async-support https://github.com/gaogaotiantian/viztracer#async-support
- knlb2022 4y agoI built https://github.com/kunalb/panopticon https://github.com/kunalb/panopticon to export perfetto/chrome compatible traces and also draw arrows between async functions. (I think the arrows are only supported in about://tracing though).
- sterlinm 4y agoLooks like you can use magic-trace with a perfetto based UI. https://github.com/janestreet/magic-trace https://github.com/janestreet/magic-trace
- kungfufrog 4y agohttps://github.com/plasma-umass/scalene https://github.com/plasma-umass/scalene check out scalene!
- Jarwain 4y ago+1 for scalene! I love how it breaks down consumption line-by-line, and it's worked very well in my async/multithreaded usecase
- p403n1x87 4y agoThis should work with anything https://github.com/P403n1x87/austin https://github.com/P403n1x87/austin