13 ms·
Memray: a memory profiler for Python
- francoacevedo 4y ago
- deleted 4y ago[deleted]
- uniqueuid 4y agoTangential question: Does anybody have a good recommendation for a profiler that works well with massively async codebases? My experience has been that the concurrent nature of coroutines can make it hard to reason about what's going on at a particular point in time. If you don't know how many things you're awaiting on at a specific moment (and what potential external stuff they may be interacting with), it's not exactly easy to identify memory usage of codepaths.
- mjsir911 4y agoI've had good luck with yappi for debugging asynchronous code, not really focused on memory usage but walltime & such. https://pypi.org/project/yappi/ https://pypi.org/project/yappi/
- uniqueuid 4y agoThanks, that looks useful!
- zackangelo 4y agoDo you mean specifically for Python? There's this for async Rust: https://tokio.rs/blog/2021-12-announcing-tokio-console https://tokio.rs/blog/2021-12-announcing-tokio-console
- erwincoumans 4y agoI like open source Perfetto UI (formerly Google Chrome about::/tracing) Wrote my own timing trace json export for Python and C++, across threads and processes. The docs have some pointers to creating your own traces with the Tracing SDK. See https://ui.perfetto.dev https://ui.perfetto.dev
- ogrisel 4y agoviztracer can create traces for python programs. Not sure about async-io awareness though.
- ogrisel 4y agoActually it has explicit support for async task based reporting: https://github.com/gaogaotiantian/viztracer#async-support https://github.com/gaogaotiantian/viztracer#async-support
- knlb2022 4y agoI built https://github.com/kunalb/panopticon https://github.com/kunalb/panopticon to export perfetto/chrome compatible traces and also draw arrows between async functions. (I think the arrows are only supported in about://tracing though).
- sterlinm 4y agoLooks like you can use magic-trace with a perfetto based UI. https://github.com/janestreet/magic-trace https://github.com/janestreet/magic-trace
- kungfufrog 4y agohttps://github.com/plasma-umass/scalene https://github.com/plasma-umass/scalene check out scalene!
- Jarwain 4y ago+1 for scalene! I love how it breaks down consumption line-by-line, and it's worked very well in my async/multithreaded usecase
- p403n1x87 4y agoThis should work with anything https://github.com/P403n1x87/austin https://github.com/P403n1x87/austin
- shcheklein 4y agoA twitter thread with some screenshots and details - https://twitter.com/1st1/status/1516859294896906241 https://twitter.com/1st1/status/1516859294896906241
- neves 4y agoI've just put in production an app that intermittently can't allocate enough memory. Is this the best tool to debug it? I've never had to debug memory problems in Python.
- orf 4y agoIf this isn't due to latent memory pressure, it's usually because _something_ in your code is trying to allocate a gigantic amount of memory due to a bug. Think a 1 billion x 1 billion numpy array, or a billion element list.
- itamarst 4y agoFil can dump memory profiling reports on failed allocations; as other commenter said, Linux by default happily just gives you memory even if doesn't have any, so failed allocation implies giant allocation. https://pythonspeed.com/fil/docs/oom.html https://pythonspeed.com/fil/docs/oom.html
- noitpmeder 4y agoIs this due to memory constraints on your system? If so, then this would definitely help you profile _what_ in your code is trying to claim so much memory.
- dbenamy 4y agoDatadog's profiler [0] might help. The python profiler has a heap profile type which shows (the call site which allocates) data using memory [1]. Feedback welcome! Disclosure: I work on this :-) [0] https://app.datadoghq.com/profiling https://app.datadoghq.com/profiling [1] From https://docs.datadoghq.com/tracing/profiler/search_profiles/ https://docs.datadoghq.com/tracing/profiler/search_profiles/ > Heap Live Size > Shows the amount of heap memory allocated by each function that has not been garbage collected (yet). This is useful for investigating the overall memory usage of your service and identifying potential memory leaks.
- vdfs 4y agoYes, I just give this tool a try and reduced our app initial memory allocation from 120MB to about 80MB in few minutes by moving imports around, ex: rarely called functions importing big 3rd party library. You can use this to figure out what need to be optimized, but in your case optimization might not need to be easy, in above example you would have to re-implement the 3rd party library in memory efficient way or find an other one
- deleted 4y ago[deleted]
- itamarst 4y agoI'm excited to see more profiling tools for Python! This sounds like it does peak memory, which is critical for batch jobs, since that's the bottleneck. Memory is fundamentally different than performance in that it's a limited resource, instead of cumulative cost; making any part of the program faster almost always helps speed up the program (at least a little, or at least reduces CPU load), but optimizing non-peak memory has no impact. You have to be able to identify the peak in order to reduce memory usage. If you want peak memory profiling for Python that also runs on macOS, check out https://pythonspeed.com/fil/ https://pythonspeed.com/fil/ (ARM support has some issues, but once I unpack my new Mac Mini I plan to fix it.) Ways memray is better than Fil: - Native callstacks. - More kinds of reports, and ability to do custom post-processing of data. - Much lower overhead (but not always, see reply). - Subprocess support. Fil I suspect has better flamegraphs: https://pythonspeed.com/articles/a-better-flamegraph/ https://pythonspeed.com/articles/a-better-flamegraph/ And if you're running Python batch jobs, and want both peak memory and performance profiling in production, check out Sciagraph: https://pythonspeed.com/sciagraph/ https://pythonspeed.com/sciagraph/ (You can probably cobble together something like Sciagraph with py-spy + memray, but you won't e.g. get timeline reports designed with batch jobs in mind.)
- pablogsal 4y agoIt does much more than that! It tracks every single allocation and dumps it to a file that can later be analysed in many ways. Currently our reporters report peak memory (and leaked memory at the end of the execution) but technically any other reporter can be used. For example, we plan to allow to generate flame-graphs at arbitrary points in the execution and much more!
- noitpmeder 4y agoWant to chime in to say it's awesome that Bloomberg is releasing this product to the world. I don't think many financial organizations would even consider releasing anything as open source, even if it is as general purpose as this tool. My current one (HFT,MM) wouldn't go for it. It also shows good engineering practices. I'd venture a guess that they are probably one of the better ones in the space. Makes me want to work there! Maybe one day...
- pjmlp 4y agoThey also release a couple other stuff, https://github.com/bloomberg https://github.com/bloomberg
- francoacevedo 4y ago
- apaprocki 4y agoWell, today is one day... :) You didn't hint at the reason for saying that, but please don't hesitate to apply if you're at all interested.[1] It would make more sense to say "financial data organization", which then makes it a bit more obvious why we'd have a whole lot of employees (6,500+) in Engineering compared to a typical "financial organization" making money primarily by participating in the public/private markets. [1]: https://www.bloomberg.com/company/what-we-do/ https://www.bloomberg.com/company/what-we-do/
- sterlinm 4y ago> I don't think many financial organizations would even consider releasing anything as open source, even if it is as general purpose as this tool. Not to discount how great it is that Bloomberg is contributing this to the community, but there are a lot of other interesting open source tools that have been open sourced by by finance firms. Pandas, for one, was originally developed internally at the hedge fund AQR.
- sterlinm 4y agoSome other interesting projects from financial services firms: https://perspective.finos.org/ https://perspective.finos.org/ (JP Morgan) https://github.com/jpmorganchase/regular-table https://github.com/jpmorganchase/regular-table (JP Morgan) https://github.com/man-group/dtale https://github.com/man-group/dtale (Man Numeric) https://github.com/deshaw/pyflyby https://github.com/deshaw/pyflyby (DE Shaw)
- polskibus 4y agoCan I plug it into existing airflow instance and check all the dags somehow? If not , what would be the easiest way to profile each dag?
- isaacfrond 4y agoIf you need this, why are you using Python? Seriously, can anyone explain that to me?
- Chiron1991 4y agoFirst and foremost for debugging purposes, e.g. to find memory leaks and such. Then, it's not a bad thing to optimize memory usage in any language. Memray could be a great tool for the authors of data science libraries, which Python has plenty of. Their average user probably doesn't have a beefy developer's laptop with 64GB of memory at hand.
- dmalecaut 4y agoSubstantial python codebase already?
- fer 4y agoBecause Python might be the best tool for a job that has memory constraints?
- anonu 4y agoWhat's your underlying point here? That people shouldn't be doing things in python that take up a lot of memory?
- ajoseps 4y agodebugging high memory usage in python can be annoying. python is used a lot for data analysis e.g. pandas, numpy. You might have scenarios where some dataframe either generated or read, is a lot bigger than others for whatever reason. This might be useful in debugging as to why that's the case.
- anonu 4y agoInteresting that this comes from Bloomberg.
- frays 4y agoCame here to say this.
- kristianpaul 4y agoThis is awesome. Is there is something like this for Ruby?
- p403n1x87 4y agoHmm some elements of that TUI look vaguely familiar ... where have I seen them before?
- user1713952 4y agoReally impressive feature set! I just wish it could run on windows too.
- sterlinm 4y agoFil doesn't run on Windows either. Are there any good memory profilers that work on Windows?
- ewuhic 4y agoCould anyone suggest a similar quality CPU load python profiler?