9 ms·

Visual profiler for Python

213 points by nvdv 10y ago

fabioz 10y ago
http://www.pyvmmonitor.com http://www.pyvmmonitor.com also provides a nice visual profiler.
- infinite8s 10y ago
  I noticed that PyVMMonitor is built on PyQt. Mind if I ask you some questions about how you licensed the underlying tech stack to be able to sell it?
gsmethells 10y ago
Cool!
romanovcode 10y ago
You can also just use Python for Visual Studio[0] if you're on Windows. [0] https://www.visualstudio.com/en-us/features/python-vs.aspx https://www.visualstudio.com/en-us/features/python-vs.aspx
- dancsi 10y ago
  Is it just me, or running programs under PTVS slows them down by quite a bit? Even when I turn off debugging, it still seems to be slower than when I run the program from the terminal.
  - chuckkir 10y ago
    It is slower for me as well, but the debugger is excellent.
    - smortaz 10y ago
      perhaps hitting F5 instead of ctrl-F5 (run w/o debugging)?
  - int_19h 10y ago
    Can you clarify what you mean by "turn off debugging"? Are you doing Project | Start without Debugging?
- smortaz 10y ago
  Here's a YT video that shows off the profiler in PTVS (warning - loud cowbells) edit: forgot link - https://www.youtube.com/watch?v=K-KqkFkp55k https://www.youtube.com/watch?v=K-KqkFkp55k
  - JBReefer 10y ago
    I think you forgot the video!
    - smortaz 10y ago
      oops, thx, fixed.
devy 10y ago
How do I use this visual profiler to profile a Flask app?
- markbnj 10y ago
  You'd just run the flask app under the profiler and send requests as you normally would, right? Am I missing something?
  - nvdv 10y ago
    I haven't profiled any webapps yet, but you can profile functions inside your Flask app. First you start profiler in remote mode vprof -r and then use vprof.profiler.run function: profiler.run(foo, 'cmh', args=(arg1, arg2), host='localhost', port=8000) (replace host and port if you run your webappp on another maching with address of machine you run "vprof -r" on). This will run profiler and send stats to server in JSON format.
devy 10y ago
Intellij-based PyCharm uses yappi[1] or cProfile to provide similar visual profiling functionalities. [1]: https://bitbucket.org/sumerc/yappi/ https://bitbucket.org/sumerc/yappi/
oneweekwonder 10y ago
I like the flame graphs. Reminded me about the Netflix JavaScript Talks - Debugging Node.js in Production[0] talk, 11 min in. [0]: https://www.youtube.com/watch?v=O1YP8QP9gLA https://www.youtube.com/watch?v=O1YP8QP9gLA
- mdaniel 10y ago
  Just as an FYI, youtube links support a time bookmark at the end of the URL with `?v=CAFEBABE#t=11m0s` (in your example); the "share" widget on the site will encode the bookmark for you, if you prefer that method.
- ghostly_s 10y ago
  As a Python learner, very excited to try this out on my code. I'm aware of other tools that can generate these visualizations but figuring out how to use them has seemed like a daunting task for someone like myself who barely understands what bytecode is; hoping this tool will serve as an accessible learning tool!
kolapuriya 10y ago
KCacheGrind relies on using valgrind, which gives you extremely detailed information on runtime information by wrapping your execution context at a very low level. For this reason, it can track individual memory allocations, function call statistics, etc. However, for Python, that can be a bit overkill; usually you're just looking to measure what parts of your code are taking up executing time and heap space. This package uses cProfile, psutils, and pstats go get runtime information on your code as it's running. It's doing what valgrind does but in a much "higher-level" fashion, in pure Python. This means that it can give you more relevant information about your Python code.
- _yosefk 10y ago
  Actually KCachegrind displays files in an open format and is great for visualizing profiling info obtained from other tools.
- deleted 10y ago
  [deleted]
- forgotpwtomain 10y ago
  > It's doing what valgrind does but in a much "higher-level" fashion, in pure Python. Valgrind is essentially a VM, this uses cProfile - I wouldn't call that doing the 'same thing' at all.
vegabook 10y ago
Looks like an awesome piece of technology, but why, whenever I see headline xxxx followed by "(github.com)" do I get a slight feeling of despair - that feeling that this project has a high chance of being future abandonware. Is it that having a proper, dedicated, expensive-looking, website, away from github, is a signal of access to resources, willingness to do proper marketing and documentation, and therefore, by correlation, longevity?
- rhinoceraptor 10y ago
  If it's going to be abandonware, isn't it better to be on Github where it's easily accessible, rather than a random git server that could disappear?
- notifier2050 10y ago
  Like with almost any open source project nobody promises you anything. But it's not framework, you don't lose much by not using it and you don't have to rewrite your code in case it would be abandoned.
  - vegabook 10y ago
    yes I realise that I'm looking a gift horse in the mouth, and I love git(hub), but I'm just pointing out that "(github.com)" might just be a net-negative marketing point for people looking for robust software.
fijal 10y ago
I would like to complain about name similarity to vmprof (my project)
- daveguy 10y ago
  You're kidding, right?
fijal 10y ago
TL;DR; it's a visualizer for cProfile (with all the cProfile problems) DISCLAIMER: I'm an author of cProfile competitor, vmprof
albasha 10y ago
Does it work with Django? If so, how?
JonathonW 10y ago
Neat-- wish I'd had this a few weeks ago when I was trying to track down some performance issues. (Ended up getting what I needed out of cProfile, which this also uses on the backend. Would've been nice to have the nice visualizations, though.)
DiThi 10y ago
Is there a similar tool of the code heatmap but for JS?
- brndn 10y ago
  Lots of info on that here: http://www.brendangregg.com/flamegraphs.html http://www.brendangregg.com/flamegraphs.html
  - DiThi 10y ago
    I mean the code coverage tool, not the flame graphs. To e.g. know what paths of the code are never executed. Chrome and Firefox already show flame graphs in their profilers.
grinich 10y ago
We did something similar at Nylas, but we built it to run in production across our entire fleet: https://nylas.com/blog/performance/ https://nylas.com/blog/performance/
roberdam 10y ago
"RuntimeError: maximum recursion depth exceeded while calling a Python object " When installing.
oxplot 10y ago
I have just recently discovered chrome://tracing and wrote a quick python profiler [1] to output in its format. [1]: https://github.com/oxplot/crprof https://github.com/oxplot/crprof

213 points by nvdv 10y ago

fabioz 10y ago

http://www.pyvmmonitor.com http://www.pyvmmonitor.com also provides a nice visual profiler.

infinite8s 10y ago
I noticed that PyVMMonitor is built on PyQt. Mind if I ask you some questions about how you licensed the underlying tech stack to be able to sell it?

gsmethells 10y ago

Cool!

romanovcode 10y ago

You can also just use Python for Visual Studio[0] if you're on Windows. [0] https://www.visualstudio.com/en-us/features/python-vs.aspx https://www.visualstudio.com/en-us/features/python-vs.aspx

dancsi 10y ago
Is it just me, or running programs under PTVS slows them down by quite a bit? Even when I turn off debugging, it still seems to be slower than when I run the program from the terminal.
- chuckkir 10y ago
  It is slower for me as well, but the debugger is excellent.
  - smortaz 10y ago
    perhaps hitting F5 instead of ctrl-F5 (run w/o debugging)?
- int_19h 10y ago
  Can you clarify what you mean by "turn off debugging"? Are you doing Project | Start without Debugging?
smortaz 10y ago
Here's a YT video that shows off the profiler in PTVS (warning - loud cowbells) edit: forgot link - https://www.youtube.com/watch?v=K-KqkFkp55k https://www.youtube.com/watch?v=K-KqkFkp55k
- JBReefer 10y ago
  I think you forgot the video!
  - smortaz 10y ago
    oops, thx, fixed.

devy 10y ago

How do I use this visual profiler to profile a Flask app?

markbnj 10y ago
You'd just run the flask app under the profiler and send requests as you normally would, right? Am I missing something?
- nvdv 10y ago
  I haven't profiled any webapps yet, but you can profile functions inside your Flask app. First you start profiler in remote mode vprof -r and then use vprof.profiler.run function: profiler.run(foo, 'cmh', args=(arg1, arg2), host='localhost', port=8000) (replace host and port if you run your webappp on another maching with address of machine you run "vprof -r" on). This will run profiler and send stats to server in JSON format.

devy 10y ago

Intellij-based PyCharm uses yappi[1] or cProfile to provide similar visual profiling functionalities. [1]: https://bitbucket.org/sumerc/yappi/ https://bitbucket.org/sumerc/yappi/

oneweekwonder 10y ago

I like the flame graphs. Reminded me about the Netflix JavaScript Talks - Debugging Node.js in Production[0] talk, 11 min in. [0]: https://www.youtube.com/watch?v=O1YP8QP9gLA https://www.youtube.com/watch?v=O1YP8QP9gLA

mdaniel 10y ago
Just as an FYI, youtube links support a time bookmark at the end of the URL with `?v=CAFEBABE#t=11m0s` (in your example); the "share" widget on the site will encode the bookmark for you, if you prefer that method.
ghostly_s 10y ago
As a Python learner, very excited to try this out on my code. I'm aware of other tools that can generate these visualizations but figuring out how to use them has seemed like a daunting task for someone like myself who barely understands what bytecode is; hoping this tool will serve as an accessible learning tool!

kolapuriya 10y ago

KCacheGrind relies on using valgrind, which gives you extremely detailed information on runtime information by wrapping your execution context at a very low level. For this reason, it can track individual memory allocations, function call statistics, etc. However, for Python, that can be a bit overkill; usually you're just looking to measure what parts of your code are taking up executing time and heap space. This package uses cProfile, psutils, and pstats go get runtime information on your code as it's running. It's doing what valgrind does but in a much "higher-level" fashion, in pure Python. This means that it can give you more relevant information about your Python code.

_yosefk 10y ago
Actually KCachegrind displays files in an open format and is great for visualizing profiling info obtained from other tools.
deleted 10y ago
[deleted]
forgotpwtomain 10y ago
> It's doing what valgrind does but in a much "higher-level" fashion, in pure Python. Valgrind is essentially a VM, this uses cProfile - I wouldn't call that doing the 'same thing' at all.

vegabook 10y ago

Looks like an awesome piece of technology, but why, whenever I see headline xxxx followed by "(github.com)" do I get a slight feeling of despair - that feeling that this project has a high chance of being future abandonware. Is it that having a proper, dedicated, expensive-looking, website, away from github, is a signal of access to resources, willingness to do proper marketing and documentation, and therefore, by correlation, longevity?

rhinoceraptor 10y ago
If it's going to be abandonware, isn't it better to be on Github where it's easily accessible, rather than a random git server that could disappear?
notifier2050 10y ago
Like with almost any open source project nobody promises you anything. But it's not framework, you don't lose much by not using it and you don't have to rewrite your code in case it would be abandoned.
- vegabook 10y ago
  yes I realise that I'm looking a gift horse in the mouth, and I love git(hub), but I'm just pointing out that "(github.com)" might just be a net-negative marketing point for people looking for robust software.

fijal 10y ago

I would like to complain about name similarity to vmprof (my project)

daveguy 10y ago
You're kidding, right?

fijal 10y ago

TL;DR; it's a visualizer for cProfile (with all the cProfile problems) DISCLAIMER: I'm an author of cProfile competitor, vmprof

albasha 10y ago

Does it work with Django? If so, how?

JonathonW 10y ago

Neat-- wish I'd had this a few weeks ago when I was trying to track down some performance issues. (Ended up getting what I needed out of cProfile, which this also uses on the backend. Would've been nice to have the nice visualizations, though.)

DiThi 10y ago

Is there a similar tool of the code heatmap but for JS?

brndn 10y ago
Lots of info on that here: http://www.brendangregg.com/flamegraphs.html http://www.brendangregg.com/flamegraphs.html
- DiThi 10y ago
  I mean the code coverage tool, not the flame graphs. To e.g. know what paths of the code are never executed. Chrome and Firefox already show flame graphs in their profilers.

grinich 10y ago

We did something similar at Nylas, but we built it to run in production across our entire fleet: https://nylas.com/blog/performance/ https://nylas.com/blog/performance/

roberdam 10y ago

"RuntimeError: maximum recursion depth exceeded while calling a Python object " When installing.

oxplot 10y ago

I have just recently discovered chrome://tracing and wrote a quick python profiler [1] to output in its format. [1]: https://github.com/oxplot/crprof https://github.com/oxplot/crprof