10 ms·

Netflix FlameScope

389 points by brendangregg 9y ago

smaili 9y ago
Link to the repo -- https://github.com/Netflix/flamescope https://github.com/Netflix/flamescope From a quick glance, it's interesting to see both Flask and Node.JS be used within the same project repo. Is this becoming more commonplace?
- mspier 9y ago
  Not Node.JS specifically, but just npm for dependencies and task running to build the React client from source. If you're not looking to build the client from source, the distributed client bundle should suffice and only Python is required.
- wooly_bully 9y ago
  Even if you're using flask, there are plenty of reasons to have a package.json or other npm resources in your repo. Webpack is a good example.
ProAm 9y ago
I wish they wouldnt host this on Medium, the pop-ups and layout are maddening and it almost seems unprofessional for a giant company like Netflix to use Medium instead of its own blog platform on their website.
- soared 9y ago
  https://outline.com/hpckBV https://outline.com/hpckBV Although you lose the images so not that helpful in this situation, but usually good for medium.
- herrvogel- 9y ago
  Yes, Medium has terrible reading experience. I found this extension (https://github.com/thebaer/MMRA https://github.com/thebaer/MMRA) a while ago. You should try it.
  - oarsinsync 9y ago
    I keep reading comments like this, but I don’t ever see any particularly glaring issues when reading from my iPad. Is it a desktop / chrome thing that iOS / Safari is automatically mitigated against, or are my standards just so low that I don’t notice the issues?
    - freehunter 9y ago
      Maybe they don’t do it on mobile but on desktop they have a pop-up demanding you create an account to keep reading.
      - drusepth 9y ago
        Even when I open the link in an incognito window, I don't see this pop-up. Do you need to be on the page for X time to see it?
        ProAm 9y ago
        Pretty sure it looks for an existing cookie to pop it up.
        kbenson 9y ago
        Second request in a certain period, I think. I opened this link twice. The second time I got the pop-up, which starts out with something like "We've seen you here before...". I just tried a third time and it did not create a popup. As things go, I think they're actually handling that part fairly well.
      - charizardd 9y ago
        answer my question (in the other thread)...
        freehunter 9y ago
        Are you really stalking me to get me to respond to a thread I participated in two days ago? That’s odd.
- eggoa 9y ago
  Some employers will block netflix, which is one reason not to put their technical blogging on their own site.
wpietri 9y ago
Nice. I had some search cluster optimization problems that this would have been very handy for. I laughed when I got to "Ah, that’s Java garbage collection," as that turned out to be one of the big contributors to variance in response time.
wkz 9y ago
Wow, great visualization! Thanks for sharing the source. A minor suggestion: For the first part of the demo i was thrown of by the fact that each second as laid out bottom->top. Maybe moving the time labels below the plot would provide a visual cue for the direction.
- mspier 9y ago
  Can you open a issue/feature request on Github?
  - wkz 9y ago
    Sure thing: https://github.com/Netflix/flamescope/issues/1 https://github.com/Netflix/flamescope/issues/1
- lscotte 9y ago
  I think this is intentional, as the norm is for 0 on a Y-axis to be at the bottom and increase going up. I can understand why it's confusing, and perhaps could be a UI option to reverse it, but I do think that's the reason.
  - brendangregg 9y ago
    Yes, that's the reason! It means we get a t=0 origin for both the x- and y-axis. But yes, we can also add UI options to reverse it and other things.
  - wkz 9y ago
    Yes, I agree with it being that way. I just think my brain parses the origin as being the corner that is closest to the "0" label. Therefore, if the labels where below the plot, there would be no parse error :)
zawerf 9y ago
What's the purpose of visualizing it as grid? I thought it might be good for debugging periodic events (e.g., every 16ms for rendering at 60fps) but it seems like this is fixed to one sec for one column. For example in the flame graph in chrome's performance tab, you can select/slice/zoom/pan arbitrary time ranges. If you want the high level view you just zoom out.
- brendangregg 9y ago
  Supporting different y-axis ranges is something we can add, and I was already thinking of adding a 1ms range for visualizing low-level CPU cycle activities (which span nano and microseconds). I like the idea of 16ms for 60fps analysis, where every column is one frame. Could help game development. :)
  - saulrh 9y ago
    Actually, seeing the diagonal stripes in the examples [1], maybe it'd be useful to have a mode that lets you rescale the axes to line the stripes up? If you're using this to inspect patterns, it seems to me that the most useful time scale would not be a nice round human number, it'd be one that matches well with the patterns you're trying to inspect - one period, not one second. Way out in the distance, maybe even a variable-interval mode, where you can click-select blocks and each one is given its own column, so you can directly compare them without the dead space in the middle and starting from the same baseline? [1] Particularly striking: https://cdn-images-1.medium.com/max/2000/1*EypUnkPtayeKr9bJuvgNEg.png https://cdn-images-1.medium.com/max/2000/1*EypUnkPtayeKr9bJu...
    - Scaevolus 9y ago
      You could autoscale the y axis to whatever makes the columns have the most similarity (minimize squared differences).
      - simsla 9y ago
        Rather than lsq, you could use autocorrelation within rows to line them up.
unictek 9y ago
Can this be used with Node.js CPU Profiles?
- brendangregg 9y ago
  Yes, I've already used it on Node.js profiles. :) Get any instructions for doing CPU flame graphs with Linux perf (I've written several), and take the output stage of "perf script" and you can browse it in FlameScope.
- mspier 9y ago
  Depends on what you mean by Node.JS CPU profiles, if captured by `perf script`, then yes, otherwise, not yet.
IamNotAtWork 9y ago
What was the root cause of the problem that motivated this visualization tool?
- brendangregg 9y ago
  It's described in the blog post under "Origin", and the sample profile that we ship with FlameScope (examples/perf.stacks01) and which I used in the video is the first minute of the original problem. It was intermittent latency that was hard to explain using existing tools.
coding123 9y ago
Pretty awesome - I recall a long time ago a few companies selling software that would do similar (but different) things, so this can help us get closer.
OnlyRepliesToBS 9y ago
Make a better front page that isn't a full page damn ad.
cortesi 9y ago
This is lovely. An interesting thing to think about here is the layout of the sequential time-series data. This visualisation breaks the sequence out into columns, which means that squares that are close to each other in the sequence don't cluster visually. I've played with using space-filling curves for this to visualise binaries: https://binvis.io/ https://binvis.io/ You can play around on there with the scan layout (similar to the Netflix layout) and compare it to the Hilbert curve. You can click and drag to select regions in both (similar to the video for this visualisation). There are tradeoffs - the Hilbert layout is not intuitive, and you have to play around to select regions accurately. On the other hand, the clustering lets you visually pick hotspots in much greater detail, e.g.: https://binvis.io/#/view/examples/tcpview.exe.bin?colors=entropy https://binvis.io/#/view/examples/tcpview.exe.bin?colors=ent... I've found it useful to be able to toggle between both scanning and clustered layouts, and plan to explore some more interaction paradigms when I get time to turn back to the project.
- brendangregg 9y ago
  I did binary visualizations many years ago too: https://github.com/brendangregg/Dump2PNG/wiki https://github.com/brendangregg/Dump2PNG/wiki So with the subsecond-offset heat map, you get clustering from one column to the next for events that happen at the same time offset. Which is common with periodic tasks: see the example patterns in the post.
  - cortesi 9y ago
    It looks like we've traveled a similar path. The precursor to binvis is scurve, a (slow, dodgy) command-line tool to do basically what dump2png does: https://github.com/cortesi/scurve https://github.com/cortesi/scurve I have a blazing fast command-line version of binvis in Rust that I plan to release in the next month or so. I really need make some time to turn back to this.
    - brendangregg 9y ago
      binvis.io is pretty great! So I was more interested in core dumps, which gets tricky since they can be 10s of Gbytes in size. Probably the most useful thing I figured out was this palette type: x86 grayscale with some (9) color indicators: green = common english chars: 'e', 't', 'a' red = common x86 instructions: movl, call, testl blue = binary values: 0x01, 0x02, 0x03
kapitalx 9y ago
Longer video with more examples: https://www.youtube.com/watch?v=gRawd7CO-Q8 https://www.youtube.com/watch?v=gRawd7CO-Q8
throwaway84742 9y ago
I’ve done something like this at Microsoft in 2002. Except in my case it was a multi-user system and each row represented a logical user, or a range of users in a range partitioned space. You could vary granularity, click through to see what’s busted, zoom into user ranges, discover correlated failures, discover cases when something is failing for just one user consistently, correlate all of it to performance counters, etc, etc. No one gave a shit.
- mistermann 9y ago
  What did you use to build it?
  - throwaway84742 9y ago
    IIRC the standard MS stack at the time: C#, ASP.NET (which had just come out), MS SQL, IE, some in memory caching and bitmap rendering (canvas didn’t exist), AJAX. Pretty bleeding edge, but it all worked OK.
    - chrisweekly 9y ago
      Given when you were using that then-bleeding-edge stack, I wonder if you used XML (ie, the "X" in the then-acronym AJAX). later de-acronym-ized to "Ajax" w/ JSON replacing XML.
      - throwaway84742 9y ago
        Of course. JSON wasn’t popularized back then. But mostly it was about reloading parts of the page through DOM manipulation.
        chrisweekly 9y ago
        Yeah, I was there too. (Web dev for a living since 1998.) I remember sending HTML fragments over the wire too...
lma21 9y ago
Haven't followed up on this lately — is sampling java stacktraces with perf going well? what happened to the frame pointer issue? do we still have to use -XX:+PreserveFramePointer to get the full stack? Awesome tool nevertheless. Always a pleasure to go through Brendan's utilities regarding performance analysis!
- brendangregg 9y ago
  Yes, perf sampling of Java is still going well, via -XX:+PreserveFramePointer. Our one microservice that had high overhead with that option (up to 10%, which is rare) has improved their code, lowered the overhead, and now enabled this option by default. But there's also a newer way that can do Java stack sampling from perf without the frame pointer: https://github.com/jvm-profiling-tools/async-profiler https://github.com/jvm-profiling-tools/async-profiler We're not running it yet. I want to try it out. Note that the stacks with async-profiler are a bit broken -- Java methods become detached from the JVM -- but I'm hoping that's fixable (it should be with a JVM change, at least).
gregdunn 9y ago
Awesome! It'd be pretty sweet if you could send things sampled using Vector over to FlameScope. I know the flamegraphs PMDA is experimental but it's always worked fine for me :) (It'd also be pretty sweet if there was a fleshed out PMDA for bcc tools. And Vector supported them. And, and, and... I have a really long wishlist.)
- brendangregg 9y ago
  We'll get it done (although I'm not exactly sure who "we" will end up being yet!). I improved the flamegraph PMDA late last year, but how FlameScope works is really the direction Vector should go: being able to make an API call to get the flame graph stacks as JSON, and then render them in d3, along with buttons for transforms.
jsmeaton 9y ago
Does anyone have any good tools or processes for generating the kind of performance counters and information needed by this tool with a running python application? I'd love to be able to generate a flamegraph for a django app, for example, and visualise it with flame scope. https://mail.python.org/pipermail/python-dev/2014-November/136981.html https://mail.python.org/pipermail/python-dev/2014-November/1... discusses the problems with python and the `perf` tool. There may have been some patches applied recently to help, but it doesn't look like there was a full resolution.
- gnufx 9y ago
  Not a direct answer, but there's Python support of some sort available for at least TAU <https://www.cs.uoregon.edu/research/tau/home.php> https://www.cs.uoregon.edu/research/tau/home.php>, Extrae <https://tools.bsc.es/extrae> https://tools.bsc.es/extrae>, and Score-P <https://github.com/score-p/scorep_binding_python> https://github.com/score-p/scorep_binding_python>.
- akx 9y ago
  I wrote https://github.com/akx/abyss https://github.com/akx/abyss for Django once upon a time. I'm not sure it works these days, but you could give it a shot.
itissid 9y ago
> a) could not reliably capture a one minute flame graph covering the issue, as the onset of it kept fluctuating; b) capturing a two or three minute flamegraph didn’t help either, and the issue was “drowned” in the normal workload profile. Vadim asked me for help. One interesting thing here is also the statistical nature of this problem: i.e. there are spikes but their duration are not exact or known when to start. Here[1] is some reference work done on Neural Spike Trains that seems relevant. [1]http://www.stat.columbia.edu/~liam/teaching/neurostat-spr12/papers/brown-et-al/time-rescaling.pdf http://www.stat.columbia.edu/~liam/teaching/neurostat-spr12/...
sandGorgon 9y ago
Given the existence of tools like this or https://github.com/uber/pyflame https://github.com/uber/pyflame, I get really amazed at the monopoly of NewRelic. Nobody seems to be even attempting a start-up in this space, which is amazing given the amount of money that everyone coughs up for NewRelic.
hie2neiL 9y ago
All the native linux thread stack visualizers I have used so far lack a way to show single-threaded bottlenecks in multi-threaded applications. Does this solve it? In the java ecosystem this is solved with utilization timelines.
- gnufx 9y ago
  I don't know which of the normal parallel performance tools support Java, but at least TAU <https://www.cs.uoregon.edu/research/tau/docs.php> https://www.cs.uoregon.edu/research/tau/docs.php> does; not that I've worked on Java, which should preferably be kept well away from compute nodes.
  - hie2neiL 9y ago
    Oh, my question wasn't about java. Quite the opposite. I'm missing the perf equivalent of some java tools I am accustomed to.
    - gnufx 9y ago
      I don't know what "perf equivalent" means. Parallel performance tools usually use PAPI, which provides similar data, possibly a superset of perf data on Linux. Score-P has an actual perf plugin. http://www.vi-hps.org/training/material/ http://www.vi-hps.org/training/material/ is one source of material on such things, though biased towards distributed (MPI) systems and OpenMP threading. [Sorry I hadn't realized <> wasn't work as URL delimiters here, as I'm used to elsewhere since the RFC <URL:...> convention got ignored.]
  - gregdunn 9y ago
    Not directly related, but all of your posts on this subject contain broken links - please add spaces between your link and your delimiter, if you are going to use a delimiter, as doing <link> makes the > part of the link, requiring manual removal.
petters 9y ago
I know I am missing something, but passing -F 49 to perf will tell the kernel to adjust the sampling interval dynamically so approximately 49 events/s are recorded, right? But then how does it make sense to plot the number of events per second in the overview? Or what exactly is being plotted in the overview? Edit: Perhaps it is simply that idle events are discarded and multiple CPUs allow more than one event in the period?
gnufx 9y ago
What's the advantage of this sort of thing over the trace and profile analysis tools that have been developed at least for HPC performance engineering over many years? (E.g. the BSC tools <https://tools.bsc.es/> https://tools.bsc.es/> amongst several others.) In particular, why are flame graphs preferred to the normal inclusive/exclusive views of profiles which seem more generally useful to me?

389 points by brendangregg 9y ago

Link to the repo -- https://github.com/Netflix/flamescope https://github.com/Netflix/flamescope From a quick glance, it's interesting to see both Flask and Node.JS be used within the same project repo. Is this becoming more commonplace?

mspier 9y ago
Not Node.JS specifically, but just npm for dependencies and task running to build the React client from source. If you're not looking to build the client from source, the distributed client bundle should suffice and only Python is required.
wooly_bully 9y ago
Even if you're using flask, there are plenty of reasons to have a package.json or other npm resources in your repo. Webpack is a good example.

ProAm 9y ago

I wish they wouldnt host this on Medium, the pop-ups and layout are maddening and it almost seems unprofessional for a giant company like Netflix to use Medium instead of its own blog platform on their website.

soared 9y ago
https://outline.com/hpckBV https://outline.com/hpckBV Although you lose the images so not that helpful in this situation, but usually good for medium.
herrvogel- 9y ago
Yes, Medium has terrible reading experience. I found this extension (https://github.com/thebaer/MMRA https://github.com/thebaer/MMRA) a while ago. You should try it.
- oarsinsync 9y ago
  I keep reading comments like this, but I don’t ever see any particularly glaring issues when reading from my iPad. Is it a desktop / chrome thing that iOS / Safari is automatically mitigated against, or are my standards just so low that I don’t notice the issues?
  - freehunter 9y ago
    Maybe they don’t do it on mobile but on desktop they have a pop-up demanding you create an account to keep reading.
    - drusepth 9y ago
      Even when I open the link in an incognito window, I don't see this pop-up. Do you need to be on the page for X time to see it?
      - ProAm 9y ago
        Pretty sure it looks for an existing cookie to pop it up.
      - kbenson 9y ago
        Second request in a certain period, I think. I opened this link twice. The second time I got the pop-up, which starts out with something like "We've seen you here before...". I just tried a third time and it did not create a popup. As things go, I think they're actually handling that part fairly well.
    - charizardd 9y ago
      answer my question (in the other thread)...
      - freehunter 9y ago
        Are you really stalking me to get me to respond to a thread I participated in two days ago? That’s odd.
eggoa 9y ago
Some employers will block netflix, which is one reason not to put their technical blogging on their own site.

wpietri 9y ago

Nice. I had some search cluster optimization problems that this would have been very handy for. I laughed when I got to "Ah, that’s Java garbage collection," as that turned out to be one of the big contributors to variance in response time.

wkz 9y ago

Wow, great visualization! Thanks for sharing the source. A minor suggestion: For the first part of the demo i was thrown of by the fact that each second as laid out bottom->top. Maybe moving the time labels below the plot would provide a visual cue for the direction.

mspier 9y ago
Can you open a issue/feature request on Github?
- wkz 9y ago
  Sure thing: https://github.com/Netflix/flamescope/issues/1 https://github.com/Netflix/flamescope/issues/1
lscotte 9y ago
I think this is intentional, as the norm is for 0 on a Y-axis to be at the bottom and increase going up. I can understand why it's confusing, and perhaps could be a UI option to reverse it, but I do think that's the reason.
- brendangregg 9y ago
  Yes, that's the reason! It means we get a t=0 origin for both the x- and y-axis. But yes, we can also add UI options to reverse it and other things.
- wkz 9y ago
  Yes, I agree with it being that way. I just think my brain parses the origin as being the corner that is closest to the "0" label. Therefore, if the labels where below the plot, there would be no parse error :)

zawerf 9y ago

What's the purpose of visualizing it as grid? I thought it might be good for debugging periodic events (e.g., every 16ms for rendering at 60fps) but it seems like this is fixed to one sec for one column. For example in the flame graph in chrome's performance tab, you can select/slice/zoom/pan arbitrary time ranges. If you want the high level view you just zoom out.

brendangregg 9y ago
Supporting different y-axis ranges is something we can add, and I was already thinking of adding a 1ms range for visualizing low-level CPU cycle activities (which span nano and microseconds). I like the idea of 16ms for 60fps analysis, where every column is one frame. Could help game development. :)
- saulrh 9y ago
  Actually, seeing the diagonal stripes in the examples [1], maybe it'd be useful to have a mode that lets you rescale the axes to line the stripes up? If you're using this to inspect patterns, it seems to me that the most useful time scale would not be a nice round human number, it'd be one that matches well with the patterns you're trying to inspect - one period, not one second. Way out in the distance, maybe even a variable-interval mode, where you can click-select blocks and each one is given its own column, so you can directly compare them without the dead space in the middle and starting from the same baseline? [1] Particularly striking: https://cdn-images-1.medium.com/max/2000/1*EypUnkPtayeKr9bJuvgNEg.png https://cdn-images-1.medium.com/max/2000/1*EypUnkPtayeKr9bJu...
  - Scaevolus 9y ago
    You could autoscale the y axis to whatever makes the columns have the most similarity (minimize squared differences).
    - simsla 9y ago
      Rather than lsq, you could use autocorrelation within rows to line them up.

unictek 9y ago

Can this be used with Node.js CPU Profiles?

brendangregg 9y ago
Yes, I've already used it on Node.js profiles. :) Get any instructions for doing CPU flame graphs with Linux perf (I've written several), and take the output stage of "perf script" and you can browse it in FlameScope.
mspier 9y ago
Depends on what you mean by Node.JS CPU profiles, if captured by `perf script`, then yes, otherwise, not yet.

IamNotAtWork 9y ago

What was the root cause of the problem that motivated this visualization tool?

brendangregg 9y ago
It's described in the blog post under "Origin", and the sample profile that we ship with FlameScope (examples/perf.stacks01) and which I used in the video is the first minute of the original problem. It was intermittent latency that was hard to explain using existing tools.

coding123 9y ago

Pretty awesome - I recall a long time ago a few companies selling software that would do similar (but different) things, so this can help us get closer.

OnlyRepliesToBS 9y ago

Make a better front page that isn't a full page damn ad.

cortesi 9y ago

This is lovely. An interesting thing to think about here is the layout of the sequential time-series data. This visualisation breaks the sequence out into columns, which means that squares that are close to each other in the sequence don't cluster visually. I've played with using space-filling curves for this to visualise binaries: https://binvis.io/ https://binvis.io/ You can play around on there with the scan layout (similar to the Netflix layout) and compare it to the Hilbert curve. You can click and drag to select regions in both (similar to the video for this visualisation). There are tradeoffs - the Hilbert layout is not intuitive, and you have to play around to select regions accurately. On the other hand, the clustering lets you visually pick hotspots in much greater detail, e.g.: https://binvis.io/#/view/examples/tcpview.exe.bin?colors=entropy https://binvis.io/#/view/examples/tcpview.exe.bin?colors=ent... I've found it useful to be able to toggle between both scanning and clustered layouts, and plan to explore some more interaction paradigms when I get time to turn back to the project.

brendangregg 9y ago
I did binary visualizations many years ago too: https://github.com/brendangregg/Dump2PNG/wiki https://github.com/brendangregg/Dump2PNG/wiki So with the subsecond-offset heat map, you get clustering from one column to the next for events that happen at the same time offset. Which is common with periodic tasks: see the example patterns in the post.
- cortesi 9y ago
  It looks like we've traveled a similar path. The precursor to binvis is scurve, a (slow, dodgy) command-line tool to do basically what dump2png does: https://github.com/cortesi/scurve https://github.com/cortesi/scurve I have a blazing fast command-line version of binvis in Rust that I plan to release in the next month or so. I really need make some time to turn back to this.
  - brendangregg 9y ago
    binvis.io is pretty great! So I was more interested in core dumps, which gets tricky since they can be 10s of Gbytes in size. Probably the most useful thing I figured out was this palette type: x86 grayscale with some (9) color indicators: green = common english chars: 'e', 't', 'a' red = common x86 instructions: movl, call, testl blue = binary values: 0x01, 0x02, 0x03

kapitalx 9y ago

Longer video with more examples: https://www.youtube.com/watch?v=gRawd7CO-Q8 https://www.youtube.com/watch?v=gRawd7CO-Q8

throwaway84742 9y ago

I’ve done something like this at Microsoft in 2002. Except in my case it was a multi-user system and each row represented a logical user, or a range of users in a range partitioned space. You could vary granularity, click through to see what’s busted, zoom into user ranges, discover correlated failures, discover cases when something is failing for just one user consistently, correlate all of it to performance counters, etc, etc. No one gave a shit.

mistermann 9y ago
What did you use to build it?
- throwaway84742 9y ago
  IIRC the standard MS stack at the time: C#, ASP.NET (which had just come out), MS SQL, IE, some in memory caching and bitmap rendering (canvas didn’t exist), AJAX. Pretty bleeding edge, but it all worked OK.
  - chrisweekly 9y ago
    Given when you were using that then-bleeding-edge stack, I wonder if you used XML (ie, the "X" in the then-acronym AJAX). later de-acronym-ized to "Ajax" w/ JSON replacing XML.
    - throwaway84742 9y ago
      Of course. JSON wasn’t popularized back then. But mostly it was about reloading parts of the page through DOM manipulation.
      - chrisweekly 9y ago
        Yeah, I was there too. (Web dev for a living since 1998.) I remember sending HTML fragments over the wire too...

lma21 9y ago

Haven't followed up on this lately — is sampling java stacktraces with perf going well? what happened to the frame pointer issue? do we still have to use -XX:+PreserveFramePointer to get the full stack? Awesome tool nevertheless. Always a pleasure to go through Brendan's utilities regarding performance analysis!

brendangregg 9y ago
Yes, perf sampling of Java is still going well, via -XX:+PreserveFramePointer. Our one microservice that had high overhead with that option (up to 10%, which is rare) has improved their code, lowered the overhead, and now enabled this option by default. But there's also a newer way that can do Java stack sampling from perf without the frame pointer: https://github.com/jvm-profiling-tools/async-profiler https://github.com/jvm-profiling-tools/async-profiler We're not running it yet. I want to try it out. Note that the stacks with async-profiler are a bit broken -- Java methods become detached from the JVM -- but I'm hoping that's fixable (it should be with a JVM change, at least).

gregdunn 9y ago

Awesome! It'd be pretty sweet if you could send things sampled using Vector over to FlameScope. I know the flamegraphs PMDA is experimental but it's always worked fine for me :) (It'd also be pretty sweet if there was a fleshed out PMDA for bcc tools. And Vector supported them. And, and, and... I have a really long wishlist.)

brendangregg 9y ago
We'll get it done (although I'm not exactly sure who "we" will end up being yet!). I improved the flamegraph PMDA late last year, but how FlameScope works is really the direction Vector should go: being able to make an API call to get the flame graph stacks as JSON, and then render them in d3, along with buttons for transforms.

jsmeaton 9y ago

Does anyone have any good tools or processes for generating the kind of performance counters and information needed by this tool with a running python application? I'd love to be able to generate a flamegraph for a django app, for example, and visualise it with flame scope. https://mail.python.org/pipermail/python-dev/2014-November/136981.html https://mail.python.org/pipermail/python-dev/2014-November/1... discusses the problems with python and the `perf` tool. There may have been some patches applied recently to help, but it doesn't look like there was a full resolution.

gnufx 9y ago
Not a direct answer, but there's Python support of some sort available for at least TAU <https://www.cs.uoregon.edu/research/tau/home.php> https://www.cs.uoregon.edu/research/tau/home.php>, Extrae <https://tools.bsc.es/extrae> https://tools.bsc.es/extrae>, and Score-P <https://github.com/score-p/scorep_binding_python> https://github.com/score-p/scorep_binding_python>.
akx 9y ago
I wrote https://github.com/akx/abyss https://github.com/akx/abyss for Django once upon a time. I'm not sure it works these days, but you could give it a shot.

itissid 9y ago

> a) could not reliably capture a one minute flame graph covering the issue, as the onset of it kept fluctuating; b) capturing a two or three minute flamegraph didn’t help either, and the issue was “drowned” in the normal workload profile. Vadim asked me for help. One interesting thing here is also the statistical nature of this problem: i.e. there are spikes but their duration are not exact or known when to start. Here[1] is some reference work done on Neural Spike Trains that seems relevant. [1]http://www.stat.columbia.edu/~liam/teaching/neurostat-spr12/papers/brown-et-al/time-rescaling.pdf http://www.stat.columbia.edu/~liam/teaching/neurostat-spr12/...

sandGorgon 9y ago

Given the existence of tools like this or https://github.com/uber/pyflame https://github.com/uber/pyflame, I get really amazed at the monopoly of NewRelic. Nobody seems to be even attempting a start-up in this space, which is amazing given the amount of money that everyone coughs up for NewRelic.

hie2neiL 9y ago

All the native linux thread stack visualizers I have used so far lack a way to show single-threaded bottlenecks in multi-threaded applications. Does this solve it? In the java ecosystem this is solved with utilization timelines.

gnufx 9y ago
I don't know which of the normal parallel performance tools support Java, but at least TAU <https://www.cs.uoregon.edu/research/tau/docs.php> https://www.cs.uoregon.edu/research/tau/docs.php> does; not that I've worked on Java, which should preferably be kept well away from compute nodes.
- hie2neiL 9y ago
  Oh, my question wasn't about java. Quite the opposite. I'm missing the perf equivalent of some java tools I am accustomed to.
  - gnufx 9y ago
    I don't know what "perf equivalent" means. Parallel performance tools usually use PAPI, which provides similar data, possibly a superset of perf data on Linux. Score-P has an actual perf plugin. http://www.vi-hps.org/training/material/ http://www.vi-hps.org/training/material/ is one source of material on such things, though biased towards distributed (MPI) systems and OpenMP threading. [Sorry I hadn't realized <> wasn't work as URL delimiters here, as I'm used to elsewhere since the RFC <URL:...> convention got ignored.]
- gregdunn 9y ago
  Not directly related, but all of your posts on this subject contain broken links - please add spaces between your link and your delimiter, if you are going to use a delimiter, as doing <link> makes the > part of the link, requiring manual removal.

petters 9y ago

I know I am missing something, but passing -F 49 to perf will tell the kernel to adjust the sampling interval dynamically so approximately 49 events/s are recorded, right? But then how does it make sense to plot the number of events per second in the overview? Or what exactly is being plotted in the overview? Edit: Perhaps it is simply that idle events are discarded and multiple CPUs allow more than one event in the period?

gnufx 9y ago

What's the advantage of this sort of thing over the trace and profile analysis tools that have been developed at least for HPC performance engineering over many years? (E.g. the BSC tools <https://tools.bsc.es/> https://tools.bsc.es/> amongst several others.) In particular, why are flame graphs preferred to the normal inclusive/exclusive views of profiles which seem more generally useful to me?