3 ms·

Looks like another faster horse. A pretty GUI on /proc is not the most burning issue to solve in Linux performance monitoring. I wish anyone making these tools

Looks like another faster horse. A pretty GUI on /proc is not the most burning issue to solve in Linux performance monitoring. I wish anyone making these tools would spend 30 minutes watching my Monitorama talk about instance monitoring requirements at Netflix: http://www.brendangregg.com/blog/2015-06-23/netflix-instance-analysis-requirements.html http://www.brendangregg.com/blog/2015-06-23/netflix-instance... . I still hate gauges. Where is the PMC support? At Facebook a few days ago, they said their number one issue was memory bandwidth. Try analyzing that without PMCs. You can't. And that's their number one issue. And it shouldn't be a surprise that you need PMC access to have a decent Linux monitoring/analysis tool. If that's a surprise to you, you're creating a tool without actual performance expertise. Should front BPF tracing as well... Maybe it will in the future and I can check again.

X86BSD 11y ago
They are more than welcome to use FreeBSD's PMC support to roll their own implementation. But I predict they won't until another decade of beating their heads against broken tools and non engineered solutions, light through yonder window will finally break.
lazyant 11y ago
Unrelated: In the middle of your Performance book, wish I had studied it several years ago. For this tool I wonder what the observer's effect is.
zokier 11y ago
Just curious; how closely you are involved in Vector development? Do you think Vector tackles the issues you mention?
- deleted 11y ago
  [deleted]
- brendangregg 11y ago
  Yes, I'm involved, and no, I don't think the current public release of Vector tackles many of the issues yet. But we've been working on them and will have it released when we can. Items include: - PMCs: Vector's backend is pcp, which has a Linux perf_events pmda for PMC support. There's no PMC counters by default in Vector since we're using it in an environment without PMC access. But I've been heavily working in this area. More later on. - Flame graph support: we already have it and use it. Ongoing work includes new flame graph types, and a rewritten flame graph implementation in d3. Need to get it all published. - Heat map support: just solved an issue with them; again, an area where we have ad hoc tools that work and bring value, but haven't wrapped it all in Vector yet. - BPF support: I've been prototyping many new metrics that we want in Vector, https://github.com/iovisor/bcc#tools https://github.com/iovisor/bcc#tools. There'll be increased demand to get Vector accessing these in a couple of months or so, when we have newer kernels in production that have BPF.
peterwwillis 11y ago
First off, I think this is actually designed mainly for embedded applications. It's the only reason I can think of that they'd make a single self-contained monitor that only works for one host. Second, it's great that you want a car instead of a faster horse, but I want 10 horses. Why? - 1 horse trained for 1 job easier & more reliable than 1 horse/car trained for 10 - 10 horses more flexible than 1 horse or 1 car - if a horse isn't doing its job, shoot it and replace it - don't need to go to horse school to learn how to use or maintain one - don't need to hire an engineer to teach horse new trick - most problems can be solved with a horse I get that with some applications, having a couple extra minutes saved by your monitoring system can mean millions of dollars saved. For those cases, there will always be custom-tailored solutions because there is value there. For 99% of the rest of the time, you get more value from simple tools that can be combined to do their jobs well. If instead of focusing on the transportation we were focusing on the road, we could build a foundation for horses and cars to coexist. I'd love to see more protocols and specifications for each kind of monitor (other than syslog, RRD & SNMP) so they could simply be made according to spec and we could mix and match as we wished.
stelfer 11y ago
Yep if you aren't looking at eBPF now you probably aren't doing it right. And BTW thanks for that Brendan.

X86BSD 11y ago

They are more than welcome to use FreeBSD's PMC support to roll their own implementation. But I predict they won't until another decade of beating their heads against broken tools and non engineered solutions, light through yonder window will finally break.

lazyant 11y ago

Unrelated: In the middle of your Performance book, wish I had studied it several years ago. For this tool I wonder what the observer's effect is.

zokier 11y ago

Just curious; how closely you are involved in Vector development? Do you think Vector tackles the issues you mention?

deleted 11y ago
[deleted]
brendangregg 11y ago
Yes, I'm involved, and no, I don't think the current public release of Vector tackles many of the issues yet. But we've been working on them and will have it released when we can. Items include: - PMCs: Vector's backend is pcp, which has a Linux perf_events pmda for PMC support. There's no PMC counters by default in Vector since we're using it in an environment without PMC access. But I've been heavily working in this area. More later on. - Flame graph support: we already have it and use it. Ongoing work includes new flame graph types, and a rewritten flame graph implementation in d3. Need to get it all published. - Heat map support: just solved an issue with them; again, an area where we have ad hoc tools that work and bring value, but haven't wrapped it all in Vector yet. - BPF support: I've been prototyping many new metrics that we want in Vector, https://github.com/iovisor/bcc#tools https://github.com/iovisor/bcc#tools. There'll be increased demand to get Vector accessing these in a couple of months or so, when we have newer kernels in production that have BPF.

peterwwillis 11y ago

First off, I think this is actually designed mainly for embedded applications. It's the only reason I can think of that they'd make a single self-contained monitor that only works for one host. Second, it's great that you want a car instead of a faster horse, but I want 10 horses. Why? - 1 horse trained for 1 job easier & more reliable than 1 horse/car trained for 10 - 10 horses more flexible than 1 horse or 1 car - if a horse isn't doing its job, shoot it and replace it - don't need to go to horse school to learn how to use or maintain one - don't need to hire an engineer to teach horse new trick - most problems can be solved with a horse I get that with some applications, having a couple extra minutes saved by your monitoring system can mean millions of dollars saved. For those cases, there will always be custom-tailored solutions because there is value there. For 99% of the rest of the time, you get more value from simple tools that can be combined to do their jobs well. If instead of focusing on the transportation we were focusing on the road, we could build a foundation for horses and cars to coexist. I'd love to see more protocols and specifications for each kind of monitor (other than syslog, RRD & SNMP) so they could simply be made according to spec and we could mix and match as we wished.

stelfer 11y ago

Yep if you aren't looking at eBPF now you probably aren't doing it right. And BTW thanks for that Brendan.