3 ms·

I should add that Linux has two tracing front ends built in: ftrace and perf_events. And perf_events is building BPF support (see lkml). bcc is another front en

I should add that Linux has two tracing front ends built in: ftrace and perf_events. And perf_events is building BPF support (see lkml). bcc is another front end for BPF. If this is confusing, I should categorize them like this: back end technologies: ftrace, kprobes, uprobes, tracepoints, BPF front ends: ftrace, perf_events, bcc ftrace is hard to categorize: it is a collection of both back end and front end capabilities (although the front end is /sys). Also, it's being renamed to just "trace".

scarmig 11y ago
Thanks Brendan! A more navel-gazey question I believe you're well positioned to answer: For a long time, there seemed to be a conventional wisdom that Linux tracing was mediocre, because it doesn't have DTrace. Has Linux approached or passed the "caught up" point?
- brendangregg 11y ago
  It hasn't caught up yet, but it's getting pretty close. Although I'd think about it in terms of three dimensions: - technical capability: we should pass it this year. We'll be technical capable of solving pretty much the same issues and then some. - ease of use for script developers: bcc is currently ~10x more verbose to program. ply is much easier, but that project is in its early stages. This won't matter that much for Netflix, since much of the time staff will use this from a GUI (eg, Vector). It also won't matter that much for people who use the many provided tools, like opensnoop, tcpconnect, ext4slower, etc. It matters most for whoever codes the GUI and those tools (eg, me). - marketing, awareness, brand: Sun marketing, sales, trainers, and evangelists, did an awesome job - everyone knows about DTrace, even if they've never used it. In comparison, bcc/BPF (and many other Linux kernel technologies, like ftrace) have virtually zero marketing budget or professional marketing staff promoting them. Awareness will take the longest to catch up. I also summarized the current state of bcc/BPF in http://www.brendangregg.com/blog/2016-03-28/linux-bpf-bcc-road-ahead-2016.html http://www.brendangregg.com/blog/2016-03-28/linux-bpf-bcc-ro...
  - brendangregg 11y ago
    ... I should add, a tipping point for awareness may be when a commercial performance monitoring company adds BPF support (and latency heat maps, etc), and then promotes this feature. There's a lot of performance monitoring companies out there, many are well funded, and have professional marketing/sales departments.
heybrendan 11y ago
New account, just to ask this. I understand how much effort is involved in writing / publishing. Are you planning a Second edition of your "Systems Performance, Enterprise and the Cloud" text, especially in light of the pace of development in the Linux performance / profiling space? Thank you for all your countless contributions.
- brendangregg 11y ago
  Yes, I'll be writing more books, more details later; as for a 2nd edition, well, the 1st edition is probably more useful in the meantime than you might think. Lets say you install bcc/BPF tomorrow. Ok, now what? The hardest part is to know what to do with these superpowers. Apart from the provided scripts in bcc, the sysperf book has other DTrace examples that you might need, and can then port over. The hardest part of DTrace was knowing what to do with it, so all the prior DTrace documentation will be good material to browse and be inspired from, including my DTrace book. Many of my bcc/tools originated as versions I wrote using DTrace.

scarmig 11y ago

Thanks Brendan! A more navel-gazey question I believe you're well positioned to answer: For a long time, there seemed to be a conventional wisdom that Linux tracing was mediocre, because it doesn't have DTrace. Has Linux approached or passed the "caught up" point?

brendangregg 11y ago
It hasn't caught up yet, but it's getting pretty close. Although I'd think about it in terms of three dimensions: - technical capability: we should pass it this year. We'll be technical capable of solving pretty much the same issues and then some. - ease of use for script developers: bcc is currently ~10x more verbose to program. ply is much easier, but that project is in its early stages. This won't matter that much for Netflix, since much of the time staff will use this from a GUI (eg, Vector). It also won't matter that much for people who use the many provided tools, like opensnoop, tcpconnect, ext4slower, etc. It matters most for whoever codes the GUI and those tools (eg, me). - marketing, awareness, brand: Sun marketing, sales, trainers, and evangelists, did an awesome job - everyone knows about DTrace, even if they've never used it. In comparison, bcc/BPF (and many other Linux kernel technologies, like ftrace) have virtually zero marketing budget or professional marketing staff promoting them. Awareness will take the longest to catch up. I also summarized the current state of bcc/BPF in http://www.brendangregg.com/blog/2016-03-28/linux-bpf-bcc-road-ahead-2016.html http://www.brendangregg.com/blog/2016-03-28/linux-bpf-bcc-ro...
- brendangregg 11y ago
  ... I should add, a tipping point for awareness may be when a commercial performance monitoring company adds BPF support (and latency heat maps, etc), and then promotes this feature. There's a lot of performance monitoring companies out there, many are well funded, and have professional marketing/sales departments.

heybrendan 11y ago

New account, just to ask this. I understand how much effort is involved in writing / publishing. Are you planning a Second edition of your "Systems Performance, Enterprise and the Cloud" text, especially in light of the pace of development in the Linux performance / profiling space? Thank you for all your countless contributions.

brendangregg 11y ago
Yes, I'll be writing more books, more details later; as for a 2nd edition, well, the 1st edition is probably more useful in the meantime than you might think. Lets say you install bcc/BPF tomorrow. Ok, now what? The hardest part is to know what to do with these superpowers. Apart from the provided scripts in bcc, the sysperf book has other DTrace examples that you might need, and can then port over. The hardest part of DTrace was knowing what to do with it, so all the prior DTrace documentation will be good material to browse and be inspired from, including my DTrace book. Many of my bcc/tools originated as versions I wrote using DTrace.