12 ms·
DTrace for Linux 2016
- lallysingh 10y agoSo we're not getting DTrace proper, it seems. Instead something else will stem up from the various linux tracing systems. Maybe this BPF-based one. It's a shame. One of the nice things about dtrace was that there was a book on it. Good, in-depth documentation on performance tools is hard to find.
- brendangregg 10y agoThanks, I wrote the DTrace book with Jim Mauro, and there will be a BPF tracing book as well. BTW, I wouldn't say "maybe" regarding BPF, as it's integrated in the Linux kernel (unlike most of the other tracers, which are add-ons). Sooner or later everyone who runs Linux is getting it.
- BrianJ123 10y agoI've been waiting for this for since Solaris. Thanks!
- cthalupa 10y agoI think I bother you about a new tracing book for Linux every time one of your articles is posted, so I'll give my obligatory: We want a new Brendan Gregg tracing book! ;) Things have been moving so fast it's probably a good thing you didn't. It sounds like 4.9 will slow a lot of that down to a more manageable pace for writing a book, though.
- AceJohnny2 10y agoI've only recently tried out DTrace on OS X, and I'll admit to being kinda floored at what it can do. To think I used to be satisfied with strace on Linux! Seeing the tracing capabilites of Linux expand is exciting indeed. Edit: the couple of tutorials that finally unlocked DTrace (on OS X) for me are: https://www.objc.io/issues/19-debugging/dtrace/ https://www.objc.io/issues/19-debugging/dtrace/ https://www.bignerdranch.com/blog/hooked-on-dtrace-part-1/ https://www.bignerdranch.com/blog/hooked-on-dtrace-part-1/
- tkinom 10y agoAgree, DTrace on OS X is supper powerful. I once try to debug the open source libusb app in Mac OS, with DTrace I can trace the App, Kernel USB API call, libusb internal thread in user space, etc. Much better visibility to system activities compare to simple strace. Absolutely love the power of what it can do. BTW, Can a DTrace script to use to monitor a system with potential "Dirty COW" type privilege escalation issue?
- helper 10y agoThe most challenging thing for us is running a new enough kernel to get these features. While upgrading to a newer kernel isn't particularly hard, small companies don't have a lot of engineering resources to run kernels that aren't maintained by their distro of choice (usually on the LTS release). The good thing is this is solved simply by waiting long enough. The bad thing is most developers can't just pick this up today without a bunch of extra effort. If you are looking for something you can use with old kernels you should definitely checkout Brendan's perf-tools repo[1]. It takes advantage of older kernel features and works with things as old as ubuntu 12.04. *Edit: Fixed Brendan's name [1]: https://github.com/brendangregg/perf-tools https://github.com/brendangregg/perf-tools
- technofiend 10y agoThis is the same problem shared by RedHat customers, although RH is great about backporting features to older kernels, I'm not sure they'll be able to move this to 3.x from 4.9. The price we pay for stability.
- 4ad 10y agoYou mean the price paid for stable kernel ABIs, so proprietary drivers can use them? The Linux kernel is the most safe operating system component to upgrade, mostly because the Linux kernel people care deeply about compatibility and not breaking user space. While it is still not free to upgrade, the cost is minimal compared to the cost (in real money, risk, security) of backporting major components to older kernels, like Red Hat is doing. Red Hat is maintaining old kernels for having a stable kernel ABI (among some other reasons), not for general "stability". Personally I run only CentOS, but always with a recent kernel, albeit usually a LTS one. Mostly to get exactly these type of features described in this article.
- technofiend 10y agoWell they show the same caution and care around curating their repositories and releasing security fixes, so in my opinion (you may disagree) they try to ensure stability beyond the kernel's ABI.
- wyldfire 10y agoCongrats, this is good news. > On Linux, some out-of-tree tracers like SystemTap could serve these needs, but brought their own challenges. I was pretty happy with stap, it had a really rich feature set. > DTrace has its own concise language, D, similar to awk, whereas bcc uses existing languages (C and Python or lua) with libraries. I think we need more creative names for languages. The short and simple ones like "go" and "D" keep on having collisions. :) >BPF adds ... uprobes uprobes + all the other stuff is really killer, I like the idea of watching for stuff like "my app has crossed this threshold and then this system condition occurs". At least when I tried it a couple years ago with stap my kernel wasn't built with uprobes support and I wasn't inclined to rebuild it. Hopefully it becomes (or has become) more mainstream.
- brendangregg 10y ago> I was pretty happy with stap, it had a really rich feature set. So are other companies. I mentioned it in the post, as in a way this hurt BPF development, as companies that normally would have contributed resources said they were satisfied with stap. Exciting times might be ahead for stap, if it continues its BPF backend. As for naming, yes, we need better names. Maybe the bcc/Python/BPF combination can be named something?
- 4ad 10y agoLinux is not my favorite operating system, but it seems like we're stuck with it. I'm very happy for all these improvements. Once you got used to a system with a quality and functional tracer, Linux was hard to get back to. But Linux tracing is getting better and better now. I am very satisfied.
- Annatar 10y agoLinux is not my favorite operating system, but it seems like we're stuck with it. It only seems that way. We're never stuck with something as long as we don't accept it. One other factor is at play which works against Linux, and that is that people in IT like shiny new things, and therefore something else always comes along. Hopefully this time around, that something else will be the old new thing (learning from the past, and re-discovery). One way or the other, the clock is ticking on Linux, and one of these days, it won't be as popular any more, because something else will be the new-new thing. It's the nature of this industry: change is the only constant. You don't have to accept anything. Don't bow to peer pressure.
- qwertyuiop924 10y agoWill there every be way to write probes/tracing scripts without dropping into C? I don't mind C in general, but I don't want to have to dig out the documentation for the eBPF C library and start writing hundreds of lines of C every time I want to run a trace. DTrace made this really nice, because you would write your tracing scripts in a high-level, awk-like language, which is the sort of thing well-suited to the purpose.
- brendangregg 10y agoYes, see the section "A higher-level language", which mentions at least two projects: SystemTap+BPF and ply. Think of the current bcc/Python/C interface as a lightweight skin that was necessary during BPF development to kick the tires on various features, prototype tools, see what else needed to be done, etc. It may be good enough to stay around, as lots of tools have been written for it that will get used and be valuable. But there's room for higher-level languages too. If Sasha keeps developing his "trace" tool (and its summary counterpart, argdist), that may serve many such custom needs (as another option). See the various examples: https://github.com/iovisor/bcc/blob/master/tools/trace_example.txt https://github.com/iovisor/bcc/blob/master/tools/trace_examp... , like: # trace 'sys_read (arg3 > 20000) "read %d bytes", arg3' TIME PID COMM FUNC - 05:18:23 4490 dd sys_read read 1048576 bytes 05:18:23 4490 dd sys_read read 1048576 bytes 05:18:23 4490 dd sys_read read 1048576 bytes
- caf 10y agoHow feasible would it be to compile the dtrace language itself to an eBPF backend?
- qwertyuiop924 10y agoHuh. Is there a formal definition anywhere? If we have some kind of spec (even if it's not formal), it might be possible, since they are roughly equivalent, AFAIK. However, since I haven't worked in-depth in either, I'm unsure what work would be involved: it's possible that only a subset would compile. Anyways, this is probably a good goal to shoot for. Dtrace is the system tracer on pretty much all of the other big unixes, so it's a good idea to support as much of its language as possible. Plus, there are a lot of scripts already written in DTrace's language: having access to them would be invaluable.
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- honkhonkpants 10y agoSo how does this relate to uprobes? I've been looking into that lately because I want frequency counts (or coverage analysis) of user space programs but without the nop-sled overhead of xray. Does dtrace supplement or replace uprobes? Or am I really just confused?
- cthalupa 10y agoDTrace is a Solaris (and BSD/OSX) tracing tool that never quite made it to Linux (There are some attempted ports, but none of them really caught on). BPF (and adding in frontends like BCC) give you the same sort of functionality in Linux. BPF can take advantage of uprobes and instrument around them, but it interacts with them, and does not replace them
- jdesfossez 10y agoIt would be worthwhile to clarify the term "tracing" to distinguish between live aggregation and post-processing approaches. The general confusion around the "tracing" terminology seems to imply a competition between these two, while they should rather be seen as complementary. DTrace, SystemTap and eBPF/BCC are designed to aggregate data in the critical path and compute a summary of the activity. Ftrace and LTTng are designed to extract traces of execution for high resolution post-processing with as small overhead as possible. Aggregation is very powerful and gives a quick overview of the current activity of the system. Tracing extracts the detailed activity at various levels and allows in-depth understanding of a particular behaviour after the fact by allowing to run as many analyses as necessary on the captured trace. In terms of impact on the traced system, trace buffering scales better with the number of cores than aggregation approaches due its ability to partition the trace data into per-core buffers. Both approaches have upsides and downsides and should not be seen as being in competition, they address different use-cases and can even complement each other.
- brendangregg 10y agoYou're right that a key feature and differentiator of DTrace/stap/BPF is kernel aggregations, but they can do per-event output as well. But I think I know what you mean, especially as I was at the sysdig summit yesterday and could see a major difference. I think the two models for tracers, playing on their strengths, are: 1. real-time analysis tracers (DTrace/stap/BPF), and 2. offline analysis tracers (LTTng, sysdig). Both can do the other as well, but I'm just pointing out strengths. sysdig (and I believe LTTng) has done great work at creating capture files that can then be analyzed offline in many many different ways, and they've optimized the way full-event dumps can be captured and saved (which I know LTTng has done as well). DTrace/stap/BPF don't have any offline capture file capabilities -- they could do it, but it's not been their focus.
- asymmetric 10y ago> In 2014 I joined the Netflix cloud performance team. Having spent years as a DTrace expert, it might have seemed crazy for me to move to Linux I thought Netflix was mostly running FreeBSD [1]. Is it only the Open Connect Appliance? [1]: https://www.freebsdfoundation.org/testimonial/netflix/ https://www.freebsdfoundation.org/testimonial/netflix/
- brendangregg 10y agoWhen you login to Netflix and browse videos, you're running on the Netflix cloud, which is massive, AWS/EC2, and mostly Ubuntu Linux. When you hit play, you're running on the OCA FreeBSD CDN, which is also a large deployment.
- Annatar 10y agoSo why didn't they just deploy FreeBSD across the entire server park? That would also give you DTrace again...
- X86BSD 10y agoI'll put $1 on politics. I mean look, you have an OS that bgregg has had to pour how much effort into to get the observability that FreeBSD already had? And that's just the observability part. Then you have the FreeBSD network stack. To me it's clear based on the work done on Linux it was a political choice.
- Annatar 10y agoYeah but if that's the case, it's really bad. There is no place for politics in computer science or information technology.
- empthought 10y agoI think it's more likely because Netflix uses the JVM -- probably the Oracle JDK -- which is supported on Linux but not on FreeBSD.
- easytiger 10y agoReally rather unfortunate that big enterprise platforms such as banks and so forth are so far behind on their kernel version that it will be approximately 7-8 years before they will have this capability, unless RH backport of course.
- twblalock 10y agoOn the other hand, I'm glad the banks who handle my money don't upgrade to the latest and greatest software without taking very, very stringent precautions to make sure everything will work.
- obitoo 10y agoIn my experience thats not the case - its more like 'It works, no-one touch it! We're spending our money on more visible things' (several years later):- "Whats that, its no longer going to be supported? Damn, now we have to upgrade"