8 ms·

Unfortunately for DTrace, this is too late. Oracle should have done this years ago. Now Linux has a more powerful tracer builtin, eBPF, and it would be a backwa

Unfortunately for DTrace, this is too late. Oracle should have done this years ago. Now Linux has a more powerful tracer builtin, eBPF, and it would be a backwards step to switch the kernel code to DTrace (assuming the DTrace port is completed, which it is not). I'm sure this will not be lost on the maintainers, who have the ultimate say as to what is included in Linux mainline. The only hope for DTrace is to have the frontend emit BPF bytecode. The bulk of this GPL DTrace code is no longer needed, only the user-level front end.

burke 9y ago
As a non-expert, I still find this exciting. While eBPF may be more powerful, I find it far less approachable than DTrace. Perhaps this is just a failure to discover the right documentation. I use DTrace now and then on macOS and have resented that I can't really use it on Linux as well.
- byefruit 9y ago
  With regards to tracing eBPF has actually moved at a fairly brisk pace the last year or two and it will probably take a while for the documentation and knowledge to permeate. The usability side you mention is what Brendan's suggesting is worth porting from DTrace, just the front-end that the users sees. That would just back onto eBPF and the user would know little different.
- brendangregg 9y ago
  eBPF lacks a high level front end, in part because that's optional. Engineering effort has gone into the essentials first: kernel development of eBPF, basic user-level frameworks to use it (bcc), and lots and lots of testing and bug fixes to make that rock solid. bcc is not as easy to code in, but I've been able to port over many of my DTraceToolkit scripts already. We're only more recently looking at higher level languages, as a nice to have. Here's one project: https://github.com/ajor/bpftrace https://github.com/ajor/bpftrace We can also look at taking the /usr/sbin/dtrace code (licenses permitting), and calling into libbcc and emitting BPF bytecode. That'd give us a D language front-end to BPF, so I can run my old DTrace scripts directly (plus sell more copies of my DTrace book -- I joke ;-) Help on either of those projects would be appreciated (bpftrace or DTrace-BPF).
  - viraptor 9y ago
    Am I reading too much into the last sentence, or are they actually trying to use existing probes with dtrace frontend: > So with all those techniques now available in the linux kernel it will be exciting to see if dtrace for linux can unite them all.
- nailer 9y ago
  IIRC one of the big things about dtrace vs systemtap is that dtrace could trace a process from userspace to kernel space and then back again. Can eBPF do this? How else do they compare?
  - brendangregg 9y ago
    That would not be recalling correctly: SystemTap does that, and so does eBPF (via uprobes and kprobes). A large problem with SystemTap is that it was developed in the open, and in the early days people were scared off with its kernel panics. SystemTap built a negative brand: first impressions matter. DTrace, however, was largely built before its public release, and with the help from many other engineers and test teams. People's first impressions with DTrace was something that was already awesome. Of course, there are other problems with SystemTap: its process of compiling code and loading kernel modules was always risker than an interpreter. As for how eBPF compares: I summarized it last year http://www.brendangregg.com/blog/2016-10-27/dtrace-for-linux-2016.html http://www.brendangregg.com/blog/2016-10-27/dtrace-for-linux...
    - fche 9y ago
      Note for freshness that systemtap can also emit eBPF directly now. So kernel modules are the fastest & most powerful way, but not the only one.
    - fche 9y ago
      > and in the early days people were scared off with its kernel panics. It is ironic that to this day those kernel panics are associated with systemtap (the visible user interface) rather than the kernel (where the vast majority of the responsible bugs actually lay). But perceptions indeed matter, even erroneous ones.
    - nailer 9y ago
      Thanks. How does the stap language being compiled into kernel modules differ from an interpreter? Both seem to have a limited surface (unless you're doing C directly in systemtap)
  - 9y ago
AceJohnny2 9y ago
Tangential, but I feel the need to quote Dan Colascione: "Ah Linux, in which a thread is a process, a semantic patch is a lint rule, and a packet filter is a system profiler." ;) (but seriously, your work in exploring and expanding it has been wonderful. Thank you!)
- aardvark291 9y ago
  What does this mean? I don't really understand the quote.
  - blinkingled 9y ago
    The Linux kernel treats threads no different than processes that share an address space which is unusual in UNIX land. Semantic patch probably refers to coccinelle - look it up, it's pretty cool, makes it easy to deal with API changes, renames etc. eBPF comes from extended Berkeley Packet Filter - where packet filter code is JITed to be fast. It's also used for dtrace like tracing along with kprobes to run profiling code in kernel safely.
    - caf 9y ago
      The Linux kernel treats threads no different than processes that share an address space which is unusual in UNIX land. They also share a signal handler table, open file table and current working directory. If a process-directed signal is sent to the process any of the threads with the signal unblocked can handle it. If a default signal disposition causes one thread to exit (eg SIGKILL or SIGSEGV) then all threads in the process exit. The exit_group(2) syscall causes all threads to exit (which is used to implement the POSIX exit() libc function). Really, "threads mostly are just processes" hasn't been true for a very long time. Mostly the only ways in which that is still true is in scheduling, where Linux schedules each thread independently (what POSIX calls PTHREAD_SCOPE_SYSTEM) and in credentials, where threads each have their own set of current, real and effective uids / gids (and glibc has to use a signal-based hack to synchronise them).
      - 9y ago
drb91 9y ago
Well, none of my dtrace knowledge works with eBPF, and eBPF is only useful on linux. This is still a very welcome change :)
- cookiecaper 9y ago
  Yeah, the compatibility element is really important. I've been doing tons of research around illumos, SmartOS, etc., lately and I'm hoping to deploy it in some projects soon. DTrace knowledge becoming properly portable across systems is a great thing. I've been very impressed with illumos from a technical standpoint, as a Linux admin for over 10 years whose never really had cause to seriously wander into the now-esoteric "proper Unices". Combination of the massive relief that is ZOL and the disastrous aftermath of the [ongoing] Linuxland "container revolution" got me looking for more. But before I commit to illumos in a substantial way, I'm trying to understand the social dynamics around it more, and whether the community is essentially just crossing fingers and hoping that Samsung doesn't shutter Joyent, as they've done to other open-source acquisitions, especially as alternate distributors like OmniTI have bowed out. So I've been doing lots of sleuthing and trying to piece together some of the drama behind this whole ecosystem. A lot of the momentum on SmartOS seems like it gave out during the first half of 2014 when Brendan and many other important Sun alumni left Joyent. Bryan Cantrill addressed this in an AMA by saying that he thinks some of them just tired of fighting an uphill battle (which, IMO, is totally understandable and actually doesn't seem like it's too far off the mark), but the close timing with which basically everyone except Bryan Cantrill made this decision makes me wonder if there was some internal event at Joyent that signaled a need to head for the hills. This theory is inflamed by Brendan's apparently complete abandonment of illumos and the tone he takes when he addresses illumos/Solaris, frequently telling people to avoid it and that Linux is much better nowadays. He didn't leave Joyent and continue to extoll the virtues of illumos or SmartOS, even though his farewell post says that his high opinion on the value of illumos tech hadn't changed. He left Joyent and seemed like he was relieved that he could finally let it all out and talk about the futility of illumos/Solaris. These days, he frequently leaves comments saying things like (paraphrasing here) "You could use [illumos distro], but I don't know why you would." and "We'd be in trouble if we picked the wrong OS at big companies like Netflix, and none of us are picking Solaris." It just feels like there are pieces of the puzzle missing here. I get that there is a long and storied history to illumos (as Cantrill points out in one talk, the codebase is older than he is!) and that everyone brings a lot of personal perspective and emotion to it, so I don't expect there to be a simple answer per se, but I just want to feel like I've got my arms more around it. SmartOS really does seem like a saner system for people who want things that work without constantly fussing, but I don't want to become dependent on a system whose future appears to hinge completely on Cantrill's continued employment with Joyent, especially post-exit, as he is likely bound by golden handcuffs. Node.js is another Joyent puzzler with its own long and tortured history (though obviously, much shorter than Solaris/illumos). It's strange, imo, that they were ever really involved in that in the first place. I can only guess that they were quick to snatch up Dahl and ride the wave before Google et al could get their wits about them and establish control over that segment, but maybe it has more to do with Joyent's relationship with web publishing via TextDrive, I don't really know. But if SmartOS weren't otherwise so charming, its integration of Node.js for system control scripts would seriously put me off. Disclaimer: I have no idea what I'm talking about, I wasn't there, I don't know any of the participants involved, and I'm not trying to assign any particular feelings or values to anyone. This is just my read of the ecosystem and dynamic from reading stuff on the internet, and it is very likely wrong. Corrections are welcome.
floatboth 9y ago
How would USDT probes work with a BPF backend?
- brendangregg 9y ago
  Like this: http://www.brendangregg.com/blog/2016-10-12/linux-bcc-nodejs-usdt.html http://www.brendangregg.com/blog/2016-10-12/linux-bcc-nodejs... Although that's a bit old, and Sasha and others have done improvements, so we probably need some updated USDT tracing docs.
kyaghmour 9y ago
Indeed. There's probably some sort of Greek tragedy to be written about the story of tracing and Linux. Or maybe we can just go for a beer at the next LPC and laugh this one out ;)
geokon 9y ago
I'm trying to "upgrade" from valgrind and understand the more advanced tools available but I'm having trouble understanding how perf relates to eBPF. I can't quite makes sense of your description "BPF makes perf tracing programmatic, and takes perf from being a counting & sampling-with-post-processing tracer, to a fully in-kernel programmable tracer" Are you saying eBPF can make your programs trigger perf tracing on-demand?
kloczek 9y ago
Better? OK, could you please show me eBPF oneliner code which is analogue of the DTrace command like: # dtrace -qn 'syscall::write:entry /execname == "mysqld"/ {self->stime = timestamp;} syscall::write:return /self->stime != 0/ {@LWrite = quantize(timestamp - self->stime);} tick-10s {printa(@LWrite);}' ?? Above is now working not only under Solaris but under Oracle UEK Linux kernel as well.
- brendangregg 9y ago
  # funclatency -i 10 -p `pgrep -n mysqld` sys_write Whoops, that was easy. I think argdist can do this as well. I'd have picked a different one-liner. It's not hard to come up with DTrace one-liners that are currently horribly hard with eBPF and bcc. Which is why I said we could have DTrace's D as a front end to bcc/eBPF. Or the bpftrace project will do that, and already has similar one-liners. Check it out[1]. Either way, we win. It's the kernel eBPF code that is more featured. It's great that Oracle is doing more with open source, and we may use some user level components, but Linux already has eBPF in the kernel now. [1] https://github.com/ajor/bpftrace https://github.com/ajor/bpftrace
  - ehvatum 9y ago
    Oh wow, I had no idea that funclatency and biolatency existed. There aren't enough hours in the day to learn about everything that could be saving me time. Huge thanks!
  - kloczek 9y ago
    Just FTR. Your example is wrong. [root@domek ~]# /usr/share/bcc/tools/funclatency -i 10 -p `pgrep -n mysqld` sys_write 0 functions matched by "sys_write". Exiting. The correct one is: funclatency -i 10 -p `pgrep -n mysqld` vfs_write Nevertheless, I'm not happy that there this no in this case analogue of the language which I can use on correlating data from different probes. Tools like funclatency cover exact class of the cases and on implementing anything more sophisticated still, I wont to have something like DTrace as frontend.

burke 9y ago

As a non-expert, I still find this exciting. While eBPF may be more powerful, I find it far less approachable than DTrace. Perhaps this is just a failure to discover the right documentation. I use DTrace now and then on macOS and have resented that I can't really use it on Linux as well.

byefruit 9y ago
With regards to tracing eBPF has actually moved at a fairly brisk pace the last year or two and it will probably take a while for the documentation and knowledge to permeate. The usability side you mention is what Brendan's suggesting is worth porting from DTrace, just the front-end that the users sees. That would just back onto eBPF and the user would know little different.
brendangregg 9y ago
eBPF lacks a high level front end, in part because that's optional. Engineering effort has gone into the essentials first: kernel development of eBPF, basic user-level frameworks to use it (bcc), and lots and lots of testing and bug fixes to make that rock solid. bcc is not as easy to code in, but I've been able to port over many of my DTraceToolkit scripts already. We're only more recently looking at higher level languages, as a nice to have. Here's one project: https://github.com/ajor/bpftrace https://github.com/ajor/bpftrace We can also look at taking the /usr/sbin/dtrace code (licenses permitting), and calling into libbcc and emitting BPF bytecode. That'd give us a D language front-end to BPF, so I can run my old DTrace scripts directly (plus sell more copies of my DTrace book -- I joke ;-) Help on either of those projects would be appreciated (bpftrace or DTrace-BPF).
- viraptor 9y ago
  Am I reading too much into the last sentence, or are they actually trying to use existing probes with dtrace frontend: > So with all those techniques now available in the linux kernel it will be exciting to see if dtrace for linux can unite them all.
nailer 9y ago
IIRC one of the big things about dtrace vs systemtap is that dtrace could trace a process from userspace to kernel space and then back again. Can eBPF do this? How else do they compare?
- brendangregg 9y ago
  That would not be recalling correctly: SystemTap does that, and so does eBPF (via uprobes and kprobes). A large problem with SystemTap is that it was developed in the open, and in the early days people were scared off with its kernel panics. SystemTap built a negative brand: first impressions matter. DTrace, however, was largely built before its public release, and with the help from many other engineers and test teams. People's first impressions with DTrace was something that was already awesome. Of course, there are other problems with SystemTap: its process of compiling code and loading kernel modules was always risker than an interpreter. As for how eBPF compares: I summarized it last year http://www.brendangregg.com/blog/2016-10-27/dtrace-for-linux-2016.html http://www.brendangregg.com/blog/2016-10-27/dtrace-for-linux...
  - fche 9y ago
    Note for freshness that systemtap can also emit eBPF directly now. So kernel modules are the fastest & most powerful way, but not the only one.
  - fche 9y ago
    > and in the early days people were scared off with its kernel panics. It is ironic that to this day those kernel panics are associated with systemtap (the visible user interface) rather than the kernel (where the vast majority of the responsible bugs actually lay). But perceptions indeed matter, even erroneous ones.
  - nailer 9y ago
    Thanks. How does the stap language being compiled into kernel modules differ from an interpreter? Both seem to have a limited surface (unless you're doing C directly in systemtap)
- 9y ago

AceJohnny2 9y ago

Tangential, but I feel the need to quote Dan Colascione: "Ah Linux, in which a thread is a process, a semantic patch is a lint rule, and a packet filter is a system profiler." ;) (but seriously, your work in exploring and expanding it has been wonderful. Thank you!)

aardvark291 9y ago
What does this mean? I don't really understand the quote.
- blinkingled 9y ago
  The Linux kernel treats threads no different than processes that share an address space which is unusual in UNIX land. Semantic patch probably refers to coccinelle - look it up, it's pretty cool, makes it easy to deal with API changes, renames etc. eBPF comes from extended Berkeley Packet Filter - where packet filter code is JITed to be fast. It's also used for dtrace like tracing along with kprobes to run profiling code in kernel safely.
  - caf 9y ago
    The Linux kernel treats threads no different than processes that share an address space which is unusual in UNIX land. They also share a signal handler table, open file table and current working directory. If a process-directed signal is sent to the process any of the threads with the signal unblocked can handle it. If a default signal disposition causes one thread to exit (eg SIGKILL or SIGSEGV) then all threads in the process exit. The exit_group(2) syscall causes all threads to exit (which is used to implement the POSIX exit() libc function). Really, "threads mostly are just processes" hasn't been true for a very long time. Mostly the only ways in which that is still true is in scheduling, where Linux schedules each thread independently (what POSIX calls PTHREAD_SCOPE_SYSTEM) and in credentials, where threads each have their own set of current, real and effective uids / gids (and glibc has to use a signal-based hack to synchronise them).
    - 9y ago

drb91 9y ago

Well, none of my dtrace knowledge works with eBPF, and eBPF is only useful on linux. This is still a very welcome change :)

cookiecaper 9y ago
Yeah, the compatibility element is really important. I've been doing tons of research around illumos, SmartOS, etc., lately and I'm hoping to deploy it in some projects soon. DTrace knowledge becoming properly portable across systems is a great thing. I've been very impressed with illumos from a technical standpoint, as a Linux admin for over 10 years whose never really had cause to seriously wander into the now-esoteric "proper Unices". Combination of the massive relief that is ZOL and the disastrous aftermath of the [ongoing] Linuxland "container revolution" got me looking for more. But before I commit to illumos in a substantial way, I'm trying to understand the social dynamics around it more, and whether the community is essentially just crossing fingers and hoping that Samsung doesn't shutter Joyent, as they've done to other open-source acquisitions, especially as alternate distributors like OmniTI have bowed out. So I've been doing lots of sleuthing and trying to piece together some of the drama behind this whole ecosystem. A lot of the momentum on SmartOS seems like it gave out during the first half of 2014 when Brendan and many other important Sun alumni left Joyent. Bryan Cantrill addressed this in an AMA by saying that he thinks some of them just tired of fighting an uphill battle (which, IMO, is totally understandable and actually doesn't seem like it's too far off the mark), but the close timing with which basically everyone except Bryan Cantrill made this decision makes me wonder if there was some internal event at Joyent that signaled a need to head for the hills. This theory is inflamed by Brendan's apparently complete abandonment of illumos and the tone he takes when he addresses illumos/Solaris, frequently telling people to avoid it and that Linux is much better nowadays. He didn't leave Joyent and continue to extoll the virtues of illumos or SmartOS, even though his farewell post says that his high opinion on the value of illumos tech hadn't changed. He left Joyent and seemed like he was relieved that he could finally let it all out and talk about the futility of illumos/Solaris. These days, he frequently leaves comments saying things like (paraphrasing here) "You could use [illumos distro], but I don't know why you would." and "We'd be in trouble if we picked the wrong OS at big companies like Netflix, and none of us are picking Solaris." It just feels like there are pieces of the puzzle missing here. I get that there is a long and storied history to illumos (as Cantrill points out in one talk, the codebase is older than he is!) and that everyone brings a lot of personal perspective and emotion to it, so I don't expect there to be a simple answer per se, but I just want to feel like I've got my arms more around it. SmartOS really does seem like a saner system for people who want things that work without constantly fussing, but I don't want to become dependent on a system whose future appears to hinge completely on Cantrill's continued employment with Joyent, especially post-exit, as he is likely bound by golden handcuffs. Node.js is another Joyent puzzler with its own long and tortured history (though obviously, much shorter than Solaris/illumos). It's strange, imo, that they were ever really involved in that in the first place. I can only guess that they were quick to snatch up Dahl and ride the wave before Google et al could get their wits about them and establish control over that segment, but maybe it has more to do with Joyent's relationship with web publishing via TextDrive, I don't really know. But if SmartOS weren't otherwise so charming, its integration of Node.js for system control scripts would seriously put me off. Disclaimer: I have no idea what I'm talking about, I wasn't there, I don't know any of the participants involved, and I'm not trying to assign any particular feelings or values to anyone. This is just my read of the ecosystem and dynamic from reading stuff on the internet, and it is very likely wrong. Corrections are welcome.

floatboth 9y ago

How would USDT probes work with a BPF backend?

brendangregg 9y ago
Like this: http://www.brendangregg.com/blog/2016-10-12/linux-bcc-nodejs-usdt.html http://www.brendangregg.com/blog/2016-10-12/linux-bcc-nodejs... Although that's a bit old, and Sasha and others have done improvements, so we probably need some updated USDT tracing docs.

kyaghmour 9y ago

Indeed. There's probably some sort of Greek tragedy to be written about the story of tracing and Linux. Or maybe we can just go for a beer at the next LPC and laugh this one out ;)

geokon 9y ago

I'm trying to "upgrade" from valgrind and understand the more advanced tools available but I'm having trouble understanding how perf relates to eBPF. I can't quite makes sense of your description "BPF makes perf tracing programmatic, and takes perf from being a counting & sampling-with-post-processing tracer, to a fully in-kernel programmable tracer" Are you saying eBPF can make your programs trigger perf tracing on-demand?

kloczek 9y ago

Better? OK, could you please show me eBPF oneliner code which is analogue of the DTrace command like: # dtrace -qn 'syscall::write:entry /execname == "mysqld"/ {self->stime = timestamp;} syscall::write:return /self->stime != 0/ {@LWrite = quantize(timestamp - self->stime);} tick-10s {printa(@LWrite);}' ?? Above is now working not only under Solaris but under Oracle UEK Linux kernel as well.

brendangregg 9y ago
# funclatency -i 10 -p `pgrep -n mysqld` sys_write Whoops, that was easy. I think argdist can do this as well. I'd have picked a different one-liner. It's not hard to come up with DTrace one-liners that are currently horribly hard with eBPF and bcc. Which is why I said we could have DTrace's D as a front end to bcc/eBPF. Or the bpftrace project will do that, and already has similar one-liners. Check it out[1]. Either way, we win. It's the kernel eBPF code that is more featured. It's great that Oracle is doing more with open source, and we may use some user level components, but Linux already has eBPF in the kernel now. [1] https://github.com/ajor/bpftrace https://github.com/ajor/bpftrace
- ehvatum 9y ago
  Oh wow, I had no idea that funclatency and biolatency existed. There aren't enough hours in the day to learn about everything that could be saving me time. Huge thanks!
- kloczek 9y ago
  Just FTR. Your example is wrong. [root@domek ~]# /usr/share/bcc/tools/funclatency -i 10 -p `pgrep -n mysqld` sys_write 0 functions matched by "sys_write". Exiting. The correct one is: funclatency -i 10 -p `pgrep -n mysqld` vfs_write Nevertheless, I'm not happy that there this no in this case analogue of the language which I can use on correlating data from different probes. Tools like funclatency cover exact class of the cases and on implementing anything more sophisticated still, I wont to have something like DTrace as frontend.