5 ms·

Just a quick clarification: while Brendan was certainly an active DTrace user and evangelist, he wasn't involved in the development of DTrace itself -- or its r

Just a quick clarification: while Brendan was certainly an active DTrace user and evangelist, he wasn't involved in the development of DTrace itself -- or its rollout. (Brendan came to Sun in 2006; DTrace was released in 2003.) As for eBPF with respect to DTrace, I would say that they are different systems with different goals and approaches rather than one eclipsing the other. (There are certainly many things that DTrace can do that eBPF/BCC cannot, some of the details of which we elaborated on in our 20th anniversary of DTrace's initial integration.[0]) Edit: We actually went into much more specific detail on eBPF/BCC in contrast to DTrace a few weeks after the 20th anniversary podcast.[1] [0] https://www.youtube.com/watch?v=IeUFzBBRilM https://www.youtube.com/watch?v=IeUFzBBRilM [1] https://www.youtube.com/watch?v=mqvVmYhclAg#t=12m7s https://www.youtube.com/watch?v=mqvVmYhclAg#t=12m7s

tanelpoder 2y ago
Thanks, yes I was more or less aware of that (I'd been using DTrace since Solaris 10 beta in 2004 or 2003?)... By rollout I really meant "getting the word out there"... that's half the battle in my experience (that's why this post here! :-) What I loved about DTrace was that once it was out, even in beta, it was pretty complete and worked - all the DTrace ports that I've tried, including on Windows (!) a few years ago were very limited or had some showstopper issues. I guess eBPF was like that too some years ago, but by now it's pretty sweet even for more regular consumer who don't keep track of its development. Edit: Oh, wasn't aware of the timeline, I may have some dates (years) wrong in my memory
abrookewood 2y ago
Yes, not involved in DTrace itself, but he did write a bunch of DTrace Tools which led to an interesting meeting with a Sun exec: https://www.brendangregg.com/blog/2021-06-04/an-unbelievable-demo.html https://www.brendangregg.com/blog/2021-06-04/an-unbelievable...
anonfordays 2y ago
>As for eBPF with respect to DTrace, I would say that they are different systems with different goals and approaches For sure. Different systems, different times. >rather than one eclipsing the other. It does seem that DTrace has been eclipsed though, at least in Linux (which runs the vast majority of the world's compute). Is there a reason to use DTrace over eBPF for tracing and observability in Linux? >There are certainly many things that DTrace can do that eBPF/BCC cannot This may be true, but that gap is closing. There are certainly many things that eBPF can do that DTrace cannot, like Cilium.
- tanelpoder 2y ago
  Perhaps familiarity with the syntax of DTrace if coming from Solaris-heavy enterprise background. But then again, too many years have passed since Solaris was a major mainstream platform. Oracle ships and supports DTrace on (Oracle) Linux by the way, but DTrace 2.0 on Linux is a scripting frontend that gets compiled to eBPF under the hood. Back when I tried to build xcapture with DTrace, I could launch the script and use something like /pid$oracle::func:entry/ but IIRC the probe was attached only to the processes that already existed and not any new ones that were started after loading the DTrace probes. Maybe I should have used some lower level APIs or something - but eBPF on Linux automatically handles both existing and new processes.
  - bch 2y ago
    > eBPF on Linux automatically handles both existing and new processes Without knowing your particular case, DTrace does too - it’d certainly be tricky to use if you’re trying to debug software that “instantly crashes on startup” if it couldn’t do that. “execname” (not “pid”) is where I’d look, or perhaps that part of the predicate is skipable; regardless, should be possible.
    - tanelpoder 2y ago
      For example I used something like "pid:module:funcname:entry" probe for userspace things (not pid$123 or pid$target, just pid to catch all PIDs using the module/funcname of interest). And back when I tested, it didn't automatically catch any new PIDs so these probes were not fired for them unless I restarted my DTrace script (but it was probably year <2010 when I last tested it). Execname is a variable in DTrace and not a probe (?), so how would it help with automatically attaching to new PIDs? Now that I recall more details, there was no issue with statically defined kernel "fbt" probes nor "profile", but the userspace pid one was where I hit this limitation.
      - bch 2y ago
        > Execname is a variable in DTrace and not a probe (?), so how would it help with automatically attaching to new PIDs? You're correct, and I may have provided "a solution" to a misunderstanding of your problem - I don't think the "not matching new procs/pids" is inherent in DTrace, so indeed you might have run into an implementation issue (as it was 15 years ago). I misunderstood you as perhaps using a predicate matching a specific pid; my fault.

tanelpoder 2y ago

Thanks, yes I was more or less aware of that (I'd been using DTrace since Solaris 10 beta in 2004 or 2003?)... By rollout I really meant "getting the word out there"... that's half the battle in my experience (that's why this post here! :-) What I loved about DTrace was that once it was out, even in beta, it was pretty complete and worked - all the DTrace ports that I've tried, including on Windows (!) a few years ago were very limited or had some showstopper issues. I guess eBPF was like that too some years ago, but by now it's pretty sweet even for more regular consumer who don't keep track of its development. Edit: Oh, wasn't aware of the timeline, I may have some dates (years) wrong in my memory

abrookewood 2y ago

Yes, not involved in DTrace itself, but he did write a bunch of DTrace Tools which led to an interesting meeting with a Sun exec: https://www.brendangregg.com/blog/2021-06-04/an-unbelievable-demo.html https://www.brendangregg.com/blog/2021-06-04/an-unbelievable...

anonfordays 2y ago

>As for eBPF with respect to DTrace, I would say that they are different systems with different goals and approaches For sure. Different systems, different times. >rather than one eclipsing the other. It does seem that DTrace has been eclipsed though, at least in Linux (which runs the vast majority of the world's compute). Is there a reason to use DTrace over eBPF for tracing and observability in Linux? >There are certainly many things that DTrace can do that eBPF/BCC cannot This may be true, but that gap is closing. There are certainly many things that eBPF can do that DTrace cannot, like Cilium.

tanelpoder 2y ago
Perhaps familiarity with the syntax of DTrace if coming from Solaris-heavy enterprise background. But then again, too many years have passed since Solaris was a major mainstream platform. Oracle ships and supports DTrace on (Oracle) Linux by the way, but DTrace 2.0 on Linux is a scripting frontend that gets compiled to eBPF under the hood. Back when I tried to build xcapture with DTrace, I could launch the script and use something like /pid$oracle::func:entry/ but IIRC the probe was attached only to the processes that already existed and not any new ones that were started after loading the DTrace probes. Maybe I should have used some lower level APIs or something - but eBPF on Linux automatically handles both existing and new processes.
- bch 2y ago
  > eBPF on Linux automatically handles both existing and new processes Without knowing your particular case, DTrace does too - it’d certainly be tricky to use if you’re trying to debug software that “instantly crashes on startup” if it couldn’t do that. “execname” (not “pid”) is where I’d look, or perhaps that part of the predicate is skipable; regardless, should be possible.
  - tanelpoder 2y ago
    For example I used something like "pid:module:funcname:entry" probe for userspace things (not pid$123 or pid$target, just pid to catch all PIDs using the module/funcname of interest). And back when I tested, it didn't automatically catch any new PIDs so these probes were not fired for them unless I restarted my DTrace script (but it was probably year <2010 when I last tested it). Execname is a variable in DTrace and not a probe (?), so how would it help with automatically attaching to new PIDs? Now that I recall more details, there was no issue with statically defined kernel "fbt" probes nor "profile", but the userspace pid one was where I hit this limitation.
    - bch 2y ago
      > Execname is a variable in DTrace and not a probe (?), so how would it help with automatically attaching to new PIDs? You're correct, and I may have provided "a solution" to a misunderstanding of your problem - I don't think the "not matching new procs/pids" is inherent in DTrace, so indeed you might have run into an implementation issue (as it was 15 years ago). I misunderstood you as perhaps using a predicate matching a specific pid; my fault.