3 ms·

> that's why I was in favor of developing a frontend that supported D, so that we could at least have a shared language with the rest of unixland. Yes, D every

> that's why I was in favor of developing a frontend that supported D, so that we could at least have a shared language with the rest of unixland. Yes, D everyone would be nice, but what exactly does it mean? We can share DTrace scripts? Docs? Blog posts? Books? I've been porting them all over to bcc/BPF. Am I missing something? People have already began work developing new languages, eg, ply https://wkz.github.io/ply/ https://wkz.github.io/ply/. What if we develop a language that's much better than D? We need to make enhancements, anyway. I should reiterate something I covered in the post: most people won't care about this. Most people didn't write DTrace scripts when they had the option to (did either of you write DTrace scripts? have some on github?). Most people used the tools. And today, people can "apt install bcc-tools" and use DTraceToolkit like tools. If someone wants to engage lawyers & Oracle and see if or what needs to be done to use DTrace, then great, it'd make my job easier when developing these tools (and I'd sell more DTrace books :). But I'd also like to see someone take a swing at developing a better language as another possibility.

qwertyuiop924 10y ago
Well, something good about the eBPF design is that we can go in both directions. It's possible to write both new language projects, and to port old ones. I'd work on it myself, if I had the requisite knowledge. Heck, maybe I'll learn some requisite knowledge and get on it.
- brendangregg 10y ago
  Out of curiosity, did you ever write DTrace scripts? SystemTap? BPF? Do you have some on github I can see?
  - qwertyuiop924 10y ago
    No. See what I mean about requisite knowledge? I've been interested in learning DTrace for some time now, as it seemed like a tremdously helpful tool, but my systems are all linux, I don't run Fedora or RHEL, so systemtap is out. In recent months, eBPF has become a viable option, so I've been looking into that, but haven't had time to dive in deep yet. This makes me a poor candidate for doing this sort of work, which is why I'm not saying for sure that I'd do it. But I might be willing to try. Don't get me wrong: I have no particular attachment to D as the tracing language. However, I do think it's worth supporting the tracing language that pretty much all of the other big unixes support if we can. Am I less than qualified to have that opinion? Yes. Think what you will.
    - brendangregg 10y ago
      Thanks, was wondering if this came from a specific language pain or not. Since all of this requires being on a newer kernel anyway, I'd do that first, then try the existing bcc tools and see where you are at. You might discover that you can do everything you want to with the existing tools, which include some powerful ad-hoc analysis tools (trace and argdist). Or, if you're missing one or two tools and let us know, and we may develop them for you and add them to bcc. Many work on the current Ubuntu Xenial 4.4. A few more on 4.6 (stack tracing), more on 4.7 (tracepoints), and all work on 4.9.
      - Annatar 10y ago
        Thanks, was wondering if this came from a specific language pain or not. No language pain with DTrace, but having to know kernel structs put a damper on things like you wouldn't believe. The problem is this: suppose you've just built mikmod on Solaris (or some variant thereof), and you're not the developer of the application. You want to see why the mikmod application's output is pausing every few seconds, causing a lag; that's all you care about - answers. DTrace to the rescue! Not so fast, you don't: with DTrace, you know what the answer to what you want to get is - why is the output freezing every few seconds? What you don't know is what to ask to get it. And I bet based on experience, that every tracing framework currently in existence has the same issue. Even in scenarios where I knew exactly what to ask, correctly interpreting the results is a laborious and involved arcane art exercise in itself - you yourself wrote several books on the subject!
        brendangregg 10y ago
        See my other comments -- Annatar, what is your real name? The problems you discuss -- knowing what to do with a tracer -- having little to do with the front-end.
caf 10y ago
Yes, D everyone would be nice, but what exactly does it mean? We can share DTrace scripts? Docs? Blog posts? Books? I've been porting them all over to bcc/BPF. Am I missing something? I think the most important thing to support would actually be DTrace's probe definition files. It would be great, for example, if the story for PostgreSQL's dtrace probe points was "your dtrace probe points are now accessible from eBPF programs too!".
- brendangregg 10y ago
  That already works! :) I wrote some MySQL examples in bcc that use the MySQL USDT probe points. Eg: https://github.com/iovisor/bcc/blob/master/examples/tracing/mysqld_query.py https://github.com/iovisor/bcc/blob/master/examples/tracing/... https://github.com/iovisor/bcc/blob/master/examples/tracing/mysqld_query_example.txt https://github.com/iovisor/bcc/blob/master/examples/tracing/... There's also a tool in /tools. I'll whip up some PostgreSQL examples too (unless someone else beats me to it).
  - caf 10y ago
    That looks great! (Why is char query[128] apparently unused?) Is USDT expected to become the de facto standard for userspace probe points?
    - brendangregg 10y ago
      Oops, that looks like a left-over from development, that's no longer needed. Thanks! One of us should fix the example. :)
Annatar 10y ago
did either of you write DTrace scripts? I did, some, in the beginning (circa 2006): I was stymied mostly by the realization that deep, deep knowledge of the kernel structures was required to make use of DTrace (I wasn't working as a kernel engineer per se at the time.) I forgot a lot of it in the meanwhile: just yesterday I was trying to get a simple ustack() using DTrace on a running process, and I even pulled open "Solaris Performance and Tools", and eventually when the process finished, I threw up my hands in frustration. All I wanted to do was see why the running process (oggenc) was taking so long. (But this BPF thing looks far, far more complicated and convoluted than D.) Nevertheless, I think D is ideal, because, in my case, it plays on my experience in programming AWK: apart from needing to know what in the kernel I wanted to probe, I could immediately start writing DTrace programs without having to learn the language. And that is amazing.
- brendangregg 10y ago
  See my other comments -- Annatar, what is your real name? I'll add: as someone who has written and published countless DTrace and BPF scripts, I don't know that pursing a DTrace front end is wise right now, for reasons I've already covered. I'm sorry you weren't able to solve that issue. I'd suggest starting with a profiler (timed sampling) if it was running on-CPU, to see where CPU time is spent.

qwertyuiop924 10y ago

Well, something good about the eBPF design is that we can go in both directions. It's possible to write both new language projects, and to port old ones. I'd work on it myself, if I had the requisite knowledge. Heck, maybe I'll learn some requisite knowledge and get on it.

brendangregg 10y ago
Out of curiosity, did you ever write DTrace scripts? SystemTap? BPF? Do you have some on github I can see?
- qwertyuiop924 10y ago
  No. See what I mean about requisite knowledge? I've been interested in learning DTrace for some time now, as it seemed like a tremdously helpful tool, but my systems are all linux, I don't run Fedora or RHEL, so systemtap is out. In recent months, eBPF has become a viable option, so I've been looking into that, but haven't had time to dive in deep yet. This makes me a poor candidate for doing this sort of work, which is why I'm not saying for sure that I'd do it. But I might be willing to try. Don't get me wrong: I have no particular attachment to D as the tracing language. However, I do think it's worth supporting the tracing language that pretty much all of the other big unixes support if we can. Am I less than qualified to have that opinion? Yes. Think what you will.
  - brendangregg 10y ago
    Thanks, was wondering if this came from a specific language pain or not. Since all of this requires being on a newer kernel anyway, I'd do that first, then try the existing bcc tools and see where you are at. You might discover that you can do everything you want to with the existing tools, which include some powerful ad-hoc analysis tools (trace and argdist). Or, if you're missing one or two tools and let us know, and we may develop them for you and add them to bcc. Many work on the current Ubuntu Xenial 4.4. A few more on 4.6 (stack tracing), more on 4.7 (tracepoints), and all work on 4.9.
    - Annatar 10y ago
      Thanks, was wondering if this came from a specific language pain or not. No language pain with DTrace, but having to know kernel structs put a damper on things like you wouldn't believe. The problem is this: suppose you've just built mikmod on Solaris (or some variant thereof), and you're not the developer of the application. You want to see why the mikmod application's output is pausing every few seconds, causing a lag; that's all you care about - answers. DTrace to the rescue! Not so fast, you don't: with DTrace, you know what the answer to what you want to get is - why is the output freezing every few seconds? What you don't know is what to ask to get it. And I bet based on experience, that every tracing framework currently in existence has the same issue. Even in scenarios where I knew exactly what to ask, correctly interpreting the results is a laborious and involved arcane art exercise in itself - you yourself wrote several books on the subject!
      - brendangregg 10y ago
        See my other comments -- Annatar, what is your real name? The problems you discuss -- knowing what to do with a tracer -- having little to do with the front-end.

caf 10y ago

Yes, D everyone would be nice, but what exactly does it mean? We can share DTrace scripts? Docs? Blog posts? Books? I've been porting them all over to bcc/BPF. Am I missing something? I think the most important thing to support would actually be DTrace's probe definition files. It would be great, for example, if the story for PostgreSQL's dtrace probe points was "your dtrace probe points are now accessible from eBPF programs too!".

brendangregg 10y ago
That already works! :) I wrote some MySQL examples in bcc that use the MySQL USDT probe points. Eg: https://github.com/iovisor/bcc/blob/master/examples/tracing/mysqld_query.py https://github.com/iovisor/bcc/blob/master/examples/tracing/... https://github.com/iovisor/bcc/blob/master/examples/tracing/mysqld_query_example.txt https://github.com/iovisor/bcc/blob/master/examples/tracing/... There's also a tool in /tools. I'll whip up some PostgreSQL examples too (unless someone else beats me to it).
- caf 10y ago
  That looks great! (Why is char query[128] apparently unused?) Is USDT expected to become the de facto standard for userspace probe points?
  - brendangregg 10y ago
    Oops, that looks like a left-over from development, that's no longer needed. Thanks! One of us should fix the example. :)

Annatar 10y ago

did either of you write DTrace scripts? I did, some, in the beginning (circa 2006): I was stymied mostly by the realization that deep, deep knowledge of the kernel structures was required to make use of DTrace (I wasn't working as a kernel engineer per se at the time.) I forgot a lot of it in the meanwhile: just yesterday I was trying to get a simple ustack() using DTrace on a running process, and I even pulled open "Solaris Performance and Tools", and eventually when the process finished, I threw up my hands in frustration. All I wanted to do was see why the running process (oggenc) was taking so long. (But this BPF thing looks far, far more complicated and convoluted than D.) Nevertheless, I think D is ideal, because, in my case, it plays on my experience in programming AWK: apart from needing to know what in the kernel I wanted to probe, I could immediately start writing DTrace programs without having to learn the language. And that is amazing.

brendangregg 10y ago
See my other comments -- Annatar, what is your real name? I'll add: as someone who has written and published countless DTrace and BPF scripts, I don't know that pursing a DTrace front end is wise right now, for reasons I've already covered. I'm sorry you weren't able to solve that issue. I'd suggest starting with a profiler (timed sampling) if it was running on-CPU, to see where CPU time is spent.