5 ms·

I'm uncertain, but I thought dtrace on illumos was interpreted. Am I wrong? If I am, than all we'd have to rewrite was the compiler backend, which is much easi

I'm uncertain, but I thought dtrace on illumos was interpreted. Am I wrong? If I am, than all we'd have to rewrite was the compiler backend, which is much easier, so that would be nice.

Annatar 10y ago
dtrace(1) itself is compiled from C into an ELF binary executable. % file `which dtrace` /usr/sbin/dtrace: ELF 32-bit LSB executable 80386 Version 1, dynamically linked, stripped The DTrace language, D, is interpreted. By DTrace. The problem lies in the fact that neither the GNU/Linux kernel, nor the GNU applications provide DTrace probe points. On Solaris and thus on illumos, and thus on SmartOS, there are tens of thousands of probe points and numerous probe providers. Some external applications, like PostgreSQL or PHP, added DTrace probes, and all is well on Solaris / illumos / SmartOS. Some, like node.js had providers and probes added by engineers at Joyent. http://dtrace.org/blogs/dap/2012/04/25/profiling-node-js/ http://dtrace.org/blogs/dap/2012/04/25/profiling-node-js/ http://dtrace.org/blogs/dap/2013/11/20/understanding-dtrace-ustack-helpers/ http://dtrace.org/blogs/dap/2013/11/20/understanding-dtrace-... http://dtrace.org/blogs/blog/category/node-js/ http://dtrace.org/blogs/blog/category/node-js/ GNU/Linux would have to do the same thing. It currently only has but a handful of DTrace probes and providers, which is understandably not very useful. http://dtrace.org/blogs/ahl/2011/10/05/dtrace-for-linux-2/ http://dtrace.org/blogs/ahl/2011/10/05/dtrace-for-linux-2/
- qwertyuiop924 10y ago
  None of that is at all what I meant: allow me to clarify. I was talking about the DTrace language, which I have been avoiding calling D up until this point, so as to avoid confusion with the other D. In this post, when I talk about D, I will be referring to the DTrace language. Linux now has a tracing system called eBPF, which provides many of the same advantages of DTrace. This is what Gregg's blog post was about. However, eBPF requires compiling tracer scripts, or at least parts of the tracer scripts, into bytecode (IIRC). Currently, the bytecode is usually compiled from C code. However, it is ungainly and impractical to write a bunch of C every time you want to run a trace. So I asked if they had plans to support compiling a higher level language. At this point, somebody suggested that somebody should work on compiling the DTrace language, D, to eBPF bytecode. I thought that this would be a good idea, and we were discussing how viable it would be. I thought that you had suggested using Illumos DTrace as a base for this compiler. Since by my impression, Illumos DTrace interprets D, I thought that this would require almost a complete rewrite, and thus it wouldn't very helpful. It seems you meant something else. So what did you mean?
  - Annatar 10y ago
    I think we both got lost, didn't we? (:-)) So let's rewind: It is a really nice language, although missing a few things that BPF can do that DTrace can't (like saving and retrieving stacks), so it'd need to be enhanced. But with the warning that I'm not a lawyer: before beginning work on a DTrace/BPF front-end, I'd start by talking to a copyright lawyer to see if permission or a license is needed from Oracle. I fail to see how Oracle would be relevant, given that the entire DTrace is released under the CDDL. My question to Brendan was why he couldn't just take or enhance the existing DTrace codebase to do the front-end for BPF? (Personally, I think the other guy working on integrating BPF deeper into the GNU/Linux kernel, rather than biting the bullet, taking DTrace and doing the same thing Solaris engineers did to the SunOS kernel, is terribly mis-guided, especially since some probes and providers already exist in Linux. In the end, it will be a "Linux zoo", like everything else in Linux: "56" competing solutions for doing one thing, none of them comprehensive, and no consistency. Linux history is being repeated again. You have one comprehensive tool which works across several operating systems, DTrace, and Linux is yet again different from everyone else. Reminds me a lot of Microsoft Windows.)
    - qwertyuiop924 10y ago
      It's not ideal, but eBPF works now, and it works on all the new kernels, which is more than can be said for any of the DTrace on Linux projects. It looks like eBPF is becoming that unified solution (mostly: ftrace and a few others still exist, but eBPF is the most capable, and is picking up steam). As for being different from what the other Unixes did, that's why I was in favor of developing a frontend that supported D, so that we could at least have a shared language with the rest of unixland. >My question to Brendan was why he couldn't just take or enhance the existing DTrace codebase to do the front-end for BPF? Brendan didn't answer, so I don't know, but my guess was that turning the interpreter into a compiler would require a near-complete rewrite.
      - brendangregg 10y ago
        > that's why I was in favor of developing a frontend that supported D, so that we could at least have a shared language with the rest of unixland. Yes, D everyone would be nice, but what exactly does it mean? We can share DTrace scripts? Docs? Blog posts? Books? I've been porting them all over to bcc/BPF. Am I missing something? People have already began work developing new languages, eg, ply https://wkz.github.io/ply/ https://wkz.github.io/ply/. What if we develop a language that's much better than D? We need to make enhancements, anyway. I should reiterate something I covered in the post: most people won't care about this. Most people didn't write DTrace scripts when they had the option to (did either of you write DTrace scripts? have some on github?). Most people used the tools. And today, people can "apt install bcc-tools" and use DTraceToolkit like tools. If someone wants to engage lawyers & Oracle and see if or what needs to be done to use DTrace, then great, it'd make my job easier when developing these tools (and I'd sell more DTrace books :). But I'd also like to see someone take a swing at developing a better language as another possibility.

I'm uncertain, but I thought dtrace on illumos was interpreted. Am I wrong? If I am, than all we'd have to rewrite was the compiler backend, which is much easier, so that would be nice.

Annatar 10y ago

dtrace(1) itself is compiled from C into an ELF binary executable. % file `which dtrace` /usr/sbin/dtrace: ELF 32-bit LSB executable 80386 Version 1, dynamically linked, stripped The DTrace language, D, is interpreted. By DTrace. The problem lies in the fact that neither the GNU/Linux kernel, nor the GNU applications provide DTrace probe points. On Solaris and thus on illumos, and thus on SmartOS, there are tens of thousands of probe points and numerous probe providers. Some external applications, like PostgreSQL or PHP, added DTrace probes, and all is well on Solaris / illumos / SmartOS. Some, like node.js had providers and probes added by engineers at Joyent. http://dtrace.org/blogs/dap/2012/04/25/profiling-node-js/ http://dtrace.org/blogs/dap/2012/04/25/profiling-node-js/ http://dtrace.org/blogs/dap/2013/11/20/understanding-dtrace-ustack-helpers/ http://dtrace.org/blogs/dap/2013/11/20/understanding-dtrace-... http://dtrace.org/blogs/blog/category/node-js/ http://dtrace.org/blogs/blog/category/node-js/ GNU/Linux would have to do the same thing. It currently only has but a handful of DTrace probes and providers, which is understandably not very useful. http://dtrace.org/blogs/ahl/2011/10/05/dtrace-for-linux-2/ http://dtrace.org/blogs/ahl/2011/10/05/dtrace-for-linux-2/

qwertyuiop924 10y ago
None of that is at all what I meant: allow me to clarify. I was talking about the DTrace language, which I have been avoiding calling D up until this point, so as to avoid confusion with the other D. In this post, when I talk about D, I will be referring to the DTrace language. Linux now has a tracing system called eBPF, which provides many of the same advantages of DTrace. This is what Gregg's blog post was about. However, eBPF requires compiling tracer scripts, or at least parts of the tracer scripts, into bytecode (IIRC). Currently, the bytecode is usually compiled from C code. However, it is ungainly and impractical to write a bunch of C every time you want to run a trace. So I asked if they had plans to support compiling a higher level language. At this point, somebody suggested that somebody should work on compiling the DTrace language, D, to eBPF bytecode. I thought that this would be a good idea, and we were discussing how viable it would be. I thought that you had suggested using Illumos DTrace as a base for this compiler. Since by my impression, Illumos DTrace interprets D, I thought that this would require almost a complete rewrite, and thus it wouldn't very helpful. It seems you meant something else. So what did you mean?
- Annatar 10y ago
  I think we both got lost, didn't we? (:-)) So let's rewind: It is a really nice language, although missing a few things that BPF can do that DTrace can't (like saving and retrieving stacks), so it'd need to be enhanced. But with the warning that I'm not a lawyer: before beginning work on a DTrace/BPF front-end, I'd start by talking to a copyright lawyer to see if permission or a license is needed from Oracle. I fail to see how Oracle would be relevant, given that the entire DTrace is released under the CDDL. My question to Brendan was why he couldn't just take or enhance the existing DTrace codebase to do the front-end for BPF? (Personally, I think the other guy working on integrating BPF deeper into the GNU/Linux kernel, rather than biting the bullet, taking DTrace and doing the same thing Solaris engineers did to the SunOS kernel, is terribly mis-guided, especially since some probes and providers already exist in Linux. In the end, it will be a "Linux zoo", like everything else in Linux: "56" competing solutions for doing one thing, none of them comprehensive, and no consistency. Linux history is being repeated again. You have one comprehensive tool which works across several operating systems, DTrace, and Linux is yet again different from everyone else. Reminds me a lot of Microsoft Windows.)
  - qwertyuiop924 10y ago
    It's not ideal, but eBPF works now, and it works on all the new kernels, which is more than can be said for any of the DTrace on Linux projects. It looks like eBPF is becoming that unified solution (mostly: ftrace and a few others still exist, but eBPF is the most capable, and is picking up steam). As for being different from what the other Unixes did, that's why I was in favor of developing a frontend that supported D, so that we could at least have a shared language with the rest of unixland. >My question to Brendan was why he couldn't just take or enhance the existing DTrace codebase to do the front-end for BPF? Brendan didn't answer, so I don't know, but my guess was that turning the interpreter into a compiler would require a near-complete rewrite.
    - brendangregg 10y ago
      > that's why I was in favor of developing a frontend that supported D, so that we could at least have a shared language with the rest of unixland. Yes, D everyone would be nice, but what exactly does it mean? We can share DTrace scripts? Docs? Blog posts? Books? I've been porting them all over to bcc/BPF. Am I missing something? People have already began work developing new languages, eg, ply https://wkz.github.io/ply/ https://wkz.github.io/ply/. What if we develop a language that's much better than D? We need to make enhancements, anyway. I should reiterate something I covered in the post: most people won't care about this. Most people didn't write DTrace scripts when they had the option to (did either of you write DTrace scripts? have some on github?). Most people used the tools. And today, people can "apt install bcc-tools" and use DTraceToolkit like tools. If someone wants to engage lawyers & Oracle and see if or what needs to be done to use DTrace, then great, it'd make my job easier when developing these tools (and I'd sell more DTrace books :). But I'd also like to see someone take a swing at developing a better language as another possibility.