7 ms·

Let me put it this way. In the last say five years, I can't think of a single article I've come across where someone describes DTrace making their life better

Let me put it this way. In the last say five years, I can't think of a single article I've come across where someone describes DTrace making their life better (except one I wrote in 2011: http://blog.reverberate.org/2011/05/using-dtrace-on-os-x-to-debug.html http://blog.reverberate.org/2011/05/using-dtrace-on-os-x-to-...), or a single time anyone I know or work with has even mentioned DTrace. This is a technology that received an innovation award from the Wall Street Journal in 2006. Whatever impact DTrace is having today, it should have been much larger.

chris_wot 11y ago
You really aught to look at Bryan Cantrill's twitter feed then. It happens frequently enough that he notices blog posts and tweets them.
- digi_owl 11y ago
  Then again Cantrill has an axe to grind. In case others are wondering: https://en.wikipedia.org/wiki/Bryan_Cantrill https://en.wikipedia.org/wiki/Bryan_Cantrill
  - chris_wot 11y ago
    I'm no fan of the man, but it doesn't mean that dtrace is not an excellent tool.
JonathonW 11y ago
Dtrace isn't completely dead; it's used to great effect as the underlying technology behind Instruments (performance analyzer/visualizer) on OSX. It never has really managed to catch on as a standalone tool by itself, though. Part of that's the licensing situation on Linux, but I'd hypothesize that part of it is also that it's a pretty complicated tool to use, interpret, and act on-- you don't hear a lot about SystemTap (same concept, GPLed for Linux-friendliness) these days, either.
- seliopou 11y ago
  I'd push back on the 'complicated' part. The dtrace language is heavily based on awk, which every modern *NIX ships with, and that every software engineer and sysadmin have at least a passing familiarity with. The main difference between the two is whereas awk is meant to operate over a stream of semi-structured lines, dtrace operates over a stream of probe firings. The high-level structures of the scripts, though, are the same: /pattern/ { statements } In a awk, when a line matches a the patter, the corresponding statements fire. In dtrace, when a probe matches a pattern, the corresponding statements fire. Integers, strings, arrays, and all those other sorts of features of the language are the same (though there are some special functions introduced by dtrace to help you collect and visualize aggregations). So from a language perspective, if you know awk, you mostly know dtrace. All that's left to do is learn the pattern language. If you want to do that effectively, you have to know not just the syntax of the patterns, but useful probes to listen for, and useful ways to use the information that's gleaned. This involves knowing things about operating systems. Most software engineers don't know much about operating systems so they think the tool is stupid.
postwait 11y ago
We use it every day at Circonus. One of us is learning something new about how our stuff works every single day.
bixu_ 11y ago
I use DTrace fairly often to look at the boundaries between whatever interpreted language my app's written in and the kernel. For example, at $job recently found that some of our Python code was doing 25k open64 calls but only 5k close calls during a scheduled job - this led us to find/fix a bug that would have been a disaster later.
kev009 11y ago
I think you've accidentally set up a straw man because mentions of low level tools don't correlate to use. In fact "trending" popularity is how a lot of bad technical decisions are made, cue crappy database flavor of the month. The hammer is a pretty useful tool but builders don't blog du jour about how elegant combining the strike face and nail puller on one implement are. None the less, millions are used every day. DTrace is used extensively in FreeBSD development these days. It's being used to bring up new generations of Systems Programmers: http://teachbsd.org/ http://teachbsd.org/ My team uses DTrace to develop TCP, which ensures open standards and a copyfree implementation that ends up in Apple products, Juniper, NetApp, and who knows where else. Of course, also Netflix and the large CDN I work for which run at least 1/3 of North American Internet volume. We also use DTrace to fix the reference copyfree Intel drivers, which may in a small way pay back Illumos for the awesome tech they share. I think those end up in some popular embedded microkernels too. I grin thinking about people that emphasize "GNU/Linux", but I will empathize with FSF for a moment to show that low level software is not particularly exciting to most. GCC and GDB are hugely influential tools for the whole open source ecosystem, especially the high growth during the 80s and 90s that led to "Enterprise" acceptance. It's hard to imagine where everyone else would be, including the BSDs and many chip companies, were it not for the GNU toolchain. Nonetheless, most people just begrudgingly use these and hope they don't give them any trouble because of course they are usually trying to do something else.
Bux10 11y ago
I use dtrace often, but am too busy to write articles about it, sorry. A couple of times, dtrace has even been THE tool that helped me understand exactly why our big production systems were dying seemingly inexplicable and horrible deaths. This has directly resulted in us still being in business instead of fading into irrelevance. That's plenty impact for me.
dap 11y ago
> Let me put it this way. In the last say five years, I can't think of a single article I've come across where someone describes DTrace making their life better (except one I wrote in 2011: http://blog.reverberate.org/2011/05/using-dtrace-on-os-x-to-... http://blog.reverberate.org/2011/05/using-dtrace-on-os-x-to-...), or a single time anyone I know or work with has even mentioned DTrace. Then you haven't been looking: http://dtrace.org/blogs/brendan/2012/08/09/10-performance-wins/ http://dtrace.org/blogs/brendan/2012/08/09/10-performance-wi... http://dtrace.org/blogs/brendan/2014/02/11/another-10-performance-wins/ http://dtrace.org/blogs/brendan/2014/02/11/another-10-perfor... https://www.joyent.com/blog/node-js-in-production-runtime-log-snooping https://www.joyent.com/blog/node-js-in-production-runtime-lo... https://twitter.com/b0rk/status/681536144362848257 https://twitter.com/b0rk/status/681536144362848257 https://speakerdeck.com/sartak/dtrace-war-stories https://speakerdeck.com/sartak/dtrace-war-stories https://www.bignerdranch.com/blog/hooked-on-dtrace-part-1/ https://www.bignerdranch.com/blog/hooked-on-dtrace-part-1/ https://medium.com/@jlouis666/erlang-dirty-scheduler-overhead-6e1219dcc7#.mppk9kx55 https://medium.com/@jlouis666/erlang-dirty-scheduler-overhea... Seriously, it wasn't that hard to find this stuff. I found the last one by searching HN: https://hn.algolia.com/?q=dtrace https://hn.algolia.com/?q=dtrace Like others who replied to you, I use DTrace every day. Very often, it solves in seconds what would have taken hours without it. Frequently, it's helped solve problems I'd never have been able to understand without it. My colleagues use it all the time as well. Sometimes, I write about the general techniques (I wrote the "runtime log snooping" blog post above), and most of our bug reports in recent years are public, but we don't write about individual bugs all that often. (Seems like you wouldn't have run into them if we had, though.)
auberlac 11y ago
I don't blog, and therefore have never written about the multiple times DTrace helped me and saved me a lot of time. It also made my job fun again as a sysadmin. Here are couple of examples I remember, from my days at a large financial institution. We're planning a samba upgrade on a server with ~700 clients. Non regression tests show the performance of copying large files has degraded by 30% Dtrace, aggregating on smbd stacks, helped us figure out the root cause in half an hour. This was reported and eventually properly patched upstream. We started to dtrace most large batch jobs or apps, and found lots of easy performance fixes. In house apps launching thousands of 'grep", "cut", etc.. with a LD_LIBRARY_PATH set to search over 50 directories OVER NFS. haha, libc's not there, not there.. ! ohhh there it is. Identifying stupid behavior in vendor apps for which we had no source. (but could complain about and sometimes get a fix) NIS groups (yeah Im old) slowing down NFS a lot, etc well, I guess I used it with profit at least once a week. You could probably do some of what DTrace does with other tools, but: it was safe in production, it had little overhead, and it was fun :-) Ah! I also used it to teach our interns a little about the OS internals. fbt probes with flowindent, watch them go "ahhhhhh that's what happens!". My current job is all about linux and I wish everyday for something half as good. So yeah, "it made my life better" is an understatement.

chris_wot 11y ago

You really aught to look at Bryan Cantrill's twitter feed then. It happens frequently enough that he notices blog posts and tweets them.

digi_owl 11y ago
Then again Cantrill has an axe to grind. In case others are wondering: https://en.wikipedia.org/wiki/Bryan_Cantrill https://en.wikipedia.org/wiki/Bryan_Cantrill
- chris_wot 11y ago
  I'm no fan of the man, but it doesn't mean that dtrace is not an excellent tool.

JonathonW 11y ago

Dtrace isn't completely dead; it's used to great effect as the underlying technology behind Instruments (performance analyzer/visualizer) on OSX. It never has really managed to catch on as a standalone tool by itself, though. Part of that's the licensing situation on Linux, but I'd hypothesize that part of it is also that it's a pretty complicated tool to use, interpret, and act on-- you don't hear a lot about SystemTap (same concept, GPLed for Linux-friendliness) these days, either.

seliopou 11y ago
I'd push back on the 'complicated' part. The dtrace language is heavily based on awk, which every modern *NIX ships with, and that every software engineer and sysadmin have at least a passing familiarity with. The main difference between the two is whereas awk is meant to operate over a stream of semi-structured lines, dtrace operates over a stream of probe firings. The high-level structures of the scripts, though, are the same: /pattern/ { statements } In a awk, when a line matches a the patter, the corresponding statements fire. In dtrace, when a probe matches a pattern, the corresponding statements fire. Integers, strings, arrays, and all those other sorts of features of the language are the same (though there are some special functions introduced by dtrace to help you collect and visualize aggregations). So from a language perspective, if you know awk, you mostly know dtrace. All that's left to do is learn the pattern language. If you want to do that effectively, you have to know not just the syntax of the patterns, but useful probes to listen for, and useful ways to use the information that's gleaned. This involves knowing things about operating systems. Most software engineers don't know much about operating systems so they think the tool is stupid.

postwait 11y ago

We use it every day at Circonus. One of us is learning something new about how our stuff works every single day.

bixu_ 11y ago

I use DTrace fairly often to look at the boundaries between whatever interpreted language my app's written in and the kernel. For example, at $job recently found that some of our Python code was doing 25k open64 calls but only 5k close calls during a scheduled job - this led us to find/fix a bug that would have been a disaster later.

kev009 11y ago

I think you've accidentally set up a straw man because mentions of low level tools don't correlate to use. In fact "trending" popularity is how a lot of bad technical decisions are made, cue crappy database flavor of the month. The hammer is a pretty useful tool but builders don't blog du jour about how elegant combining the strike face and nail puller on one implement are. None the less, millions are used every day. DTrace is used extensively in FreeBSD development these days. It's being used to bring up new generations of Systems Programmers: http://teachbsd.org/ http://teachbsd.org/ My team uses DTrace to develop TCP, which ensures open standards and a copyfree implementation that ends up in Apple products, Juniper, NetApp, and who knows where else. Of course, also Netflix and the large CDN I work for which run at least 1/3 of North American Internet volume. We also use DTrace to fix the reference copyfree Intel drivers, which may in a small way pay back Illumos for the awesome tech they share. I think those end up in some popular embedded microkernels too. I grin thinking about people that emphasize "GNU/Linux", but I will empathize with FSF for a moment to show that low level software is not particularly exciting to most. GCC and GDB are hugely influential tools for the whole open source ecosystem, especially the high growth during the 80s and 90s that led to "Enterprise" acceptance. It's hard to imagine where everyone else would be, including the BSDs and many chip companies, were it not for the GNU toolchain. Nonetheless, most people just begrudgingly use these and hope they don't give them any trouble because of course they are usually trying to do something else.

Bux10 11y ago

I use dtrace often, but am too busy to write articles about it, sorry. A couple of times, dtrace has even been THE tool that helped me understand exactly why our big production systems were dying seemingly inexplicable and horrible deaths. This has directly resulted in us still being in business instead of fading into irrelevance. That's plenty impact for me.

dap 11y ago

> Let me put it this way. In the last say five years, I can't think of a single article I've come across where someone describes DTrace making their life better (except one I wrote in 2011: http://blog.reverberate.org/2011/05/using-dtrace-on-os-x-to-... http://blog.reverberate.org/2011/05/using-dtrace-on-os-x-to-...), or a single time anyone I know or work with has even mentioned DTrace. Then you haven't been looking: http://dtrace.org/blogs/brendan/2012/08/09/10-performance-wins/ http://dtrace.org/blogs/brendan/2012/08/09/10-performance-wi... http://dtrace.org/blogs/brendan/2014/02/11/another-10-performance-wins/ http://dtrace.org/blogs/brendan/2014/02/11/another-10-perfor... https://www.joyent.com/blog/node-js-in-production-runtime-log-snooping https://www.joyent.com/blog/node-js-in-production-runtime-lo... https://twitter.com/b0rk/status/681536144362848257 https://twitter.com/b0rk/status/681536144362848257 https://speakerdeck.com/sartak/dtrace-war-stories https://speakerdeck.com/sartak/dtrace-war-stories https://www.bignerdranch.com/blog/hooked-on-dtrace-part-1/ https://www.bignerdranch.com/blog/hooked-on-dtrace-part-1/ https://medium.com/@jlouis666/erlang-dirty-scheduler-overhead-6e1219dcc7#.mppk9kx55 https://medium.com/@jlouis666/erlang-dirty-scheduler-overhea... Seriously, it wasn't that hard to find this stuff. I found the last one by searching HN: https://hn.algolia.com/?q=dtrace https://hn.algolia.com/?q=dtrace Like others who replied to you, I use DTrace every day. Very often, it solves in seconds what would have taken hours without it. Frequently, it's helped solve problems I'd never have been able to understand without it. My colleagues use it all the time as well. Sometimes, I write about the general techniques (I wrote the "runtime log snooping" blog post above), and most of our bug reports in recent years are public, but we don't write about individual bugs all that often. (Seems like you wouldn't have run into them if we had, though.)

auberlac 11y ago

I don't blog, and therefore have never written about the multiple times DTrace helped me and saved me a lot of time. It also made my job fun again as a sysadmin. Here are couple of examples I remember, from my days at a large financial institution. We're planning a samba upgrade on a server with ~700 clients. Non regression tests show the performance of copying large files has degraded by 30% Dtrace, aggregating on smbd stacks, helped us figure out the root cause in half an hour. This was reported and eventually properly patched upstream. We started to dtrace most large batch jobs or apps, and found lots of easy performance fixes. In house apps launching thousands of 'grep", "cut", etc.. with a LD_LIBRARY_PATH set to search over 50 directories OVER NFS. haha, libc's not there, not there.. ! ohhh there it is. Identifying stupid behavior in vendor apps for which we had no source. (but could complain about and sometimes get a fix) NIS groups (yeah Im old) slowing down NFS a lot, etc well, I guess I used it with profit at least once a week. You could probably do some of what DTrace does with other tools, but: it was safe in production, it had little overhead, and it was fun :-) Ah! I also used it to teach our interns a little about the OS internals. fbt probes with flowindent, watch them go "ahhhhhh that's what happens!". My current job is all about linux and I wish everyday for something half as good. So yeah, "it made my life better" is an understatement.