5 ms·

Debugging Emacs, or How I Learned to Stop Worrying and Love DTrace (2018)

258 points by lelf 8y ago

logitares 8y ago
hacker news is shadow banning ? how low can hn go ?
tedunangst 8y ago
But the pty bug still hasn't been found?
- josteink 8y ago
  That’s certainly how I read it.
masklinn 8y ago
> Naturally I wanted to get my hands on this stuff to try it out for myself. Is it really as good as they say? Normally I stick to Linux, but it (generally) doesn’t have these Sun technologies. Since then, bpftrace[0] has been released, supposedly it's actually better than dtrace, in the sense that it's even more capable[1]. See https://www.joyfulbikeshedding.com/blog/2019-01-31-full-system-dynamic-tracing-on-linux-using-ebpf-and-bpftrace.html https://www.joyfulbikeshedding.com/blog/2019-01-31-full-syst... for a recent basic ebpf/bpftrace tutorial. Brendan Gregg also has a primer on eBPF: http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-traci... [0] http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux... [1] that's Brendan Gregg saying it: http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux... — having contributed to it and being one of the leading dtrace experts
- lelf 8y ago
  > * in the sense that it's even more capable* Examples? I don't see how it's more capable. Catching up with dtrace, maybe.
  - Birch-san 8y ago
    bpftrace's potential is higher than DTrace's, but at the moment it's definitely not reached parity. For example: we cannot yet read strings bigger than about 240 bytes (any bigger won't fit on the BPF stack).
- rurban 8y ago
  The extra capabilities Linux added made it insecure (arrays), so that eBPF needs to be disabled in production SMP systems. There's also no existing userland support. Everyone provides DTrace probes, no one systemtap probes.
  - deivid 8y ago
    You can trace dtrace probes with ebpf.. see https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md#7-usdt-static-tracing-user-level https://github.com/iovisor/bpftrace/blob/master/docs/referen...
  - Birch-san 8y ago
    BPF is compatible with DTrace probes. The kernel verifier does extensive bounds checks on array uses and jumps in BPF programs.
    - rurban 8y ago
      Those checks don't save you from spectre-like cache exploitation. DTrace was designed with maps-only for a reason. Thanksfully there's now Oracle Linux with DTrace-proper. When the usdt syntax can use static DTrace probes, why isn't it documented then? and why doesn't it use the familiar DTrace syntax then? Why is nobody using it?
      - dan-robertson 8y ago
        > DTrace was designed with maps-only for a reason. You write this like you are implying that the DTrace designers somehow foresaw that spectre would be a thing and therefore array accesses would be a risk. This is obviously false. I’m sure there are reasonable reasons to not have arrays but predicting spectre 10 years before it was found is not one of them.
        rurban 8y ago
        Ask Brian, he did it on purpose.
ink_13 8y ago
> Linux has a reputation for Not Invented Here (NIH) syndrome, and these licensing issues certainly contribute to that I don't think the author understands with NIH means in practise, and this completely misplaces the blame. It's not as though the kernel team was completely unwilling to consider porting ZFS or DTrace, it's that for legal reasons they were unable.
- lelf 8y ago
  No (unfortunately) it's NIH and nothing more. https://youtu.be/TgmA48fILq8?t=4266 https://youtu.be/TgmA48fILq8?t=4266 https://marc.info/?l=linux-kernel&m=154714516832389&w=2 https://marc.info/?l=linux-kernel&m=154714516832389&w=2
  - simcop2387 8y ago
    You've also missed what was said on the mailing list message. Sun specifically made a new license to make it legally impossible to include DTrace and ZFS (and other things) into a GPL licensed project. That's a huge difference to what NIH means, which is that just because I didn't make it, I'm not going to use it, I'll make my own version instead. They couldn't ever use the original version of things because they're legally bound to not do so. The kernel has had a long standing policy of not keeping APIs that either they or userspace aren't using to reduce their burden and the case of keeping that API just for ZFS since literally nothing else in the kernel was using it wasn't a good enough argument to keep it. ZFS on Linux has now worked around the removal of the API.
    - pmoriarty 8y ago
      "ZFS on Linux has now worked around the removal of the API." I'm just curious: how did they do that?
      - Arnavion 8y ago
        https://github.com/zfsonlinux/zfs/commit/0c593296e98e3ac95f14704e4cee6cedb2134990 https://github.com/zfsonlinux/zfs/commit/0c593296e98e3ac95f1... + https://github.com/zfsonlinux/zfs/commit/26a856594f731db62446cf537659f9360261fe97 https://github.com/zfsonlinux/zfs/commit/26a856594f731db6244...
drewg123 8y ago
I'm a big fan of dtrace and I use it daily. However, the system call tracing that the author did here could easily have been done without dtrace. Eg, via strace on the curl process (or truss, ktrace, or whatever your OSe's system call tracer is).
- mjw1007 8y ago
  The author evidently didn't find that option attractive: « To run curl under strace from Emacs, I’d have to modify Emacs’ behavior to do so. With DTrace I can instrument every curl process without making a single change to Emacs, and with negligible impact to Emacs. That’s a big deal. »
  - greglindahl 8y ago
    The usual trick to use strace in that kind of situation is to write a script named "curl" that runs curl under strace, and stick the script in your path. If the author is familiar with DTrace, though, sounds like that's a great solution for them.
    - __david__ 8y ago
      I don't see why "strace -f" wouldn't suffice. I use that all the time for exactly these kinds of things.
      - ithkuil 8y ago
        Perhaps because you don't want to strace the whole emacs just to hit the curl process that gets forked off emacs.
  - dima55 8y ago
    Linux now has a number of tools that can do this just fine. strace was the only tool LOONG ago. For the issue described in the article sysdig would be the tool I'd use, but that's just one option.
agumonkey 8y ago
Once again, a lovely article by Chris Wellons. Fun to see him having idols :) ps: the hammer story linked at the end is worth the minute reading http://archive.is/OV1kQ http://archive.is/OV1kQ
- nlefler 8y ago
  Seems likely the origin of the parable is Charles Proteus Steinmetz: https://www.smithsonianmag.com/history/charles-proteus-steinmetz-the-wizard-of-schenectady-51912022/ https://www.smithsonianmag.com/history/charles-proteus-stein... From the article: """ Ford, whose electrical engineers couldn’t solve some problems they were having with a gigantic generator, called Steinmetz in to the plant. Upon arriving, Steinmetz rejected all assistance and asked only for a notebook, pencil and cot. According to Scott, Steinmetz listened to the generator and scribbled computations on the notepad for two straight days and nights. On the second night, he asked for a ladder, climbed up the generator and made a chalk mark on its side. Then he told Ford’s skeptical engineers to remove a plate at the mark and replace sixteen windings from the field coil. They did, and the generator performed to perfection. Henry Ford was thrilled until he got an invoice from General Electric in the amount of $10,000. Ford acknowledged Steinmetz’s success but balked at the figure. He asked for an itemized bill. Steinmetz, Scott wrote, responded personally to Ford’s request with the following: Making chalk mark on generator $1. Knowing where to make mark $9,999. """
m0zg 8y ago
While we're on the topic of tracing, does anyone know an _easy to use_ (!) tool that would roughly mirror the functionality of Apple Instruments for user-space tracking? I know there's lttng-ust and TraceCompass, but using either (and especially TraceCompas) is worse than pulling teeth. To display a simple trace in a "piano roll" / Gantt chart view with TraceCompass one has to read an encyclopedia sized tutorial and fumble their way through obscure XML (!) files to create their own visualizations. Let's just say, ease of use did not seem to be the goal at all.
teacpde 8y ago
For people like me who are not familiar with what line buffering is, I found a good short explanation here https://www.gnu.org/software/libc/manual/html_node/Buffering-Concepts.html https://www.gnu.org/software/libc/manual/html_node/Buffering...

258 points by lelf 8y ago

logitares 8y ago

hacker news is shadow banning ? how low can hn go ?

tedunangst 8y ago

But the pty bug still hasn't been found?

josteink 8y ago
That’s certainly how I read it.

masklinn 8y ago

> Naturally I wanted to get my hands on this stuff to try it out for myself. Is it really as good as they say? Normally I stick to Linux, but it (generally) doesn’t have these Sun technologies. Since then, bpftrace[0] has been released, supposedly it's actually better than dtrace, in the sense that it's even more capable[1]. See https://www.joyfulbikeshedding.com/blog/2019-01-31-full-system-dynamic-tracing-on-linux-using-ebpf-and-bpftrace.html https://www.joyfulbikeshedding.com/blog/2019-01-31-full-syst... for a recent basic ebpf/bpftrace tutorial. Brendan Gregg also has a primer on eBPF: http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-traci... [0] http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux... [1] that's Brendan Gregg saying it: http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux... — having contributed to it and being one of the leading dtrace experts

lelf 8y ago
> * in the sense that it's even more capable* Examples? I don't see how it's more capable. Catching up with dtrace, maybe.
- Birch-san 8y ago
  bpftrace's potential is higher than DTrace's, but at the moment it's definitely not reached parity. For example: we cannot yet read strings bigger than about 240 bytes (any bigger won't fit on the BPF stack).
rurban 8y ago
The extra capabilities Linux added made it insecure (arrays), so that eBPF needs to be disabled in production SMP systems. There's also no existing userland support. Everyone provides DTrace probes, no one systemtap probes.
- deivid 8y ago
  You can trace dtrace probes with ebpf.. see https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md#7-usdt-static-tracing-user-level https://github.com/iovisor/bpftrace/blob/master/docs/referen...
- Birch-san 8y ago
  BPF is compatible with DTrace probes. The kernel verifier does extensive bounds checks on array uses and jumps in BPF programs.
  - rurban 8y ago
    Those checks don't save you from spectre-like cache exploitation. DTrace was designed with maps-only for a reason. Thanksfully there's now Oracle Linux with DTrace-proper. When the usdt syntax can use static DTrace probes, why isn't it documented then? and why doesn't it use the familiar DTrace syntax then? Why is nobody using it?
    - dan-robertson 8y ago
      > DTrace was designed with maps-only for a reason. You write this like you are implying that the DTrace designers somehow foresaw that spectre would be a thing and therefore array accesses would be a risk. This is obviously false. I’m sure there are reasonable reasons to not have arrays but predicting spectre 10 years before it was found is not one of them.
      - rurban 8y ago
        Ask Brian, he did it on purpose.

ink_13 8y ago

> Linux has a reputation for Not Invented Here (NIH) syndrome, and these licensing issues certainly contribute to that I don't think the author understands with NIH means in practise, and this completely misplaces the blame. It's not as though the kernel team was completely unwilling to consider porting ZFS or DTrace, it's that for legal reasons they were unable.

lelf 8y ago
No (unfortunately) it's NIH and nothing more. https://youtu.be/TgmA48fILq8?t=4266 https://youtu.be/TgmA48fILq8?t=4266 https://marc.info/?l=linux-kernel&m=154714516832389&w=2 https://marc.info/?l=linux-kernel&m=154714516832389&w=2
- simcop2387 8y ago
  You've also missed what was said on the mailing list message. Sun specifically made a new license to make it legally impossible to include DTrace and ZFS (and other things) into a GPL licensed project. That's a huge difference to what NIH means, which is that just because I didn't make it, I'm not going to use it, I'll make my own version instead. They couldn't ever use the original version of things because they're legally bound to not do so. The kernel has had a long standing policy of not keeping APIs that either they or userspace aren't using to reduce their burden and the case of keeping that API just for ZFS since literally nothing else in the kernel was using it wasn't a good enough argument to keep it. ZFS on Linux has now worked around the removal of the API.
  - pmoriarty 8y ago
    "ZFS on Linux has now worked around the removal of the API." I'm just curious: how did they do that?
    - Arnavion 8y ago
      https://github.com/zfsonlinux/zfs/commit/0c593296e98e3ac95f14704e4cee6cedb2134990 https://github.com/zfsonlinux/zfs/commit/0c593296e98e3ac95f1... + https://github.com/zfsonlinux/zfs/commit/26a856594f731db62446cf537659f9360261fe97 https://github.com/zfsonlinux/zfs/commit/26a856594f731db6244...

drewg123 8y ago

I'm a big fan of dtrace and I use it daily. However, the system call tracing that the author did here could easily have been done without dtrace. Eg, via strace on the curl process (or truss, ktrace, or whatever your OSe's system call tracer is).

mjw1007 8y ago
The author evidently didn't find that option attractive: « To run curl under strace from Emacs, I’d have to modify Emacs’ behavior to do so. With DTrace I can instrument every curl process without making a single change to Emacs, and with negligible impact to Emacs. That’s a big deal. »
- greglindahl 8y ago
  The usual trick to use strace in that kind of situation is to write a script named "curl" that runs curl under strace, and stick the script in your path. If the author is familiar with DTrace, though, sounds like that's a great solution for them.
  - __david__ 8y ago
    I don't see why "strace -f" wouldn't suffice. I use that all the time for exactly these kinds of things.
    - ithkuil 8y ago
      Perhaps because you don't want to strace the whole emacs just to hit the curl process that gets forked off emacs.
- dima55 8y ago
  Linux now has a number of tools that can do this just fine. strace was the only tool LOONG ago. For the issue described in the article sysdig would be the tool I'd use, but that's just one option.

agumonkey 8y ago

Once again, a lovely article by Chris Wellons. Fun to see him having idols :) ps: the hammer story linked at the end is worth the minute reading http://archive.is/OV1kQ http://archive.is/OV1kQ

nlefler 8y ago
Seems likely the origin of the parable is Charles Proteus Steinmetz: https://www.smithsonianmag.com/history/charles-proteus-steinmetz-the-wizard-of-schenectady-51912022/ https://www.smithsonianmag.com/history/charles-proteus-stein... From the article: """ Ford, whose electrical engineers couldn’t solve some problems they were having with a gigantic generator, called Steinmetz in to the plant. Upon arriving, Steinmetz rejected all assistance and asked only for a notebook, pencil and cot. According to Scott, Steinmetz listened to the generator and scribbled computations on the notepad for two straight days and nights. On the second night, he asked for a ladder, climbed up the generator and made a chalk mark on its side. Then he told Ford’s skeptical engineers to remove a plate at the mark and replace sixteen windings from the field coil. They did, and the generator performed to perfection. Henry Ford was thrilled until he got an invoice from General Electric in the amount of $10,000. Ford acknowledged Steinmetz’s success but balked at the figure. He asked for an itemized bill. Steinmetz, Scott wrote, responded personally to Ford’s request with the following: Making chalk mark on generator $1. Knowing where to make mark $9,999. """

m0zg 8y ago

While we're on the topic of tracing, does anyone know an _easy to use_ (!) tool that would roughly mirror the functionality of Apple Instruments for user-space tracking? I know there's lttng-ust and TraceCompass, but using either (and especially TraceCompas) is worse than pulling teeth. To display a simple trace in a "piano roll" / Gantt chart view with TraceCompass one has to read an encyclopedia sized tutorial and fumble their way through obscure XML (!) files to create their own visualizations. Let's just say, ease of use did not seem to be the goal at all.

teacpde 8y ago

For people like me who are not familiar with what line buffering is, I found a good short explanation here https://www.gnu.org/software/libc/manual/html_node/Buffering-Concepts.html https://www.gnu.org/software/libc/manual/html_node/Buffering...