4 ms·

TFA actually gives an example where DWARF alone doesn't work.

Dylan16807 4y ago
It doesn't give any insight as to why, though, which makes it very weak evidence toward whether DWARF is the real problem.
- loeg 4y ago
  It's not so much DWARF is the problem as it is that the information is just lost in omit-framepointer mode. You can't reconstruct it from nothing. There's a related problem of compilers generating imperfect debuginfo that fails to reconstruct local variables from registers/memory due to an incomplete DWARF representation, but I don't think that plays a roll in unwinding. (But maybe something similar does.)
  - Dylan16807 4y ago
    What information is lost? The call stack can't be lost or you'd crash immediately.
    - saagarjha 4y ago
      It's lost in the translation to DWARF. Of course the program knows how to unwind its own stack correctly, but explaining how to do this for arbitrary programs is untenable without DWARF to help, and it's annoying to parse and compilers don't emit it correctly.
      - Dylan16807 4y ago
        If the information is lost by DWARF then I'd say that DWARF is clearly the problem. Unless DWARF can handle it fine and compilers are thoroughly broken, in which case it's a tossup between "DWARF is fundamentally flawed" and "compiler designers are lazy for some unknown reason"?
        saagarjha 4y ago
        DWARF can handle it fine, it’s that compilers don’t generate correct information or the tools to parse DWARF don’t do it properly.
- xerxes901 4y ago
  The problem is that with perf the stack trace is collected by the kernel (when the interrupt from the performance counters fires), and the kernel doesn’t implement dwarf unwinding of user space code. It _could_, but the kernel folks have made it pretty clear they don’t want to put any code related to dwarf in the kernel.
  - deleted 4y ago
    [deleted]
  - jeltz 4y ago
    There are tools which do dwarf unwinding with eBPF but I have not used them myself.
wahern 4y ago
They give an example of where perf was unable to decipher the stack using DWARF; it only begs the question of whether the issue is perf or DWARF. It does show that frame pointers make things easier if all you care about are named call traces, but nobody disputes that. (It could also be the case that enabling frame pointers had the side effect of preventing certain optimizations. I wouldn't be surprised if flame graph generators use frame pointers (e.g. libunwind) preferentially, and using DWARF on that same binary would have resulted in the same graph.) DWARF is useful for so much more than flame graphs. Flame graphs are cool and useful, but they're over hyped. Anybody who has ever spent any significant portion of their career debugging compiled binaries knows that proper debug information is infinitely more useful than flame graphs could ever be. Flame graphs are also useful... very useful. But that's my point: if they advocated even half as much for ensuring everybody made debug info more readily accessible by default as they did crying for frame pointers, the world would be a much better place. Is DWARF too complex? There are many layers to DWARF, and there are compromises everybody could make so we can get at an optimal balance for which DWARF info we should ensure is always there. And we could also spend more time fixing tooling so it works properly with that minimal debug data.
- kouteiheika 4y ago
  > it only begs the question of whether the issue is perf or DWARF AFAIK there are two issues at play here. 1. There's a bug in the Linux kernel where it sometimes return an invalid RBP register through `perf_event_open`. 2. The compiler doesn't generate the necessary DWARF info to support unwinding asynchronously, so depending on where exactly you land in a function you might not be able to unwind the stack. (This is why unwinding from *within* the program always works correctly but unwinding from *outside* the program with perf doesn't.) Source: I wrote my own sampling profiler.
  - ShroudedNight 4y ago
    I would be interested in taking / motivating a deeper dive into the failure modes of -fasychronous-unwind-tables, would you happen to recall a scenario where the undwinding problems you discuss reproduce ~consistentlyish?

It doesn't give any insight as to why, though, which makes it very weak evidence toward whether DWARF is the real problem.

loeg 4y ago
It's not so much DWARF is the problem as it is that the information is just lost in omit-framepointer mode. You can't reconstruct it from nothing. There's a related problem of compilers generating imperfect debuginfo that fails to reconstruct local variables from registers/memory due to an incomplete DWARF representation, but I don't think that plays a roll in unwinding. (But maybe something similar does.)
- Dylan16807 4y ago
  What information is lost? The call stack can't be lost or you'd crash immediately.
  - saagarjha 4y ago
    It's lost in the translation to DWARF. Of course the program knows how to unwind its own stack correctly, but explaining how to do this for arbitrary programs is untenable without DWARF to help, and it's annoying to parse and compilers don't emit it correctly.
    - Dylan16807 4y ago
      If the information is lost by DWARF then I'd say that DWARF is clearly the problem. Unless DWARF can handle it fine and compilers are thoroughly broken, in which case it's a tossup between "DWARF is fundamentally flawed" and "compiler designers are lazy for some unknown reason"?
      - saagarjha 4y ago
        DWARF can handle it fine, it’s that compilers don’t generate correct information or the tools to parse DWARF don’t do it properly.
xerxes901 4y ago
The problem is that with perf the stack trace is collected by the kernel (when the interrupt from the performance counters fires), and the kernel doesn’t implement dwarf unwinding of user space code. It _could_, but the kernel folks have made it pretty clear they don’t want to put any code related to dwarf in the kernel.
- deleted 4y ago
  [deleted]
- jeltz 4y ago
  There are tools which do dwarf unwinding with eBPF but I have not used them myself.

wahern 4y ago

They give an example of where perf was unable to decipher the stack using DWARF; it only begs the question of whether the issue is perf or DWARF. It does show that frame pointers make things easier if all you care about are named call traces, but nobody disputes that. (It could also be the case that enabling frame pointers had the side effect of preventing certain optimizations. I wouldn't be surprised if flame graph generators use frame pointers (e.g. libunwind) preferentially, and using DWARF on that same binary would have resulted in the same graph.) DWARF is useful for so much more than flame graphs. Flame graphs are cool and useful, but they're over hyped. Anybody who has ever spent any significant portion of their career debugging compiled binaries knows that proper debug information is infinitely more useful than flame graphs could ever be. Flame graphs are also useful... very useful. But that's my point: if they advocated even half as much for ensuring everybody made debug info more readily accessible by default as they did crying for frame pointers, the world would be a much better place. Is DWARF too complex? There are many layers to DWARF, and there are compromises everybody could make so we can get at an optimal balance for which DWARF info we should ensure is always there. And we could also spend more time fixing tooling so it works properly with that minimal debug data.

kouteiheika 4y ago
> it only begs the question of whether the issue is perf or DWARF AFAIK there are two issues at play here. 1. There's a bug in the Linux kernel where it sometimes return an invalid RBP register through `perf_event_open`. 2. The compiler doesn't generate the necessary DWARF info to support unwinding asynchronously, so depending on where exactly you land in a function you might not be able to unwind the stack. (This is why unwinding from *within* the program always works correctly but unwinding from *outside* the program with perf doesn't.) Source: I wrote my own sampling profiler.
- ShroudedNight 4y ago
  I would be interested in taking / motivating a deeper dive into the failure modes of -fasychronous-unwind-tables, would you happen to recall a scenario where the undwinding problems you discuss reproduce ~consistentlyish?