5 ms·

They give an example of where perf was unable to decipher the stack using DWARF; it only begs the question of whether the issue is perf or DWARF. It does show t

by wahern 4y ago

They give an example of where perf was unable to decipher the stack using DWARF; it only begs the question of whether the issue is perf or DWARF. It does show that frame pointers make things easier if all you care about are named call traces, but nobody disputes that. (It could also be the case that enabling frame pointers had the side effect of preventing certain optimizations. I wouldn't be surprised if flame graph generators use frame pointers (e.g. libunwind) preferentially, and using DWARF on that same binary would have resulted in the same graph.) DWARF is useful for so much more than flame graphs. Flame graphs are cool and useful, but they're over hyped. Anybody who has ever spent any significant portion of their career debugging compiled binaries knows that proper debug information is infinitely more useful than flame graphs could ever be. Flame graphs are also useful... very useful. But that's my point: if they advocated even half as much for ensuring everybody made debug info more readily accessible by default as they did crying for frame pointers, the world would be a much better place. Is DWARF too complex? There are many layers to DWARF, and there are compromises everybody could make so we can get at an optimal balance for which DWARF info we should ensure is always there. And we could also spend more time fixing tooling so it works properly with that minimal debug data.

kouteiheika 4y ago
> it only begs the question of whether the issue is perf or DWARF AFAIK there are two issues at play here. 1. There's a bug in the Linux kernel where it sometimes return an invalid RBP register through `perf_event_open`. 2. The compiler doesn't generate the necessary DWARF info to support unwinding asynchronously, so depending on where exactly you land in a function you might not be able to unwind the stack. (This is why unwinding from *within* the program always works correctly but unwinding from *outside* the program with perf doesn't.) Source: I wrote my own sampling profiler.
- ShroudedNight 4y ago
  I would be interested in taking / motivating a deeper dive into the failure modes of -fasychronous-unwind-tables, would you happen to recall a scenario where the undwinding problems you discuss reproduce ~consistentlyish?
  - AshamedCaptain 4y ago
    asynchronous-unwind-tables is not the default. Ever since I enabled it, I have never seen gdb fail to unwind.
  - kouteiheika 4y ago
    It was quite a long time ago, so unfortunately at this point I don't remember what exact problems I encountered with `-fasychronous-unwind-tables`. (I think it might have been completely broken and did nothing on 32-bit ARM? But I might be misremembering.) And there's of course the issue that you'd need everything to be compiled with it (because the program you're profiling is going to call into external dynamically linked libraries), which on an average system you most likely won't have.
anarazel 4y ago
I suspect the dwarf issue from the post is just missing debug info for some library, or too deep stacks. But leaving that aside, for profiling, dwarf is just very expensive (perf, where the data volume explodes, due to copying stacks) or not available (bpf stacks), because the unwinding has to happen in the kernel. Even if the kernel had a dwarf unwinder, it's vastly more expensive to do that, compared to unwinding via frame pointers. Try running a system wide profile on larger and busy box. Dwarf based call graphs are basically not usable. The profile quickly is ginormous, and viewing profiles is extremely slow.
saagarjha 4y ago
> DWARF is useful for so much more than flame graphs. Flame graphs are cool and useful, but they're over hyped. Anybody who has ever spent any significant portion of their career debugging compiled binaries knows that proper debug information is infinitely more useful than flame graphs could ever be. Flame graphs are also useful... very useful. But that's my point: if they advocated even half as much for ensuring everybody made debug info more readily accessible by default as they did crying for frame pointers, the world would be a much better place. I want both really badly but the fact is that flamegraphs drive performance optimizations which save companies millions of dollars, while symbols sometimes fix bugs and sometimes get blamed for making their software easy to reverse engineer. You can understand why one gets more focus than the other.
- Annatar 4y ago
  [dead]

by wahern 4y ago

kouteiheika 4y ago

> it only begs the question of whether the issue is perf or DWARF AFAIK there are two issues at play here. 1. There's a bug in the Linux kernel where it sometimes return an invalid RBP register through `perf_event_open`. 2. The compiler doesn't generate the necessary DWARF info to support unwinding asynchronously, so depending on where exactly you land in a function you might not be able to unwind the stack. (This is why unwinding from *within* the program always works correctly but unwinding from *outside* the program with perf doesn't.) Source: I wrote my own sampling profiler.

ShroudedNight 4y ago
I would be interested in taking / motivating a deeper dive into the failure modes of -fasychronous-unwind-tables, would you happen to recall a scenario where the undwinding problems you discuss reproduce ~consistentlyish?
- AshamedCaptain 4y ago
  asynchronous-unwind-tables is not the default. Ever since I enabled it, I have never seen gdb fail to unwind.
- kouteiheika 4y ago
  It was quite a long time ago, so unfortunately at this point I don't remember what exact problems I encountered with `-fasychronous-unwind-tables`. (I think it might have been completely broken and did nothing on 32-bit ARM? But I might be misremembering.) And there's of course the issue that you'd need everything to be compiled with it (because the program you're profiling is going to call into external dynamically linked libraries), which on an average system you most likely won't have.

anarazel 4y ago

I suspect the dwarf issue from the post is just missing debug info for some library, or too deep stacks. But leaving that aside, for profiling, dwarf is just very expensive (perf, where the data volume explodes, due to copying stacks) or not available (bpf stacks), because the unwinding has to happen in the kernel. Even if the kernel had a dwarf unwinder, it's vastly more expensive to do that, compared to unwinding via frame pointers. Try running a system wide profile on larger and busy box. Dwarf based call graphs are basically not usable. The profile quickly is ginormous, and viewing profiles is extremely slow.

saagarjha 4y ago

> DWARF is useful for so much more than flame graphs. Flame graphs are cool and useful, but they're over hyped. Anybody who has ever spent any significant portion of their career debugging compiled binaries knows that proper debug information is infinitely more useful than flame graphs could ever be. Flame graphs are also useful... very useful. But that's my point: if they advocated even half as much for ensuring everybody made debug info more readily accessible by default as they did crying for frame pointers, the world would be a much better place. I want both really badly but the fact is that flamegraphs drive performance optimizations which save companies millions of dollars, while symbols sometimes fix bugs and sometimes get blamed for making their software easy to reverse engineer. You can understand why one gets more focus than the other.

Annatar 4y ago
[dead]