3 ms·
> Frame pointers have some corner cases which they don’t handle well (certain leaf and most inlined functions aren’t collected), but these don’t matter a great
by ot 4y ago
> Frame pointers have some corner cases which they don’t handle well (certain leaf and most inlined functions aren’t collected), but these don’t matter a great deal in reality.
> DWARF unwinding can show inlined functions as if they are separate stack frames. (Opinions differ as to whether or not this is an advantage.)
This conflates unwinding and symbolization. Unwinding collects the list of frames, which by definition cannot have inlined functions (functions that don't have their own frame); the unwinding mechanism does not matter here.
DWARF can be used to resolve the "stack" of inlined functions for an instruction address, even if that was collected via frame pointers. That can be done in post-processing, possibly on a different machine, so the cost does not affect the workload being profiled.
For example, using addr2line from the LLVM distribution
$ llvm-addr2line -pfi --demangle -e /path/to/unstripped/binary
# Enter instruction address in the form 0x...
# Outputs inlined stack
To summarize, unwinding via frame pointers does not miss any information that would be collected with DWARF unwinding. Everything can be recovered later at symbolization time.
And on whether reporting inlined functions is important or not, at least for C++, which heavily relies on inlining to mitigate abstraction penalty, I'd say it is crucial.
- the_mitsuhiko 4y ago> To summarize, unwinding via frame pointers does not miss any information that would be collected with DWARF unwinding. Everything can be recovered later at symbolization time. The real issue with DWARF based unwinding/symbolication is that it's really complex on the user experience around it. We at Sentry support stack walking from minidumps, yet we often cannot unwind on Linux platforms on the server because executables and object files are typically not available. Despite us supporting debuginfod (we're hitting the canonical service today only) we are unable to collect executables/object files from there, making it completely impossible to produce proper stack traces if frame pointers are omitted. In a world were DWARF unwinding is a thing and people want to use, there has to be an ecosystem of sharing binaries too. This issue is particular bad on Android where there are millions of devices out there with tons of different proprietary system libraries linked in, destroying stacks.
- conconpon 4y agoCouldn't this be solved by uploading the binaries along with the crash dumps, if you don't already have a copy of it, as determined by checking hashes or something?
- the_mitsuhiko 4y agoExactly, except debuginfod from Canonical and some others misses the executables.
- lathiat 4y agoI would expect them to be there but debuginfod for Ubuntu is very new and did not (at least currently) import older releases (including the current LTS) or older package versions. Which is maybe what you’re seeing. Should work better with the most recent release only. Do you have a specific example of which executables are missing? There are also other ways to get all the debug symbols but it’s much fiddlier and not nearly as nice as debuginfod. Data’s all there, finding it is harder. You can query the apt index for buildids but only for the currently released package version and not superseded ones. There were discussions about importing all the history and a way to do it was determined but not sure where the implementation of that has gotten too.
- account42 4y agoThe debug info is usually stripped out of the binaries for Linux distros and has to be installed separately. debuginfod is supposed to make possible to download the debug info in a distro-agnostic way based on the build ID embedded in the binaries.