3 ms·

I somewhat painstakingly figured this out the hard way pulling core dumps off of embedded linux devices that gdb had a hard time working with. At the time I was

by ghotli 3y ago

I somewhat painstakingly figured this out the hard way pulling core dumps off of embedded linux devices that gdb had a hard time working with. At the time I was like whew why is omitting frame pointers the default at all in so many places when it didn't seem to make a measurable difference in the performance of the software. I guess it's just vestigial these days and yes please use a compiler flag like this when it measurably makes sense. Making debugging simpler for the rest of us is the way to go.

o11c 3y ago
Something has to be seriously wrong with the way it was compiled for gdb to have trouble. Debugging (or in-process exception dumps), which only does a reasonable number of backtraces, should always be able to use separate unwind data sections. Including frame pointers should only have a performance effect for sample-based profiling, which does a very large number of backtraces. And the general fact is - people don't profile, and if they do, they don't do it correctly. Omitting frame pointers has significant performance wins on platforms with about 6 registers, like 32-bit intel x86. It's much less of a win on platforms with about 14 registers, like 64-bit x86 or 32-bit ARM, let alone platforms with about 30 registers, like 64-bit ARM. Since modern architectures are strongly trending toward designs that support more registers, a frame pointer isn't unreasonable to choose. But that's still no excuse for all the shitty software that refuses to work correctly without them, rather than merely more slowly. (Note that theoretically it is possible to design an ISA/ABI combo that supports easy and fast unwinding even without frame pointers, but there's always going to be some overhead and to my knowledge this choice hasn't been done.)
- brancz 3y ago
  Founder of Polar Signals here, the profiling product that's mentioned in the blog post. We've already made it possible to unwind without frame pointers relatively cheaply (<1% overhead), and have written about this extensively [1]. That said, no matter how we spin it, frame pointer unwinding is always going to be cheaper, and while profiling is getting better, I think I'm almost more excited about the other aspects of debuggability this is gaining: out-of-the-box working bpftrace, bcc-tool and anything else that needs to deal with unwinding with just about anything that's running on the box. I think we'll see a huge gain in capabilities over the next few years with frame pointers more prevalent in fedora and ubuntu and I'm sure more will now follow. [1] https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-based-stack-walking-using-ebpf https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas...
- stefan_ 3y ago
  Embedded devices, certainly in the days of 16 MiB NOR flash, do not contain unwind or debug information. Even today OpenWrt and similar will routinely strip all binaries installed to the final firmware image. There are some structural issues in the Linux world, too; the default of debug data contained within the binary is often undesirable, symbol servers (they finally learned about those in Ubuntu 22) require extra setup & tooling support that isn't often invested in, widely used libraries like libunwind are both arcane and terrible (yes, an instruction pointer of 0 will not have associated unwind information; use your brain and realize someone called a NULL function pointer).
- ghotli 3y ago
  > Something has to be seriously wrong with the way it was compiled for gdb to have trouble. Cargo culting culture in embedded devices of just using some old toolchain copy pasted from some vendor seems to be pervasive. I cut fresh compilers and align their output with the old crusty toolchains. Made entire classes of issues go away. Regardless, agreed the omission of the frame pointer was merely one issue at play with those particular core dumps on those particular devices, years back at this point. :)
deleted 3y ago
[deleted]

by ghotli 3y ago

o11c 3y ago

Something has to be seriously wrong with the way it was compiled for gdb to have trouble. Debugging (or in-process exception dumps), which only does a reasonable number of backtraces, should always be able to use separate unwind data sections. Including frame pointers should only have a performance effect for sample-based profiling, which does a very large number of backtraces. And the general fact is - people don't profile, and if they do, they don't do it correctly. Omitting frame pointers has significant performance wins on platforms with about 6 registers, like 32-bit intel x86. It's much less of a win on platforms with about 14 registers, like 64-bit x86 or 32-bit ARM, let alone platforms with about 30 registers, like 64-bit ARM. Since modern architectures are strongly trending toward designs that support more registers, a frame pointer isn't unreasonable to choose. But that's still no excuse for all the shitty software that refuses to work correctly without them, rather than merely more slowly. (Note that theoretically it is possible to design an ISA/ABI combo that supports easy and fast unwinding even without frame pointers, but there's always going to be some overhead and to my knowledge this choice hasn't been done.)

brancz 3y ago
Founder of Polar Signals here, the profiling product that's mentioned in the blog post. We've already made it possible to unwind without frame pointers relatively cheaply (<1% overhead), and have written about this extensively [1]. That said, no matter how we spin it, frame pointer unwinding is always going to be cheaper, and while profiling is getting better, I think I'm almost more excited about the other aspects of debuggability this is gaining: out-of-the-box working bpftrace, bcc-tool and anything else that needs to deal with unwinding with just about anything that's running on the box. I think we'll see a huge gain in capabilities over the next few years with frame pointers more prevalent in fedora and ubuntu and I'm sure more will now follow. [1] https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-based-stack-walking-using-ebpf https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas...
stefan_ 3y ago
Embedded devices, certainly in the days of 16 MiB NOR flash, do not contain unwind or debug information. Even today OpenWrt and similar will routinely strip all binaries installed to the final firmware image. There are some structural issues in the Linux world, too; the default of debug data contained within the binary is often undesirable, symbol servers (they finally learned about those in Ubuntu 22) require extra setup & tooling support that isn't often invested in, widely used libraries like libunwind are both arcane and terrible (yes, an instruction pointer of 0 will not have associated unwind information; use your brain and realize someone called a NULL function pointer).
ghotli 3y ago
> Something has to be seriously wrong with the way it was compiled for gdb to have trouble. Cargo culting culture in embedded devices of just using some old toolchain copy pasted from some vendor seems to be pervasive. I cut fresh compilers and align their output with the old crusty toolchains. Made entire classes of issues go away. Regardless, agreed the omission of the frame pointer was merely one issue at play with those particular core dumps on those particular devices, years back at this point. :)

deleted 3y ago

[deleted]