4 ms·
Unwinding the Stack the Hard Way
- rwmj 4y agoI would suggest not omitting the frame pointer. Fedora recently changed the default and it makes collecting stack traces vastly simpler, leading to better profiling support (https://rwmj.wordpress.com/2023/02/14/frame-pointers-vs-dwarf-my-verdict/ https://rwmj.wordpress.com/2023/02/14/frame-pointers-vs-dwar...) Since then I gave a short (15 min) talk about producing and understanding flame graphs: http://oirase.annexia.org/tmp/2023-03-08-flamegraphs.mp4 http://oirase.annexia.org/tmp/2023-03-08-flamegraphs.mp4
- boulos 4y agoYeah, mandating rbp on amd64 would have saved years of headache. There are still cases like tail calls that wouldn't work, but the vast majority of code doesn't need to care about dedicating a register to having usable backtraces.
- fweimer 4y agoI still hope we can revert that once we have complete, CPU-verified backtraces. As a linked list, frame pointers are still somewhat slow. Just copying the hardware address stack should be quite a bit faster, and more importantly, the CPU will enforce that the addresses are correct.
- userbinator 4y agoThat was discussed at the time on HN: https://news.ycombinator.com/item?id=34632677 https://news.ycombinator.com/item?id=34632677 My comment on that article still stands: https://news.ycombinator.com/item?id=34660474 https://news.ycombinator.com/item?id=34660474 tl;dr: don't make code generation worse for everyone else just to appease your one tiny use-case; fix your goddamn tools instead.
- aktau 4y agoWalking the stack via frame pointers is also faster than some other methods (https://blog.felixge.de/reducing-gos-execution-tracer-overhead-with-frame-pointer-unwinding/ https://blog.felixge.de/reducing-gos-execution-tracer-overhe...), it appears. This benefits use cases like profilers and tracers. In big deployments, continuous (sampled) profiling and tracing are common, helping not only with continuous performance improvements in fleets but also helping to root cause outages. Reducing the cost of such infrastructure can pay off.
- xobs 4y agoYeah, and I like I mentioned in the earlier comment, omitting the frame pointer reduces code size by 10% on RISC-V targets, which is huge when dealing with embedded flash: https://github.com/tock/tock/pull/1660 https://github.com/tock/tock/pull/1660
- fulafel 4y agoThis says the penalty is 10% on risc-v vs 2-3% on arm - anyone have background for why it's so?
- Someone 4y agoThere can only be two answers: - ARM frame pointer handling is more compact (relative to the non-frame pointer code) than that on RISC-V. - ARM code does less frame pointer handling. The first might be true because ARM has the stmdb instruction which (https://developer.arm.com/documentation/ddi0406/b/Application-Level-Architecture/Instruction-Details/Alphabetical-list-of-instructions/STMDB---STMFD https://developer.arm.com/documentation/ddi0406/b/Applicatio...) stores multiple registers to consecutive memory locations using an address from a base register. The consecutive memory locations end just below this address, and the address of the first of those locations can optionally be written back to the base register. So, code can save registers on the stack _and_ decrease the stack pointer in a single instruction. The second might be true because, of the compilers/compiler settings used for those measurements the ARM one did more aggressive function inlining.
- ddevault 4y agoStrongly agreed. Unwinding with libunwind, i.e. DWARF, is way, way more complex. A simple linked list is much better for the common use-cases.
- brancz 4y agoFor the always-on open-source profiler we happen to work on at work we had to do similar things and it was even more involved since we base the whole thing on eBPF to lower the overhead and therefore needed to get the verifier to accept it. [1] We really wish frame pointers were always present, but here we are. [1] https://www.polarsignals.com/blog/posts/2022/11/29/profiling-without-frame-pointers/ https://www.polarsignals.com/blog/posts/2022/11/29/profiling...
- userbinator 4y agoThere's a far simpler and also very generic method that I'm surprised no one has really mentioned much, although it's well-known in some corners of the RE/Asm community and I believe some debuggers (but not gdb?) use it: scan the stack for values that look like valid code addresses, then disassemble backwards from there to see if they were actually return addresses written there by a call instruction. You will find a chain that leads back to the entry point. Of course it won't work in edge cases like handwritten Asm that uses the stack in more clever ways, but when you're dealing with compiler output, it'll be fine. No need for all this complexity, and works in almost all cases.
- loeg 4y agoDo you analyze the dynamic loader segments (to determine code-like addresses) or just do some hard-coded approximations per platform? Pretty cute idea and I agree it will often work.
- userbinator 4y agoIt just needs to be on an executable page. The processor doesn't care about anything else, so this will work with things like JIT-generated code too. The key idea is that in essentially all cases of compiler-generated code, every return address will be immediately after the call that wrote it.
- loeg 4y agoYeah, but how do you look up executable page mappings? (Dynamic loader segments will have at least one executable Seg mapped somewhere.)
- userbinator 4y agoLinux: /proc/$pid/maps Windows: VirtualQuery() Mac: vm_region() Your own OS (like this article): you should know.
- 4y ago
- rco8786 4y agoI figured this article would be about Golang. if err := nil { return err }
- the_mitsuhiko 4y agoPlease just stop omitting frame pointers. You might lose out on 6 months of CPU speed advances, but from then on out you will reap the benefits of better production observability for years to come.
- zX41ZdbW 4y agoWe have implemented asynchronous signal-safe in-process stack unwinding for always-on profiler in ClickHouse: https://clickhouse.com/docs/en/operations/optimizing-performance/sampling-query-profiler https://clickhouse.com/docs/en/operations/optimizing-perform... The downside - it required many patches to LLVM's libunwind, and not all of them are accepted yet: https://bugs.llvm.org/show_bug.cgi?id=48186 https://bugs.llvm.org/show_bug.cgi?id=48186 ClickHouse source code: https://github.com/ClickHouse/ClickHouse https://github.com/ClickHouse/ClickHouse
- ShroudedNight 4y ago> ...yet I’ve seen programs handling SIGSEGV: for God’s sake, don’t do that! I recommend the author not examine JVM null pointer handling too closely then.
- lesenek 4y ago(I’m the author.) I will not then. It surprises me that the JVM would handle null pointers by hoping dereferencing one would raise a segfault, considering that the JVM is written in C AFAIK where such dereferencing invokes undefined behavior, not a memory access at address 0 which would segfault most of the time.
- bheadmaster 4y agoI think it's an optimization. Checking for null pointer on every object access can become costly, as it's a fundamental operation (everything is a pointer in Java). By handling SIGSEGV, you save time in non-null case, at the expense of more complex code and, as you've mentioned, non-standard behavior. Also wanted to mention your comment was [dead], had to vouch for it to resurrect it.
- lesenek 4y agoI understand the problem, but I am concerned by the solution. The problem is that C semantic is quite clear on the subject[1]: dereferencing a null pointer is undefined behavior, not "non-standard behavior" which would suggests correct albeit non-portable behavior. I don't know of any compiler switch that tells the compiler to "define" this behavior, like `-fno-strict-aliasing` or `-fwrapv` do. [1] ISO/IEC 9899:2011 §6.5.3.2
- bheadmaster 4y agoI understand - I deliberately used the vague term "non-standard behavior", since the solution somehow works. There is a GCC flag "-fno-delete-null-pointer-checks" [0] which seems like it might enable this behavior, but I'm not sure. [0] https://gcc.gnu.org/onlinedocs/gcc-7.3.0/gcc/Optimize-Options.html#:~:text=Use%20%2Dfno%2Ddelete%2Dnull,by%20default%20on%20most%20targets https://gcc.gnu.org/onlinedocs/gcc-7.3.0/gcc/Optimize-Option.... P.S. Seems like your account is shadowbanned. Might want to contact dang: hn@ycombinator.com if you plan on posting more.
- sylware 4y agoI code in assembly and most of the time I don't follow the ABI or be adamant on stack frames.
- subarctic 4y agoKind of cool that the register you look at after a segmentation fault is called `%rip`