5 ms·

Custom fast stack unwinding sounds interesting. How’s the performance on ARMv8?

by vardump 2y ago

quotemstr 2y ago
> fast stack unwinding sounds interesting Frame-pointer-based, I imagine?
- kouteiheika 2y ago
  No. It's DWARF based. The main two tricks are: it preprocesses all of the DWARF info at startup for faster lookups, and it dynamically patches the return addresses of functions on the stack injecting an address to its own trampoline, which allows it to skip going through the whole stack trace every time it needs to dump a backtrace. For example, if you're running a function nested 100 stack frames deep and that function calls malloc 100 times then Bytehound will only go through ~300 stack frames in total (~100 times for the first call then only ~2 frames for each successive call, if my math is right), while other similar tools will go through 10000 stack frames (going through all ~100 frames to the very bottom for every call).
  - vlovich123 2y ago
    Any plans to extend this idea into a performance profiler? Also nice use of Gimli - did something similar to make creating stack traces on crash cheaper to symbolicate.
    - kouteiheika 2y ago
      Not currently. For performance profiling I find that `perf`-like sampling profiling works well enough to find the hot spots, and then Valgrind's Callgrind is great for micro-optimizing the hot spots code on the assembly level. Of course, it would be cool to have a unified memory + performance analysis tool like this, but I don't think I can justify the time investment to write one in my spare time. Yeah, I'm really happy that Gimli exists, considering the absolute insanity/complexity pit of DWARF.
      - vlovich123 2y ago
        For what it’s worth, Valgrind completely fails to run on the Glommio runtime (something about it causes some threading code on startup to deadlock), so I’ve been looking for an alternate profiler that can give me better insights than perf. Also a profiler that can give me deeper insights without all the overhead of Valgrind would be sweet.
  - felixge 2y ago
    Dynamic patching of return addresses is a very cool trick. I don't think I've seen this before. Have you run into any situations where this crashes programs or otherwise interferes with their execution?
    - peterfirefly 2y ago
      Turbo Pascal used it for the overlay implementation (for DOS) -- overlays = virtual memory at home. TP 5.0 from 1988 was the first version that had it. The idea was to make sure the code the CPU returned to would actually be in memory. I'm pretty sure Windows 1.0 did something very similar.
    - tdullien 2y ago
      It's going to play poorly when C++ exceptions are thrown/caught.
      - felixge 2y ago
        Looking at the code [1] it seems like the library is actively trying to handle this problem. [1] https://github.com/koute/not-perf/blob/master/nwind/src/local_unwinding.rs https://github.com/koute/not-perf/blob/master/nwind/src/loca...
      - kouteiheika 2y ago
        It should support C++ exceptions. The trampolines have exception landing pads included to catch and rethrow any exceptions which are thrown through them.
    - kouteiheika 2y ago
      If the program's already doing weird stuff with the stack/control flow/etc., yes, but that should be relatively rare and for the majority of the programs it should work fine.
      - felixge 2y ago
        Thanks for the reply. I ended up implementing this idea in Go and wrote a blog post about the results: https://blog.felixge.de/blazingly-fast-shadow-stacks-for-go/ https://blog.felixge.de/blazingly-fast-shadow-stacks-for-go/ I'm curious if you've done any benchmarking for your implementation as well?
  - tubs 2y ago
    Any way this can work on arm64 without dwarf info at runtime? Would be very interested.
    - lathiat 2y ago
      I'm not sure about this implementation, but the parca implementation only needs the .eh_frame section of the binary (which is part of, but not all of "DWARF") which still exists even in stripped binaries. However you then still need debug symbols of some kind to convert those to names.
    - kouteiheika 2y ago
      Yes, it should also work without any debugging info. You'll still need unwinding tables though (used for handling exceptions in C++/panics in Rust/etc.), which are technically DWARF too (except on 32-bit ARM, which is special).
kouteiheika 2y ago
Haven't used it on ARM in a very long time, but should work just as well as on AMD64. (As long as you disable pointer authentication/CFI/whatever it was called on ARM.)

Custom fast stack unwinding sounds interesting. How’s the performance on ARMv8?

quotemstr 2y ago

> fast stack unwinding sounds interesting Frame-pointer-based, I imagine?

kouteiheika 2y ago
No. It's DWARF based. The main two tricks are: it preprocesses all of the DWARF info at startup for faster lookups, and it dynamically patches the return addresses of functions on the stack injecting an address to its own trampoline, which allows it to skip going through the whole stack trace every time it needs to dump a backtrace. For example, if you're running a function nested 100 stack frames deep and that function calls malloc 100 times then Bytehound will only go through ~300 stack frames in total (~100 times for the first call then only ~2 frames for each successive call, if my math is right), while other similar tools will go through 10000 stack frames (going through all ~100 frames to the very bottom for every call).
- vlovich123 2y ago
  Any plans to extend this idea into a performance profiler? Also nice use of Gimli - did something similar to make creating stack traces on crash cheaper to symbolicate.
  - kouteiheika 2y ago
    Not currently. For performance profiling I find that `perf`-like sampling profiling works well enough to find the hot spots, and then Valgrind's Callgrind is great for micro-optimizing the hot spots code on the assembly level. Of course, it would be cool to have a unified memory + performance analysis tool like this, but I don't think I can justify the time investment to write one in my spare time. Yeah, I'm really happy that Gimli exists, considering the absolute insanity/complexity pit of DWARF.
    - vlovich123 2y ago
      For what it’s worth, Valgrind completely fails to run on the Glommio runtime (something about it causes some threading code on startup to deadlock), so I’ve been looking for an alternate profiler that can give me better insights than perf. Also a profiler that can give me deeper insights without all the overhead of Valgrind would be sweet.
- felixge 2y ago
  Dynamic patching of return addresses is a very cool trick. I don't think I've seen this before. Have you run into any situations where this crashes programs or otherwise interferes with their execution?
  - peterfirefly 2y ago
    Turbo Pascal used it for the overlay implementation (for DOS) -- overlays = virtual memory at home. TP 5.0 from 1988 was the first version that had it. The idea was to make sure the code the CPU returned to would actually be in memory. I'm pretty sure Windows 1.0 did something very similar.
  - tdullien 2y ago
    It's going to play poorly when C++ exceptions are thrown/caught.
    - felixge 2y ago
      Looking at the code [1] it seems like the library is actively trying to handle this problem. [1] https://github.com/koute/not-perf/blob/master/nwind/src/local_unwinding.rs https://github.com/koute/not-perf/blob/master/nwind/src/loca...
    - kouteiheika 2y ago
      It should support C++ exceptions. The trampolines have exception landing pads included to catch and rethrow any exceptions which are thrown through them.
  - kouteiheika 2y ago
    If the program's already doing weird stuff with the stack/control flow/etc., yes, but that should be relatively rare and for the majority of the programs it should work fine.
    - felixge 2y ago
      Thanks for the reply. I ended up implementing this idea in Go and wrote a blog post about the results: https://blog.felixge.de/blazingly-fast-shadow-stacks-for-go/ https://blog.felixge.de/blazingly-fast-shadow-stacks-for-go/ I'm curious if you've done any benchmarking for your implementation as well?
- tubs 2y ago
  Any way this can work on arm64 without dwarf info at runtime? Would be very interested.
  - lathiat 2y ago
    I'm not sure about this implementation, but the parca implementation only needs the .eh_frame section of the binary (which is part of, but not all of "DWARF") which still exists even in stripped binaries. However you then still need debug symbols of some kind to convert those to names.
  - kouteiheika 2y ago
    Yes, it should also work without any debugging info. You'll still need unwinding tables though (used for handling exceptions in C++/panics in Rust/etc.), which are technically DWARF too (except on 32-bit ARM, which is special).

kouteiheika 2y ago

Haven't used it on ARM in a very long time, but should work just as well as on AMD64. (As long as you disable pointer authentication/CFI/whatever it was called on ARM.)