4 ms·

The biggest problem with frame pointer unwinding isn't even the performance hit, but the inability to support unwinding of anything that isn't AOT compiled nati

The biggest problem with frame pointer unwinding isn't even the performance hit, but the inability to support unwinding of anything that isn't AOT compiled native code. If you unwind using frame pointers through a Python program, you'll get a bunch of meaningless interpreter frames instead of information about the operation of the program being interpreted. That, and since the FP unwinder can't sleep and can't take page faults, it'll fail to traverse stack frames that happen to be paged out. The right way to solve the kernel unwind of userspace problem is to get the kernel out of the business of unwinding userspace. Concrete proposal here: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/IMBATOSTV7BIX7WQKMUWGDFYJG2TSBIF/ https://lists.fedoraproject.org/archives/list/devel@lists.fe... Unwinding only paged-in, AOT-compiled native code is a step in the wrong direction, especially because it's precisely the people for whom unwinding kinda-sorta works now (kernel, libc, C language diehards, etc.) who would need to approve and coordinate a more robust and comprehensive solution that also, unlike frame pointers, has zero cost when not in use.

rwmj 4y ago
You want the kernel to be able to collect the trace as fast as possible. You don't want to add half a million context switches to your trace.
- quotemstr 4y ago
  Where are the extra "context switches"? Can you count them for me? Under my proposal, we defer stack collection until we're going to return to user space anyway. The user stack of a thread cannot change while that thread is running in user mode.
  - rwmj 4y ago
    So after looking at this, I do now agree [can't edit my original comment]. Are you planning to implement anything like this? I think similar to the suggestion to fix DWARF, enabling frame pointers may not be the best solution, it may be reversed in future when we get something better, but it's the one that actually works right now.
nemetroid 4y ago
The Linux perf profiler has support for symbolization of dynamic code generated by a JIT. This is supported by at least the JVM, Node, and Python (3.12). https://github.com/torvalds/linux/blob/0513e464f9007b70b96740271a948ca5ab6e7dd7/tools/perf/Documentation/jit-interface.txt https://github.com/torvalds/linux/blob/0513e464f9007b70b9674... https://www.brendangregg.com/Slides/KernelRecipes_Perf_Events.pdf https://www.brendangregg.com/Slides/KernelRecipes_Perf_Event... (pp. 53-58) https://docs.python.org/3.12/howto/perf_profiling.html#perf-profiling https://docs.python.org/3.12/howto/perf_profiling.html#perf-...
- quotemstr 4y ago
  This is an incredibly hacky mechanism that has to be individually enabled in each context in which it's used. What if I can't write to tmp? What if PIDs wrap around?
  - nemetroid 4y ago
    I don't see why the "individually enabled" part is inherent to the design. The JITs could emit these files automatically, if they wanted to. But the main point is that "inability to support unwinding of anything that isn't AOT compiled native code" is factually incorrect: as far as I know, frame pointers plus this "hack" is currently the only widely available way to do just that.
    - quotemstr 4y ago
      No, they can't, because doing so would be a constant drag on performance for processes not being profiled. Plus, there's no provision for VA reuse for the trampolines, so as the set of described functions grows, so does the magic text file in /tmp.
saagarjha 4y ago
It would be nice to have something like this, but there are generally implementation challenges with getting userspace to do anything on behalf of the kernel.

rwmj 4y ago

You want the kernel to be able to collect the trace as fast as possible. You don't want to add half a million context switches to your trace.

quotemstr 4y ago
Where are the extra "context switches"? Can you count them for me? Under my proposal, we defer stack collection until we're going to return to user space anyway. The user stack of a thread cannot change while that thread is running in user mode.
- rwmj 4y ago
  So after looking at this, I do now agree [can't edit my original comment]. Are you planning to implement anything like this? I think similar to the suggestion to fix DWARF, enabling frame pointers may not be the best solution, it may be reversed in future when we get something better, but it's the one that actually works right now.

nemetroid 4y ago

The Linux perf profiler has support for symbolization of dynamic code generated by a JIT. This is supported by at least the JVM, Node, and Python (3.12). https://github.com/torvalds/linux/blob/0513e464f9007b70b96740271a948ca5ab6e7dd7/tools/perf/Documentation/jit-interface.txt https://github.com/torvalds/linux/blob/0513e464f9007b70b9674... https://www.brendangregg.com/Slides/KernelRecipes_Perf_Events.pdf https://www.brendangregg.com/Slides/KernelRecipes_Perf_Event... (pp. 53-58) https://docs.python.org/3.12/howto/perf_profiling.html#perf-profiling https://docs.python.org/3.12/howto/perf_profiling.html#perf-...

quotemstr 4y ago
This is an incredibly hacky mechanism that has to be individually enabled in each context in which it's used. What if I can't write to tmp? What if PIDs wrap around?
- nemetroid 4y ago
  I don't see why the "individually enabled" part is inherent to the design. The JITs could emit these files automatically, if they wanted to. But the main point is that "inability to support unwinding of anything that isn't AOT compiled native code" is factually incorrect: as far as I know, frame pointers plus this "hack" is currently the only widely available way to do just that.
  - quotemstr 4y ago
    No, they can't, because doing so would be a constant drag on performance for processes not being profiled. Plus, there's no provision for VA reuse for the trampolines, so as the set of described functions grows, so does the magic text file in /tmp.

saagarjha 4y ago

It would be nice to have something like this, but there are generally implementation challenges with getting userspace to do anything on behalf of the kernel.