4 ms·

Sure, but that's not really counter to my point. An interrupt processed in the kernel isn't comparable to sending a signal to a userspace process, scheduling it

Sure, but that's not really counter to my point. An interrupt processed in the kernel isn't comparable to sending a signal to a userspace process, scheduling it (might not even be possible immediately!), walking the stack there, returning to the kernel and continuing. For stuff like bpf maps the kernel is even going to need the stacktrace in some form. Aside from the overhead, the delay between sample interrupt and the collected stack will be substantial. And then there's the whole fun of doing the stack walk in userspace in an async signal safe way.

AshamedCaptain 4y ago
You're overly exaggerating the impact slow stack unwinding can have on the reliability of profiles. Most people are literally just fine with the poor man's profiler (aka "watch gdb"), which would be significantly slower than anything being suggested here, and doesn't even have kernel help in freezing the traced process (the kernel can freeze it the moment the profiling event happens). The proprietary OS I'm working with right now actually does this, and the only visible limitation is that there are no stack traces on kernel crash. It does CoW the stack frames while the unwinder is going on. I still wouldn't do this, but I also see no reason why a monolithic kernel like Linux that already embeds multiple NTFS implementations can't embed a single decent DWARF unwinder.
- anarazel 4y ago
  Shrug. Our experiences differ drastically then. IME profiling using the gdb style manual sampling method work kinda OK if you have a very skewed profile, but basically not at all after that. Sure, lots of people don't know how to use a profiler with call graphs and everything. But a) partially that's because it's so hard to get a useful call-graph profile b) lots of people don't work on performance sensitive code, but plenty others do. The cost of walking stacks is pretty easy to see if you use perf report on a dwarf profile. A small part of that is just the larger data file, but if you profile perf report you can see that overwhelming the cost is unwinding. In my quick test case at least 70%. I'm sure that libunwind (as used by perf) could be improved by 2-3x - but you'd still be vastly slower.
  - AshamedCaptain 4y ago
    > IME profiling using the gdb style manual sampling method work kinda OK if you have a very skewed profile, but basically not at all after that. I really don't see what you mean here. How can you have a skewed profile with "manual sampling"? Do you really care about skew between your samples? Isn't it actually _better_ if you space your samples at random intervals, rather than strictly 0-skew regular? Also this is completely orthogonal with whether you end up with a call graph or not. I would think most people end up with a call graph even if you did your samples with poor man's. There are multiple scripts floating around that do it even if you start from one-symbol-per-line text dumps. Even Google had one of this? Note I can totally understand the point if you are tracing e.g. in response to a performance counter. But in these cases the kernel will help you freeze anyway, and I don't think this is what most users are doing, nor what should be doing.
    - anarazel 4y ago
      "Skewed" was perhaps not a good descriptor. What I mean is workloads where most of the time is spent in a small part of the program. Many programs have more flat profiles though. Profiling those with gdb doesn't work well. It also doesn't work very well when you have many processes/threads.

AshamedCaptain 4y ago

You're overly exaggerating the impact slow stack unwinding can have on the reliability of profiles. Most people are literally just fine with the poor man's profiler (aka "watch gdb"), which would be significantly slower than anything being suggested here, and doesn't even have kernel help in freezing the traced process (the kernel can freeze it the moment the profiling event happens). The proprietary OS I'm working with right now actually does this, and the only visible limitation is that there are no stack traces on kernel crash. It does CoW the stack frames while the unwinder is going on. I still wouldn't do this, but I also see no reason why a monolithic kernel like Linux that already embeds multiple NTFS implementations can't embed a single decent DWARF unwinder.

anarazel 4y ago
Shrug. Our experiences differ drastically then. IME profiling using the gdb style manual sampling method work kinda OK if you have a very skewed profile, but basically not at all after that. Sure, lots of people don't know how to use a profiler with call graphs and everything. But a) partially that's because it's so hard to get a useful call-graph profile b) lots of people don't work on performance sensitive code, but plenty others do. The cost of walking stacks is pretty easy to see if you use perf report on a dwarf profile. A small part of that is just the larger data file, but if you profile perf report you can see that overwhelming the cost is unwinding. In my quick test case at least 70%. I'm sure that libunwind (as used by perf) could be improved by 2-3x - but you'd still be vastly slower.
- AshamedCaptain 4y ago
  > IME profiling using the gdb style manual sampling method work kinda OK if you have a very skewed profile, but basically not at all after that. I really don't see what you mean here. How can you have a skewed profile with "manual sampling"? Do you really care about skew between your samples? Isn't it actually _better_ if you space your samples at random intervals, rather than strictly 0-skew regular? Also this is completely orthogonal with whether you end up with a call graph or not. I would think most people end up with a call graph even if you did your samples with poor man's. There are multiple scripts floating around that do it even if you start from one-symbol-per-line text dumps. Even Google had one of this? Note I can totally understand the point if you are tracing e.g. in response to a performance counter. But in these cases the kernel will help you freeze anyway, and I don't think this is what most users are doing, nor what should be doing.
  - anarazel 4y ago
    "Skewed" was perhaps not a good descriptor. What I mean is workloads where most of the time is spent in a small part of the program. Many programs have more flat profiles though. Profiling those with gdb doesn't work well. It also doesn't work very well when you have many processes/threads.