4 ms·

Introducing a context switch, performing unwinding in process, with a relatively expensive method, would distort profiles sufficiently to make them not that use

Introducing a context switch, performing unwinding in process, with a relatively expensive method, would distort profiles sufficiently to make them not that useful anymore.

AshamedCaptain 4y ago
Kernel-based profiling already involves multiple context switches by definition.
- anarazel 4y ago
  Sure, but that's not really counter to my point. An interrupt processed in the kernel isn't comparable to sending a signal to a userspace process, scheduling it (might not even be possible immediately!), walking the stack there, returning to the kernel and continuing. For stuff like bpf maps the kernel is even going to need the stacktrace in some form. Aside from the overhead, the delay between sample interrupt and the collected stack will be substantial. And then there's the whole fun of doing the stack walk in userspace in an async signal safe way.
  - AshamedCaptain 4y ago
    You're overly exaggerating the impact slow stack unwinding can have on the reliability of profiles. Most people are literally just fine with the poor man's profiler (aka "watch gdb"), which would be significantly slower than anything being suggested here, and doesn't even have kernel help in freezing the traced process (the kernel can freeze it the moment the profiling event happens). The proprietary OS I'm working with right now actually does this, and the only visible limitation is that there are no stack traces on kernel crash. It does CoW the stack frames while the unwinder is going on. I still wouldn't do this, but I also see no reason why a monolithic kernel like Linux that already embeds multiple NTFS implementations can't embed a single decent DWARF unwinder.
    - anarazel 4y ago
      Shrug. Our experiences differ drastically then. IME profiling using the gdb style manual sampling method work kinda OK if you have a very skewed profile, but basically not at all after that. Sure, lots of people don't know how to use a profiler with call graphs and everything. But a) partially that's because it's so hard to get a useful call-graph profile b) lots of people don't work on performance sensitive code, but plenty others do. The cost of walking stacks is pretty easy to see if you use perf report on a dwarf profile. A small part of that is just the larger data file, but if you profile perf report you can see that overwhelming the cost is unwinding. In my quick test case at least 70%. I'm sure that libunwind (as used by perf) could be improved by 2-3x - but you'd still be vastly slower.
      - AshamedCaptain 4y ago
        > IME profiling using the gdb style manual sampling method work kinda OK if you have a very skewed profile, but basically not at all after that. I really don't see what you mean here. How can you have a skewed profile with "manual sampling"? Do you really care about skew between your samples? Isn't it actually _better_ if you space your samples at random intervals, rather than strictly 0-skew regular? Also this is completely orthogonal with whether you end up with a call graph or not. I would think most people end up with a call graph even if you did your samples with poor man's. There are multiple scripts floating around that do it even if you start from one-symbol-per-line text dumps. Even Google had one of this? Note I can totally understand the point if you are tracing e.g. in response to a performance counter. But in these cases the kernel will help you freeze anyway, and I don't think this is what most users are doing, nor what should be doing.
        anarazel 4y ago
        "Skewed" was perhaps not a good descriptor. What I mean is workloads where most of the time is spent in a small part of the program. Many programs have more flat profiles though. Profiling those with gdb doesn't work well. It also doesn't work very well when you have many processes/threads.
quotemstr 4y ago
There is no "context switch". Read my post, please. The kernel already has to return from kernel mode to user mode at some point. Until it does, the user stack is stable. We collect the user stack when we return to user space. There are no extra privilege level transitions, which is what I think you meant by "context switch". As for the speed of that collection: as I mention in my post, there's nothing the kernel can do that gives it magical performance superpowers that userspace lacks. User space can walk frame pointers too.
- saagarjha 4y ago
  What happens after userspace hands back its stack trace (and wants to execute code now)? What happens if the stack trace unwinder is preempted?
- anarazel 4y ago
  Which blog post is yours? You posted https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/IMBATOSTV7BIX7WQKMUWGDFYJG2TSBIF/ https://lists.fedoraproject.org/archives/list/devel@lists.fe... earlier - but I don't know if that's yours. Based on the approach there I do think you'd end up with several times more context switches. For one, signal delivery alone causes additional syscalls. But more importantly, for things like bpf maps keyed by the stack id you need the stack at that point, not later. So you can't just defer the walk. > As for the speed of that collection: as I mention in my post, there's nothing the kernel can do that gives it magical performance superpowers that userspace lacks. User space can walk frame pointers too. It's not that the kernel is magically more efficient, it's that fp based stack walking is more efficient than the alternatives. And once you have fp based stack walks, there's no need for the additional dispatch for AOT languages. I think it could be useful to have something like you describe for more dynamic languages. But even then I think you'd want to make that a profile time opt-in thing.
  - quotemstr 4y ago
    > For one, signal delivery alone causes additional syscalls. No, it doesn't. The kernel is going to return to user space eventually anyway. The signal just makes the kernel "return" to a place in userspace other than the point it entered the kernel: the signal handler. When the signal handler returns (and no, you don't have to use the sigreturn system call), userspace resumes execution where it left off. > But more importantly, for things like bpf maps keyed by the stack id you need the stack at that point, not later. So you can't just defer the walk. Insertion into those maps can be deferred until userspace gets around to collecting its stack. Furthermore, kernel side immediate stack collection still doesn't solve the page fault problem.

Introducing a context switch, performing unwinding in process, with a relatively expensive method, would distort profiles sufficiently to make them not that useful anymore.

AshamedCaptain 4y ago

Kernel-based profiling already involves multiple context switches by definition.

anarazel 4y ago
Sure, but that's not really counter to my point. An interrupt processed in the kernel isn't comparable to sending a signal to a userspace process, scheduling it (might not even be possible immediately!), walking the stack there, returning to the kernel and continuing. For stuff like bpf maps the kernel is even going to need the stacktrace in some form. Aside from the overhead, the delay between sample interrupt and the collected stack will be substantial. And then there's the whole fun of doing the stack walk in userspace in an async signal safe way.
- AshamedCaptain 4y ago
  You're overly exaggerating the impact slow stack unwinding can have on the reliability of profiles. Most people are literally just fine with the poor man's profiler (aka "watch gdb"), which would be significantly slower than anything being suggested here, and doesn't even have kernel help in freezing the traced process (the kernel can freeze it the moment the profiling event happens). The proprietary OS I'm working with right now actually does this, and the only visible limitation is that there are no stack traces on kernel crash. It does CoW the stack frames while the unwinder is going on. I still wouldn't do this, but I also see no reason why a monolithic kernel like Linux that already embeds multiple NTFS implementations can't embed a single decent DWARF unwinder.
  - anarazel 4y ago
    Shrug. Our experiences differ drastically then. IME profiling using the gdb style manual sampling method work kinda OK if you have a very skewed profile, but basically not at all after that. Sure, lots of people don't know how to use a profiler with call graphs and everything. But a) partially that's because it's so hard to get a useful call-graph profile b) lots of people don't work on performance sensitive code, but plenty others do. The cost of walking stacks is pretty easy to see if you use perf report on a dwarf profile. A small part of that is just the larger data file, but if you profile perf report you can see that overwhelming the cost is unwinding. In my quick test case at least 70%. I'm sure that libunwind (as used by perf) could be improved by 2-3x - but you'd still be vastly slower.
    - AshamedCaptain 4y ago
      > IME profiling using the gdb style manual sampling method work kinda OK if you have a very skewed profile, but basically not at all after that. I really don't see what you mean here. How can you have a skewed profile with "manual sampling"? Do you really care about skew between your samples? Isn't it actually _better_ if you space your samples at random intervals, rather than strictly 0-skew regular? Also this is completely orthogonal with whether you end up with a call graph or not. I would think most people end up with a call graph even if you did your samples with poor man's. There are multiple scripts floating around that do it even if you start from one-symbol-per-line text dumps. Even Google had one of this? Note I can totally understand the point if you are tracing e.g. in response to a performance counter. But in these cases the kernel will help you freeze anyway, and I don't think this is what most users are doing, nor what should be doing.
      - anarazel 4y ago
        "Skewed" was perhaps not a good descriptor. What I mean is workloads where most of the time is spent in a small part of the program. Many programs have more flat profiles though. Profiling those with gdb doesn't work well. It also doesn't work very well when you have many processes/threads.

quotemstr 4y ago

There is no "context switch". Read my post, please. The kernel already has to return from kernel mode to user mode at some point. Until it does, the user stack is stable. We collect the user stack when we return to user space. There are no extra privilege level transitions, which is what I think you meant by "context switch". As for the speed of that collection: as I mention in my post, there's nothing the kernel can do that gives it magical performance superpowers that userspace lacks. User space can walk frame pointers too.

saagarjha 4y ago
What happens after userspace hands back its stack trace (and wants to execute code now)? What happens if the stack trace unwinder is preempted?
anarazel 4y ago
Which blog post is yours? You posted https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/IMBATOSTV7BIX7WQKMUWGDFYJG2TSBIF/ https://lists.fedoraproject.org/archives/list/devel@lists.fe... earlier - but I don't know if that's yours. Based on the approach there I do think you'd end up with several times more context switches. For one, signal delivery alone causes additional syscalls. But more importantly, for things like bpf maps keyed by the stack id you need the stack at that point, not later. So you can't just defer the walk. > As for the speed of that collection: as I mention in my post, there's nothing the kernel can do that gives it magical performance superpowers that userspace lacks. User space can walk frame pointers too. It's not that the kernel is magically more efficient, it's that fp based stack walking is more efficient than the alternatives. And once you have fp based stack walks, there's no need for the additional dispatch for AOT languages. I think it could be useful to have something like you describe for more dynamic languages. But even then I think you'd want to make that a profile time opt-in thing.
- quotemstr 4y ago
  > For one, signal delivery alone causes additional syscalls. No, it doesn't. The kernel is going to return to user space eventually anyway. The signal just makes the kernel "return" to a place in userspace other than the point it entered the kernel: the signal handler. When the signal handler returns (and no, you don't have to use the sigreturn system call), userspace resumes execution where it left off. > But more importantly, for things like bpf maps keyed by the stack id you need the stack at that point, not later. So you can't just defer the walk. Insertion into those maps can be deferred until userspace gets around to collecting its stack. Furthermore, kernel side immediate stack collection still doesn't solve the page fault problem.