5 ms·

I predicted this would happen when ORC was introduced to the kernel. I'm tired of this eternal wheel of history in computing where everyone feels the need to re

I predicted this would happen when ORC was introduced to the kernel. I'm tired of this eternal wheel of history in computing where everyone feels the need to rewrite stuff that works. If it wasn't for this we'd already have a DWARF unwinder in the kernel and on every other profiling-tool-of-the-week and this would be a non-issue. But no! Let's come up with a new debuginfo format, that should simplify things right? And a couple years later the main distros have to downgrade to the least common denominator (in this case, frame pointers) just because profilers and toolchains can't implement one. single. common. debuginfo. format for once.

gpderetta 4y ago
The linux kernel: NIH as an art form.
quotemstr 4y ago
> And a couple years later the main distros have to downgrade to the least common denominator (in this case, frame pointers) just because profilers and toolchains can't implement one. single. common. debuginfo. format for once. You don't even need a common debug information format. All you need is a mechanism through which the kernel can ask userspace to collect its own stack. But because the Linux world has elevated shipping the org chart to the highest level of virtue, it's hard to coordinate reasonable solutions.
- anarazel 4y ago
  Introducing a context switch, performing unwinding in process, with a relatively expensive method, would distort profiles sufficiently to make them not that useful anymore.
  - AshamedCaptain 4y ago
    Kernel-based profiling already involves multiple context switches by definition.
    - anarazel 4y ago
      Sure, but that's not really counter to my point. An interrupt processed in the kernel isn't comparable to sending a signal to a userspace process, scheduling it (might not even be possible immediately!), walking the stack there, returning to the kernel and continuing. For stuff like bpf maps the kernel is even going to need the stacktrace in some form. Aside from the overhead, the delay between sample interrupt and the collected stack will be substantial. And then there's the whole fun of doing the stack walk in userspace in an async signal safe way.
      - AshamedCaptain 4y ago
        You're overly exaggerating the impact slow stack unwinding can have on the reliability of profiles. Most people are literally just fine with the poor man's profiler (aka "watch gdb"), which would be significantly slower than anything being suggested here, and doesn't even have kernel help in freezing the traced process (the kernel can freeze it the moment the profiling event happens). The proprietary OS I'm working with right now actually does this, and the only visible limitation is that there are no stack traces on kernel crash. It does CoW the stack frames while the unwinder is going on. I still wouldn't do this, but I also see no reason why a monolithic kernel like Linux that already embeds multiple NTFS implementations can't embed a single decent DWARF unwinder.
        anarazel 4y ago
        Shrug. Our experiences differ drastically then. IME profiling using the gdb style manual sampling method work kinda OK if you have a very skewed profile, but basically not at all after that. Sure, lots of people don't know how to use a profiler with call graphs and everything. But a) partially that's because it's so hard to get a useful call-graph profile b) lots of people don't work on performance sensitive code, but plenty others do. The cost of walking stacks is pretty easy to see if you use perf report on a dwarf profile. A small part of that is just the larger data file, but if you profile perf report you can see that overwhelming the cost is unwinding. In my quick test case at least 70%. I'm sure that libunwind (as used by perf) could be improved by 2-3x - but you'd still be vastly slower.
  - quotemstr 4y ago
    There is no "context switch". Read my post, please. The kernel already has to return from kernel mode to user mode at some point. Until it does, the user stack is stable. We collect the user stack when we return to user space. There are no extra privilege level transitions, which is what I think you meant by "context switch". As for the speed of that collection: as I mention in my post, there's nothing the kernel can do that gives it magical performance superpowers that userspace lacks. User space can walk frame pointers too.
    - saagarjha 4y ago
      What happens after userspace hands back its stack trace (and wants to execute code now)? What happens if the stack trace unwinder is preempted?
    - anarazel 4y ago
      Which blog post is yours? You posted https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/IMBATOSTV7BIX7WQKMUWGDFYJG2TSBIF/ https://lists.fedoraproject.org/archives/list/devel@lists.fe... earlier - but I don't know if that's yours. Based on the approach there I do think you'd end up with several times more context switches. For one, signal delivery alone causes additional syscalls. But more importantly, for things like bpf maps keyed by the stack id you need the stack at that point, not later. So you can't just defer the walk. > As for the speed of that collection: as I mention in my post, there's nothing the kernel can do that gives it magical performance superpowers that userspace lacks. User space can walk frame pointers too. It's not that the kernel is magically more efficient, it's that fp based stack walking is more efficient than the alternatives. And once you have fp based stack walks, there's no need for the additional dispatch for AOT languages. I think it could be useful to have something like you describe for more dynamic languages. But even then I think you'd want to make that a profile time opt-in thing.
      - quotemstr 4y ago
        > For one, signal delivery alone causes additional syscalls. No, it doesn't. The kernel is going to return to user space eventually anyway. The signal just makes the kernel "return" to a place in userspace other than the point it entered the kernel: the signal handler. When the signal handler returns (and no, you don't have to use the sigreturn system call), userspace resumes execution where it left off. > But more importantly, for things like bpf maps keyed by the stack id you need the stack at that point, not later. So you can't just defer the walk. Insertion into those maps can be deferred until userspace gets around to collecting its stack. Furthermore, kernel side immediate stack collection still doesn't solve the page fault problem.

gpderetta 4y ago

The linux kernel: NIH as an art form.

quotemstr 4y ago

> And a couple years later the main distros have to downgrade to the least common denominator (in this case, frame pointers) just because profilers and toolchains can't implement one. single. common. debuginfo. format for once. You don't even need a common debug information format. All you need is a mechanism through which the kernel can ask userspace to collect its own stack. But because the Linux world has elevated shipping the org chart to the highest level of virtue, it's hard to coordinate reasonable solutions.

anarazel 4y ago
Introducing a context switch, performing unwinding in process, with a relatively expensive method, would distort profiles sufficiently to make them not that useful anymore.
- AshamedCaptain 4y ago
  Kernel-based profiling already involves multiple context switches by definition.
  - anarazel 4y ago
    Sure, but that's not really counter to my point. An interrupt processed in the kernel isn't comparable to sending a signal to a userspace process, scheduling it (might not even be possible immediately!), walking the stack there, returning to the kernel and continuing. For stuff like bpf maps the kernel is even going to need the stacktrace in some form. Aside from the overhead, the delay between sample interrupt and the collected stack will be substantial. And then there's the whole fun of doing the stack walk in userspace in an async signal safe way.
    - AshamedCaptain 4y ago
      You're overly exaggerating the impact slow stack unwinding can have on the reliability of profiles. Most people are literally just fine with the poor man's profiler (aka "watch gdb"), which would be significantly slower than anything being suggested here, and doesn't even have kernel help in freezing the traced process (the kernel can freeze it the moment the profiling event happens). The proprietary OS I'm working with right now actually does this, and the only visible limitation is that there are no stack traces on kernel crash. It does CoW the stack frames while the unwinder is going on. I still wouldn't do this, but I also see no reason why a monolithic kernel like Linux that already embeds multiple NTFS implementations can't embed a single decent DWARF unwinder.
      - anarazel 4y ago
        Shrug. Our experiences differ drastically then. IME profiling using the gdb style manual sampling method work kinda OK if you have a very skewed profile, but basically not at all after that. Sure, lots of people don't know how to use a profiler with call graphs and everything. But a) partially that's because it's so hard to get a useful call-graph profile b) lots of people don't work on performance sensitive code, but plenty others do. The cost of walking stacks is pretty easy to see if you use perf report on a dwarf profile. A small part of that is just the larger data file, but if you profile perf report you can see that overwhelming the cost is unwinding. In my quick test case at least 70%. I'm sure that libunwind (as used by perf) could be improved by 2-3x - but you'd still be vastly slower.
- quotemstr 4y ago
  There is no "context switch". Read my post, please. The kernel already has to return from kernel mode to user mode at some point. Until it does, the user stack is stable. We collect the user stack when we return to user space. There are no extra privilege level transitions, which is what I think you meant by "context switch". As for the speed of that collection: as I mention in my post, there's nothing the kernel can do that gives it magical performance superpowers that userspace lacks. User space can walk frame pointers too.
  - saagarjha 4y ago
    What happens after userspace hands back its stack trace (and wants to execute code now)? What happens if the stack trace unwinder is preempted?
  - anarazel 4y ago
    Which blog post is yours? You posted https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/IMBATOSTV7BIX7WQKMUWGDFYJG2TSBIF/ https://lists.fedoraproject.org/archives/list/devel@lists.fe... earlier - but I don't know if that's yours. Based on the approach there I do think you'd end up with several times more context switches. For one, signal delivery alone causes additional syscalls. But more importantly, for things like bpf maps keyed by the stack id you need the stack at that point, not later. So you can't just defer the walk. > As for the speed of that collection: as I mention in my post, there's nothing the kernel can do that gives it magical performance superpowers that userspace lacks. User space can walk frame pointers too. It's not that the kernel is magically more efficient, it's that fp based stack walking is more efficient than the alternatives. And once you have fp based stack walks, there's no need for the additional dispatch for AOT languages. I think it could be useful to have something like you describe for more dynamic languages. But even then I think you'd want to make that a profile time opt-in thing.
    - quotemstr 4y ago
      > For one, signal delivery alone causes additional syscalls. No, it doesn't. The kernel is going to return to user space eventually anyway. The signal just makes the kernel "return" to a place in userspace other than the point it entered the kernel: the signal handler. When the signal handler returns (and no, you don't have to use the sigreturn system call), userspace resumes execution where it left off. > But more importantly, for things like bpf maps keyed by the stack id you need the stack at that point, not later. So you can't just defer the walk. Insertion into those maps can be deferred until userspace gets around to collecting its stack. Furthermore, kernel side immediate stack collection still doesn't solve the page fault problem.