3 ms·

Which blog post is yours? You posted https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/IMBATOSTV7BIX7WQKMUWGDFYJG2TSBIF/ https:

Which blog post is yours? You posted https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/IMBATOSTV7BIX7WQKMUWGDFYJG2TSBIF/ https://lists.fedoraproject.org/archives/list/devel@lists.fe... earlier - but I don't know if that's yours. Based on the approach there I do think you'd end up with several times more context switches. For one, signal delivery alone causes additional syscalls. But more importantly, for things like bpf maps keyed by the stack id you need the stack at that point, not later. So you can't just defer the walk. > As for the speed of that collection: as I mention in my post, there's nothing the kernel can do that gives it magical performance superpowers that userspace lacks. User space can walk frame pointers too. It's not that the kernel is magically more efficient, it's that fp based stack walking is more efficient than the alternatives. And once you have fp based stack walks, there's no need for the additional dispatch for AOT languages. I think it could be useful to have something like you describe for more dynamic languages. But even then I think you'd want to make that a profile time opt-in thing.

quotemstr 4y ago
> For one, signal delivery alone causes additional syscalls. No, it doesn't. The kernel is going to return to user space eventually anyway. The signal just makes the kernel "return" to a place in userspace other than the point it entered the kernel: the signal handler. When the signal handler returns (and no, you don't have to use the sigreturn system call), userspace resumes execution where it left off. > But more importantly, for things like bpf maps keyed by the stack id you need the stack at that point, not later. So you can't just defer the walk. Insertion into those maps can be deferred until userspace gets around to collecting its stack. Furthermore, kernel side immediate stack collection still doesn't solve the page fault problem.
- anarazel 4y ago
  Re signals, see https://man7.org/linux/man-pages/man2/sigreturn.2.html https://man7.org/linux/man-pages/man2/sigreturn.2.html - you'd at least end up with that. Re bpf: I don't think you can just defer the insertion into maps, that'd block execution, as code can depend an the contents of the bpf maps before continuing on with the bpf program. Iirc for bpf based tracing, the evaluation happens synchronously at the insertion points.
  - quotemstr 4y ago
    > Re signals, see https://man7.org/linux/man-pages/man2/sigreturn.2.html https://man7.org/linux/man-pages/man2/sigreturn.2.html - you'd at least end up with that. Not necessarily. sigreturn doesn't do anything that userspace can't do on its own. It's kind of an ugly vestige of the days when system calls were "no big deal" instead of the performance-killers they are today. While it's true that user threads can't reset their signal masks without a system call, 1) SA_NODEFER means we don't have to, and 2) in theory, we could make an rseq field that would allow us to implement sigprocmask(2) without an entry into the kernel! > Re bpf: I don't think you can just defer the insertion into maps, that'd block execution, as code can depend an the contents of the bpf maps before continuing on with the bpf program. Iirc for bpf based tracing, the evaluation happens synchronously at the insertion points. I think you're right, and that's unfortunate. I'm still not sure, however, whether the need to support BPF stack-map insertion is sufficient to kill the userspace-driven unwinding scheme I've described, because (at least in my experience) this BPF-map-key scenario is a pretty fringe case. Let's look at it this way: is it possible in principle to reconcile a solution to the pagefault problem with the needs of BPF stack-hash map keys? Even if I were to concede that userspace stack unwinding belonged in the kernel, and even if I were to additionally concede that we should do userspace stack unwinding via frame pointer walking, we'd still need to make the kernel-mode unwinder capable of taking page faults (and thus sleeping) in order to make unwinding fully robust. Let's suppose we did userspace unwinding via sleepable eBPF instead of a signal: would it be possible to accommodate, in this model, the needs of eBPF-based stack-keyed maps? In other words: it's fundamentally impossible for an eBPF program attached to a tracepoint to query a map the key of which is a complete (user+kernel) stack if, in that context, eBPF cannot sleep. This impossibility is independent of my specific proposal and is inherent to the need for any robust stack-walker, user- or kernel-mode, to be able to take page faults and thus sleep. If I'm right about this impossibility, then we have to give up on one of 1) fully robust full (user+kernel) stack walking, and 2) stack-keyed eBPF maps. Which would you prefer?
    - anarazel 4y ago
      > While it's true that user threads can't reset their signal masks without a system call, 1) SA_NODEFER means we don't have to, and 2) in theory, we could make an rseq field that would allow us to implement sigprocmask(2) without an entry into the kernel! Writing a signal handler without SA_NODEFER is ... not fun. > I think you're right, and that's unfortunate. I'm still not sure, however, whether the need to support BPF stack-map insertion is sufficient to kill the userspace-driven unwinding scheme I've described, because (at least in my experience) this BPF-map-key scenario is a pretty fringe case. I don't think it's all that fringe, but yes, it's not as common as plain profiling. The thing that makes it the most niche is the difficulty around lack of frame pointers in libc - other libraries matter far less. > If I'm right about this impossibility, then we have to give up on one of 1) fully robust full (user+kernel) stack walking, and 2) stack-keyed eBPF maps. Which would you prefer? I'd give up 1). The stack page fault issue isn't all that big a deal in practice, although I do agree that it's quite ugly. Whereas being able to build things like "offwake" profiles (showing the stack of the waiter and the wakee + duration of sleeps) allows to debug/profile issues that are extremely hard to debug otherwise. And rely quite fundamentally on stack-keyed BPF maps.

quotemstr 4y ago

> For one, signal delivery alone causes additional syscalls. No, it doesn't. The kernel is going to return to user space eventually anyway. The signal just makes the kernel "return" to a place in userspace other than the point it entered the kernel: the signal handler. When the signal handler returns (and no, you don't have to use the sigreturn system call), userspace resumes execution where it left off. > But more importantly, for things like bpf maps keyed by the stack id you need the stack at that point, not later. So you can't just defer the walk. Insertion into those maps can be deferred until userspace gets around to collecting its stack. Furthermore, kernel side immediate stack collection still doesn't solve the page fault problem.

anarazel 4y ago
Re signals, see https://man7.org/linux/man-pages/man2/sigreturn.2.html https://man7.org/linux/man-pages/man2/sigreturn.2.html - you'd at least end up with that. Re bpf: I don't think you can just defer the insertion into maps, that'd block execution, as code can depend an the contents of the bpf maps before continuing on with the bpf program. Iirc for bpf based tracing, the evaluation happens synchronously at the insertion points.
- quotemstr 4y ago
  > Re signals, see https://man7.org/linux/man-pages/man2/sigreturn.2.html https://man7.org/linux/man-pages/man2/sigreturn.2.html - you'd at least end up with that. Not necessarily. sigreturn doesn't do anything that userspace can't do on its own. It's kind of an ugly vestige of the days when system calls were "no big deal" instead of the performance-killers they are today. While it's true that user threads can't reset their signal masks without a system call, 1) SA_NODEFER means we don't have to, and 2) in theory, we could make an rseq field that would allow us to implement sigprocmask(2) without an entry into the kernel! > Re bpf: I don't think you can just defer the insertion into maps, that'd block execution, as code can depend an the contents of the bpf maps before continuing on with the bpf program. Iirc for bpf based tracing, the evaluation happens synchronously at the insertion points. I think you're right, and that's unfortunate. I'm still not sure, however, whether the need to support BPF stack-map insertion is sufficient to kill the userspace-driven unwinding scheme I've described, because (at least in my experience) this BPF-map-key scenario is a pretty fringe case. Let's look at it this way: is it possible in principle to reconcile a solution to the pagefault problem with the needs of BPF stack-hash map keys? Even if I were to concede that userspace stack unwinding belonged in the kernel, and even if I were to additionally concede that we should do userspace stack unwinding via frame pointer walking, we'd still need to make the kernel-mode unwinder capable of taking page faults (and thus sleeping) in order to make unwinding fully robust. Let's suppose we did userspace unwinding via sleepable eBPF instead of a signal: would it be possible to accommodate, in this model, the needs of eBPF-based stack-keyed maps? In other words: it's fundamentally impossible for an eBPF program attached to a tracepoint to query a map the key of which is a complete (user+kernel) stack if, in that context, eBPF cannot sleep. This impossibility is independent of my specific proposal and is inherent to the need for any robust stack-walker, user- or kernel-mode, to be able to take page faults and thus sleep. If I'm right about this impossibility, then we have to give up on one of 1) fully robust full (user+kernel) stack walking, and 2) stack-keyed eBPF maps. Which would you prefer?
  - anarazel 4y ago
    > While it's true that user threads can't reset their signal masks without a system call, 1) SA_NODEFER means we don't have to, and 2) in theory, we could make an rseq field that would allow us to implement sigprocmask(2) without an entry into the kernel! Writing a signal handler without SA_NODEFER is ... not fun. > I think you're right, and that's unfortunate. I'm still not sure, however, whether the need to support BPF stack-map insertion is sufficient to kill the userspace-driven unwinding scheme I've described, because (at least in my experience) this BPF-map-key scenario is a pretty fringe case. I don't think it's all that fringe, but yes, it's not as common as plain profiling. The thing that makes it the most niche is the difficulty around lack of frame pointers in libc - other libraries matter far less. > If I'm right about this impossibility, then we have to give up on one of 1) fully robust full (user+kernel) stack walking, and 2) stack-keyed eBPF maps. Which would you prefer? I'd give up 1). The stack page fault issue isn't all that big a deal in practice, although I do agree that it's quite ugly. Whereas being able to build things like "offwake" profiles (showing the stack of the waiter and the wakee + duration of sleeps) allows to debug/profile issues that are extremely hard to debug otherwise. And rely quite fundamentally on stack-keyed BPF maps.