4 ms·

Re signals, see https://man7.org/linux/man-pages/man2/sigreturn.2.html https://man7.org/linux/man-pages/man2/sigreturn.2.html - you'd at least end up with that.

Re signals, see https://man7.org/linux/man-pages/man2/sigreturn.2.html https://man7.org/linux/man-pages/man2/sigreturn.2.html - you'd at least end up with that. Re bpf: I don't think you can just defer the insertion into maps, that'd block execution, as code can depend an the contents of the bpf maps before continuing on with the bpf program. Iirc for bpf based tracing, the evaluation happens synchronously at the insertion points.

quotemstr 4y ago
> Re signals, see https://man7.org/linux/man-pages/man2/sigreturn.2.html https://man7.org/linux/man-pages/man2/sigreturn.2.html - you'd at least end up with that. Not necessarily. sigreturn doesn't do anything that userspace can't do on its own. It's kind of an ugly vestige of the days when system calls were "no big deal" instead of the performance-killers they are today. While it's true that user threads can't reset their signal masks without a system call, 1) SA_NODEFER means we don't have to, and 2) in theory, we could make an rseq field that would allow us to implement sigprocmask(2) without an entry into the kernel! > Re bpf: I don't think you can just defer the insertion into maps, that'd block execution, as code can depend an the contents of the bpf maps before continuing on with the bpf program. Iirc for bpf based tracing, the evaluation happens synchronously at the insertion points. I think you're right, and that's unfortunate. I'm still not sure, however, whether the need to support BPF stack-map insertion is sufficient to kill the userspace-driven unwinding scheme I've described, because (at least in my experience) this BPF-map-key scenario is a pretty fringe case. Let's look at it this way: is it possible in principle to reconcile a solution to the pagefault problem with the needs of BPF stack-hash map keys? Even if I were to concede that userspace stack unwinding belonged in the kernel, and even if I were to additionally concede that we should do userspace stack unwinding via frame pointer walking, we'd still need to make the kernel-mode unwinder capable of taking page faults (and thus sleeping) in order to make unwinding fully robust. Let's suppose we did userspace unwinding via sleepable eBPF instead of a signal: would it be possible to accommodate, in this model, the needs of eBPF-based stack-keyed maps? In other words: it's fundamentally impossible for an eBPF program attached to a tracepoint to query a map the key of which is a complete (user+kernel) stack if, in that context, eBPF cannot sleep. This impossibility is independent of my specific proposal and is inherent to the need for any robust stack-walker, user- or kernel-mode, to be able to take page faults and thus sleep. If I'm right about this impossibility, then we have to give up on one of 1) fully robust full (user+kernel) stack walking, and 2) stack-keyed eBPF maps. Which would you prefer?
- anarazel 4y ago
  > While it's true that user threads can't reset their signal masks without a system call, 1) SA_NODEFER means we don't have to, and 2) in theory, we could make an rseq field that would allow us to implement sigprocmask(2) without an entry into the kernel! Writing a signal handler without SA_NODEFER is ... not fun. > I think you're right, and that's unfortunate. I'm still not sure, however, whether the need to support BPF stack-map insertion is sufficient to kill the userspace-driven unwinding scheme I've described, because (at least in my experience) this BPF-map-key scenario is a pretty fringe case. I don't think it's all that fringe, but yes, it's not as common as plain profiling. The thing that makes it the most niche is the difficulty around lack of frame pointers in libc - other libraries matter far less. > If I'm right about this impossibility, then we have to give up on one of 1) fully robust full (user+kernel) stack walking, and 2) stack-keyed eBPF maps. Which would you prefer? I'd give up 1). The stack page fault issue isn't all that big a deal in practice, although I do agree that it's quite ugly. Whereas being able to build things like "offwake" profiles (showing the stack of the waiter and the wakee + duration of sleeps) allows to debug/profile issues that are extremely hard to debug otherwise. And rely quite fundamentally on stack-keyed BPF maps.
  - quotemstr 4y ago
    You don't need eBPF maps for wake-up profiles. If you just log wakeups the usual way, postprocessing tools get all the information they need to figure out what woke what. That people use eBPF maps for this purpose is a consequence of the inadequacy of Linux profiling tools.
    - anarazel 4y ago
      IME that doesn't work, because the un-aggregated data volumes are far too high. The whole point of using something like eBPF is that you don't need to log all that data.
      - quotemstr 4y ago
        Is data volume too high? I don't see it. Windows and Android both collect useful traces and do post-hoc analysis on them instead of using things like eBPF stack maps to do online analysis.

quotemstr 4y ago

> Re signals, see https://man7.org/linux/man-pages/man2/sigreturn.2.html https://man7.org/linux/man-pages/man2/sigreturn.2.html - you'd at least end up with that. Not necessarily. sigreturn doesn't do anything that userspace can't do on its own. It's kind of an ugly vestige of the days when system calls were "no big deal" instead of the performance-killers they are today. While it's true that user threads can't reset their signal masks without a system call, 1) SA_NODEFER means we don't have to, and 2) in theory, we could make an rseq field that would allow us to implement sigprocmask(2) without an entry into the kernel! > Re bpf: I don't think you can just defer the insertion into maps, that'd block execution, as code can depend an the contents of the bpf maps before continuing on with the bpf program. Iirc for bpf based tracing, the evaluation happens synchronously at the insertion points. I think you're right, and that's unfortunate. I'm still not sure, however, whether the need to support BPF stack-map insertion is sufficient to kill the userspace-driven unwinding scheme I've described, because (at least in my experience) this BPF-map-key scenario is a pretty fringe case. Let's look at it this way: is it possible in principle to reconcile a solution to the pagefault problem with the needs of BPF stack-hash map keys? Even if I were to concede that userspace stack unwinding belonged in the kernel, and even if I were to additionally concede that we should do userspace stack unwinding via frame pointer walking, we'd still need to make the kernel-mode unwinder capable of taking page faults (and thus sleeping) in order to make unwinding fully robust. Let's suppose we did userspace unwinding via sleepable eBPF instead of a signal: would it be possible to accommodate, in this model, the needs of eBPF-based stack-keyed maps? In other words: it's fundamentally impossible for an eBPF program attached to a tracepoint to query a map the key of which is a complete (user+kernel) stack if, in that context, eBPF cannot sleep. This impossibility is independent of my specific proposal and is inherent to the need for any robust stack-walker, user- or kernel-mode, to be able to take page faults and thus sleep. If I'm right about this impossibility, then we have to give up on one of 1) fully robust full (user+kernel) stack walking, and 2) stack-keyed eBPF maps. Which would you prefer?

anarazel 4y ago
> While it's true that user threads can't reset their signal masks without a system call, 1) SA_NODEFER means we don't have to, and 2) in theory, we could make an rseq field that would allow us to implement sigprocmask(2) without an entry into the kernel! Writing a signal handler without SA_NODEFER is ... not fun. > I think you're right, and that's unfortunate. I'm still not sure, however, whether the need to support BPF stack-map insertion is sufficient to kill the userspace-driven unwinding scheme I've described, because (at least in my experience) this BPF-map-key scenario is a pretty fringe case. I don't think it's all that fringe, but yes, it's not as common as plain profiling. The thing that makes it the most niche is the difficulty around lack of frame pointers in libc - other libraries matter far less. > If I'm right about this impossibility, then we have to give up on one of 1) fully robust full (user+kernel) stack walking, and 2) stack-keyed eBPF maps. Which would you prefer? I'd give up 1). The stack page fault issue isn't all that big a deal in practice, although I do agree that it's quite ugly. Whereas being able to build things like "offwake" profiles (showing the stack of the waiter and the wakee + duration of sleeps) allows to debug/profile issues that are extremely hard to debug otherwise. And rely quite fundamentally on stack-keyed BPF maps.
- quotemstr 4y ago
  You don't need eBPF maps for wake-up profiles. If you just log wakeups the usual way, postprocessing tools get all the information they need to figure out what woke what. That people use eBPF maps for this purpose is a consequence of the inadequacy of Linux profiling tools.
  - anarazel 4y ago
    IME that doesn't work, because the un-aggregated data volumes are far too high. The whole point of using something like eBPF is that you don't need to log all that data.
    - quotemstr 4y ago
      Is data volume too high? I don't see it. Windows and Android both collect useful traces and do post-hoc analysis on them instead of using things like eBPF stack maps to do online analysis.