4 ms·

Are the authors here? Thanks for this! I'm always thrilled to see advances in profiling tools. I'm curious what they have to say about complexity/necessity of

Are the authors here? Thanks for this! I'm always thrilled to see advances in profiling tools. I'm curious what they have to say about complexity/necessity of interpreting all of DWARF. cargo-trace (an neat and conceptually similar but abandoned project, I think) [1] says: > It can be empirically determined that almost all dwarf programs consist of a single instruction and use only three different instructions. rip+offset, rsp+offset or *cfa+offset, where cfa is the rsp value of the previous frame. The result of the unwinding is an array of instruction pointers. Do you find this to be true? Is more complex interpreting of DWARF necessary? And in the lkml thread linked from the article, Linus is extremely pessimistic about DWARF unwinding, [2] I'm sure not without justification. He's talking about kernel stacks, and I think the trade-off is different when you're trying to profile existing userspace applications and libraries compiled and implemented however, but nonetheless I'm curious to hear the authors say how applicable they think his points are. The authors' pointer to .ctf_frame is also a very interesting development in this space. Not sure how long that will take to be widely usable (assuming it goes through at all). [1] https://github.com/dvc94ch/cargo-trace https://github.com/dvc94ch/cargo-trace [2] https://lkml.org/lkml/2012/2/10/356 https://lkml.org/lkml/2012/2/10/356

javierhonduco 4y ago
Author here! This project looks interesting and as you mention it seems conceptually similar! I would agree with that quote, generally speaking, but I think it really depends on a binary to binary basis. Some binaries outsource most of the work to dynamic libraries. Unfortunately, DWARF expressions are typically emitted for these program counter ranges, so it's desirable to at least implement a subset of expressions [0]. Even if that's not the case, we want to produce profiles that are as accurate as possible :) You are totally right, they were discussing kernel stacks where stakes are higher as it needs to work perfectly, otherwise kernel live patching would not reliably work, among others. The kernel has now some unwind table format, that can be used in x86_64, called ORC [1]. That being said the parser for DWARF would still have to live in the kernel, and I am not sure if kernel devs would like to accept such a patch. Ideally we would transition to an unwind-specific format for user-space (something like ours, for example) and perhaps have a suitable unwinder in the kernel, rather than having to implement it in a BPF program. This is something we are considering for the future, but it's not free from problems (increased executable size, redundant unwind information, etc). But this is exactly why we wanted to have a conversation with the communities interested in this work! [0]: https://github.com/parca-dev/parca-agent/pull/1058/commits https://github.com/parca-dev/parca-agent/pull/1058/commits [1]: https://lwn.net/Articles/728339/ https://lwn.net/Articles/728339/
- scottlamb 4y ago
  > Ideally we would transition to an unwind-specific format for user-space (something like ours, for example) and perhaps have a suitable unwinder in the kernel, rather than having to implement it in a BPF program. This is something we are considering for the future, but it's not free from problems (increased executable size, redundant unwind information, etc). But this is exactly why we wanted to have a conversation with the communities interested in this work! I'm just an interested user rather than someone who has an ownership stake in any major distribution, language, library, or application, but fwiw: I'd love for every binary/library to embed enough information to cheaply unwind during profiling [edit: and symbolize shortly afterward], in one simple format, so that this sort of profiling tool Just Works, as well as backtraces generated by the program itself (e.g. Rust's panics). I'd prefer it support inlining well (so ideally not just turning on frame pointers everywhere but that'd still be a huge improvement over the status quo). As you know, this is all a complete mess now, and I hate that. I think a reasonable amount of space overhead for this is fine. I want to be able to profile pre-packaged applications and my own Rust applications that use some C libraries without hassle. And have profiles written in a format that I can ship to another machine without trouble reading the symbols there or concern about excessive sensitive information. All stuff you've touched on. I would consider it a genuine miracle if the parties involved could all get behind this. Full DWARF information, e.g. Rust's debug = 2 [1] (to be clear: my understanding is this is more or less to allow debuggers to print variable contents reasonably well?) is another matter. Seems like that adds a huge amount of bloat and is more rarely used. I like split debuginfo. As long as there's an easy way to get it on demand via a debuginfod service and/or installing an additional package, it's fine. [1] https://doc.rust-lang.org/cargo/reference/profiles.html https://doc.rust-lang.org/cargo/reference/profiles.html
  - javierhonduco 4y ago
    > I'd love for every binary/library to embed enough information to cheaply unwind during profiling Definitely! We hope to be able to change this. There's the .ctf_frame format [0] that could be used too. While its design goals are very much aligned to ours, implementing a reader for this format in BPF will be complicated. > and symbolize shortly afterward Definitely, I think that C Type Format (CTF) [1] could be a candidate here. Another possibility would be to get BTF (BPF Type Format) [2] working in userspace. While it wouldn't cover everything that DWARF deals with, it might be a reasonable tradeoff for symbolization. FWIW both in Polar Signals and Parca (our OSS offering) we resolve to async symbolization done in the server, as it's very expensive to do and we want to reduce load where the Agent runs. [0]: https://gcc.gnu.org/wiki/cauldron2022talks https://gcc.gnu.org/wiki/cauldron2022talks [1]: https://lwn.net/Articles/795384/#:~:text=The%20Compact%20C%20Type%20Format%20(CTF)%20is%20a%20way%20of,use%20in%20the%20Linux%20world https://lwn.net/Articles/795384/#:~:text=The%20Compact%20C%2.... [2]: https://docs.kernel.org/bpf/btf.html https://docs.kernel.org/bpf/btf.html
jcalabro 4y ago
I've written a DWARF expression parser and stack unwinder and its not true that those three use-cases are enough to be robust [0]. Based on my own observations, you need to implement: - undefined - same value - offset(N) - val_offset(N) - register(N) I've never come across expression(E), val_expression(E), or architectural. But you definitely do need the above in the common cases. More generally, I'm kinda confused by that quote, I think the author may be conflating DWARF expressions and stack unwinding a bit? What I think they might have been getting at is that with DWARF expression evaluation [1] in particular (i.e. not just stack unwinding), I've never even seen +80% of the opcodes I've implemented in the wild across several programming languages and compilers. I also left about 1/4 of them unimplemented and I've never had an issue. The intention of the DWARF spec writers was to provide folks with a stack machine that was Turing-complete, but in practice, yeah, nobody really takes advantage of it because there's not really a need. [0] https://dwarfstd.org/doc/DWARF5.pdf https://dwarfstd.org/doc/DWARF5.pdf (section 6.4) [1] https://dwarfstd.org/doc/DWARF5.pdf https://dwarfstd.org/doc/DWARF5.pdf (section 7.7)
- scottlamb 4y ago
  > More generally, I'm kinda confused by that quote, I think the author may be conflating DWARF expressions and stack unwinding a bit? I suspect they only cared about / looked at the portion of DWARF information needed for stack unwinding. I'd read the quote with that implicit qualifier after "almost all dwarf programs".
  - jcalabro 4y ago
    Ah gotcha, in that case, as I said, I've never come across a DWARF expression while doing stack unwinding in the wild. Just registers and offsets :)
    - javierhonduco 4y ago
      I find this surprising! Was this for off the shelf applications or some custom binaries? We see DWARF expressions such as `DW_CFA_def_cfa_expression` on the regular. See the "Test Plan" section and commit messages of the PR that introduced support for this particular opcode, as well as the simple executable that shows this opcode [0] and its printed unwind table [1] [0]: https://github.com/parca-dev/parca-agent/pull/1058 https://github.com/parca-dev/parca-agent/pull/1058 [1]: https://github.com/parca-dev/testdata/blob/7f9206570a743206fdb6190d6240ced355b9e523/tables/ours_basic-cpp-plt.txt#L12 https://github.com/parca-dev/testdata/blob/7f9206570a743206f...

javierhonduco 4y ago

Author here! This project looks interesting and as you mention it seems conceptually similar! I would agree with that quote, generally speaking, but I think it really depends on a binary to binary basis. Some binaries outsource most of the work to dynamic libraries. Unfortunately, DWARF expressions are typically emitted for these program counter ranges, so it's desirable to at least implement a subset of expressions [0]. Even if that's not the case, we want to produce profiles that are as accurate as possible :) You are totally right, they were discussing kernel stacks where stakes are higher as it needs to work perfectly, otherwise kernel live patching would not reliably work, among others. The kernel has now some unwind table format, that can be used in x86_64, called ORC [1]. That being said the parser for DWARF would still have to live in the kernel, and I am not sure if kernel devs would like to accept such a patch. Ideally we would transition to an unwind-specific format for user-space (something like ours, for example) and perhaps have a suitable unwinder in the kernel, rather than having to implement it in a BPF program. This is something we are considering for the future, but it's not free from problems (increased executable size, redundant unwind information, etc). But this is exactly why we wanted to have a conversation with the communities interested in this work! [0]: https://github.com/parca-dev/parca-agent/pull/1058/commits https://github.com/parca-dev/parca-agent/pull/1058/commits [1]: https://lwn.net/Articles/728339/ https://lwn.net/Articles/728339/

scottlamb 4y ago
> Ideally we would transition to an unwind-specific format for user-space (something like ours, for example) and perhaps have a suitable unwinder in the kernel, rather than having to implement it in a BPF program. This is something we are considering for the future, but it's not free from problems (increased executable size, redundant unwind information, etc). But this is exactly why we wanted to have a conversation with the communities interested in this work! I'm just an interested user rather than someone who has an ownership stake in any major distribution, language, library, or application, but fwiw: I'd love for every binary/library to embed enough information to cheaply unwind during profiling [edit: and symbolize shortly afterward], in one simple format, so that this sort of profiling tool Just Works, as well as backtraces generated by the program itself (e.g. Rust's panics). I'd prefer it support inlining well (so ideally not just turning on frame pointers everywhere but that'd still be a huge improvement over the status quo). As you know, this is all a complete mess now, and I hate that. I think a reasonable amount of space overhead for this is fine. I want to be able to profile pre-packaged applications and my own Rust applications that use some C libraries without hassle. And have profiles written in a format that I can ship to another machine without trouble reading the symbols there or concern about excessive sensitive information. All stuff you've touched on. I would consider it a genuine miracle if the parties involved could all get behind this. Full DWARF information, e.g. Rust's debug = 2 [1] (to be clear: my understanding is this is more or less to allow debuggers to print variable contents reasonably well?) is another matter. Seems like that adds a huge amount of bloat and is more rarely used. I like split debuginfo. As long as there's an easy way to get it on demand via a debuginfod service and/or installing an additional package, it's fine. [1] https://doc.rust-lang.org/cargo/reference/profiles.html https://doc.rust-lang.org/cargo/reference/profiles.html
- javierhonduco 4y ago
  > I'd love for every binary/library to embed enough information to cheaply unwind during profiling Definitely! We hope to be able to change this. There's the .ctf_frame format [0] that could be used too. While its design goals are very much aligned to ours, implementing a reader for this format in BPF will be complicated. > and symbolize shortly afterward Definitely, I think that C Type Format (CTF) [1] could be a candidate here. Another possibility would be to get BTF (BPF Type Format) [2] working in userspace. While it wouldn't cover everything that DWARF deals with, it might be a reasonable tradeoff for symbolization. FWIW both in Polar Signals and Parca (our OSS offering) we resolve to async symbolization done in the server, as it's very expensive to do and we want to reduce load where the Agent runs. [0]: https://gcc.gnu.org/wiki/cauldron2022talks https://gcc.gnu.org/wiki/cauldron2022talks [1]: https://lwn.net/Articles/795384/#:~:text=The%20Compact%20C%20Type%20Format%20(CTF)%20is%20a%20way%20of,use%20in%20the%20Linux%20world https://lwn.net/Articles/795384/#:~:text=The%20Compact%20C%2.... [2]: https://docs.kernel.org/bpf/btf.html https://docs.kernel.org/bpf/btf.html

jcalabro 4y ago

I've written a DWARF expression parser and stack unwinder and its not true that those three use-cases are enough to be robust [0]. Based on my own observations, you need to implement: - undefined - same value - offset(N) - val_offset(N) - register(N) I've never come across expression(E), val_expression(E), or architectural. But you definitely do need the above in the common cases. More generally, I'm kinda confused by that quote, I think the author may be conflating DWARF expressions and stack unwinding a bit? What I think they might have been getting at is that with DWARF expression evaluation [1] in particular (i.e. not just stack unwinding), I've never even seen +80% of the opcodes I've implemented in the wild across several programming languages and compilers. I also left about 1/4 of them unimplemented and I've never had an issue. The intention of the DWARF spec writers was to provide folks with a stack machine that was Turing-complete, but in practice, yeah, nobody really takes advantage of it because there's not really a need. [0] https://dwarfstd.org/doc/DWARF5.pdf https://dwarfstd.org/doc/DWARF5.pdf (section 6.4) [1] https://dwarfstd.org/doc/DWARF5.pdf https://dwarfstd.org/doc/DWARF5.pdf (section 7.7)

scottlamb 4y ago
> More generally, I'm kinda confused by that quote, I think the author may be conflating DWARF expressions and stack unwinding a bit? I suspect they only cared about / looked at the portion of DWARF information needed for stack unwinding. I'd read the quote with that implicit qualifier after "almost all dwarf programs".
- jcalabro 4y ago
  Ah gotcha, in that case, as I said, I've never come across a DWARF expression while doing stack unwinding in the wild. Just registers and offsets :)
  - javierhonduco 4y ago
    I find this surprising! Was this for off the shelf applications or some custom binaries? We see DWARF expressions such as `DW_CFA_def_cfa_expression` on the regular. See the "Test Plan" section and commit messages of the PR that introduced support for this particular opcode, as well as the simple executable that shows this opcode [0] and its printed unwind table [1] [0]: https://github.com/parca-dev/parca-agent/pull/1058 https://github.com/parca-dev/parca-agent/pull/1058 [1]: https://github.com/parca-dev/testdata/blob/7f9206570a743206fdb6190d6240ced355b9e523/tables/ours_basic-cpp-plt.txt#L12 https://github.com/parca-dev/testdata/blob/7f9206570a743206f...