7 ms·

What everybody in this discussion seems to miss is that you don't need to unwind the DWARF data structures during profiling time, you are free to convert DWARF

What everybody in this discussion seems to miss is that you don't need to unwind the DWARF data structures during profiling time, you are free to convert DWARF to a fast-lookup data structure on the machine. DWARF needs to support every CPU under the sun. Every unwinder on the other hand is CPU-specific. For prodfiler.com's continuous in-production unwinding, we convert DWARF into something compact and fast-to-lookup that is then placed in eBPF maps. It all works like a charm. We can have our cake (e.g. use RBP as GPR) and eat it too (e.g. use .eh_frame, converted at runtime into a fast-to-lookup format) to do reliable whole-system unwinding in production.

irogers 4y ago
prodfiler clearly has a market. It would be interesting to see the approach as something standard in the kernel tree, perhaps it can be added to perf's synthesis, etc. There is already BPF based profiling within perf to avoid file descriptor overheads. If engineering resources are the issue then this could be a good GSoC project: https://wiki.linuxfoundation.org/gsoc/2023-gsoc-perf https://wiki.linuxfoundation.org/gsoc/2023-gsoc-perf
- javierhonduco 4y ago
  This would be ideal. There's some great work by folks at Oracle in this space: SFrame (https://www.phoronix.com/news/GNU-Binutils-SFrame https://www.phoronix.com/news/GNU-Binutils-SFrame) née ctf_frame that I hope will be integrated in the kernel. As this will take few years, in the meantime I've developed a DWARF-based unwinder in BPF [0]. Some perf maintainers showed interest in this, so thanks for bringing up the GSoC project idea, didn't occur to me! [0]: https://news.ycombinator.com/item?id=33788794 https://news.ycombinator.com/item?id=33788794
- tdullien 4y ago
  Yeah. I really like the ideas proposed by Brendan Gregg -- essentially encouraging every HLL runtime to embed an eBPF-based unwinder in it's own executable. The upshot of that would be "generic, in-production unwinding of native code and HLL code", similar to what prodfiler is doing, but inside the main kernel tree...

irogers 4y ago

prodfiler clearly has a market. It would be interesting to see the approach as something standard in the kernel tree, perhaps it can be added to perf's synthesis, etc. There is already BPF based profiling within perf to avoid file descriptor overheads. If engineering resources are the issue then this could be a good GSoC project: https://wiki.linuxfoundation.org/gsoc/2023-gsoc-perf https://wiki.linuxfoundation.org/gsoc/2023-gsoc-perf

javierhonduco 4y ago
This would be ideal. There's some great work by folks at Oracle in this space: SFrame (https://www.phoronix.com/news/GNU-Binutils-SFrame https://www.phoronix.com/news/GNU-Binutils-SFrame) née ctf_frame that I hope will be integrated in the kernel. As this will take few years, in the meantime I've developed a DWARF-based unwinder in BPF [0]. Some perf maintainers showed interest in this, so thanks for bringing up the GSoC project idea, didn't occur to me! [0]: https://news.ycombinator.com/item?id=33788794 https://news.ycombinator.com/item?id=33788794
tdullien 4y ago
Yeah. I really like the ideas proposed by Brendan Gregg -- essentially encouraging every HLL runtime to embed an eBPF-based unwinder in it's own executable. The upshot of that would be "generic, in-production unwinding of native code and HLL code", similar to what prodfiler is doing, but inside the main kernel tree...