4 ms·

There is a lot of nuance missing here - virtual frame pointers can be reproduced by debug info, but it requires your tools understanding debug info. So it's no

There is a lot of nuance missing here - virtual frame pointers can be reproduced by debug info, but it requires your tools understanding debug info. So it's not "all or nothing". You can turn them off and enable separate debug info, at a cost of tools having to understand the debug info. Compilers produce good enough virtual frame pointers to do unwinding/exceptions, and debugging. They are not perfect however, and people throw rocks more than they fix things, as usual. (IE producing better info for inline assembly)

kentonv 4y ago
It seems like argument is that cross-referencing the debug info is much slower than following frame pointers, so much so that whole-system profiling becomes impractical. This is new to me, but I suppose it makes sense. You pause a thread to take a stack trace. Each frame of the trace might require reading some debug info which potentially needs to be loaded from disk. You can't predict in advance which debug info the next frame will need, because you have no idea what address it will be until you process the current frame. You can't let the thread continue execution until you're done tracing -- unless you want to clone the whole stack in the meantime? That all seems... pretty bad. If you're profiling a single binary, maybe you can fault in all the debug info in advance, but for whole-system that's probably a lot of RAM burned. (Disclaimer: I'm just speculating here, I don't personally have experience with whole-system profiling.)
- DannyBee 4y ago
  "It seems like argument is that cross-referencing the debug info is much slower than following frame pointers, so much so that whole-system profiling becomes impractical. " Think of all of this like eBPF, which can be done fast enough to work fine. First, the debug_frame sections are small separate, and self-contained. You don't have to fault in the rest of debug info to decode the frame. Second, I was, at one point heavily involved in such a thing. Without breaking confidentiality - it can be done. Also, for the larger ones, you are sampling anyway if you are using production systems the scale of facebook, because it's cheaper to sample a small percent of the time, and you will get the same coverage/results. So it is not slow all the time, nor slow on all your machines, nor ... See, e.g., https://research.google/pubs/pub45290/ https://research.google/pubs/pub45290/ This obviously does not help with running perf live, but .... Third, if you are concerned about speed, this can also be cached quite well. You can go further too - you can JIT the debug_frame section into executable code if you want, or even compile ahead of time.
ungamedplayer 4y ago
In case anyone following along can't read between the lines, fedora already does this.

kentonv 4y ago

It seems like argument is that cross-referencing the debug info is much slower than following frame pointers, so much so that whole-system profiling becomes impractical. This is new to me, but I suppose it makes sense. You pause a thread to take a stack trace. Each frame of the trace might require reading some debug info which potentially needs to be loaded from disk. You can't predict in advance which debug info the next frame will need, because you have no idea what address it will be until you process the current frame. You can't let the thread continue execution until you're done tracing -- unless you want to clone the whole stack in the meantime? That all seems... pretty bad. If you're profiling a single binary, maybe you can fault in all the debug info in advance, but for whole-system that's probably a lot of RAM burned. (Disclaimer: I'm just speculating here, I don't personally have experience with whole-system profiling.)

DannyBee 4y ago
"It seems like argument is that cross-referencing the debug info is much slower than following frame pointers, so much so that whole-system profiling becomes impractical. " Think of all of this like eBPF, which can be done fast enough to work fine. First, the debug_frame sections are small separate, and self-contained. You don't have to fault in the rest of debug info to decode the frame. Second, I was, at one point heavily involved in such a thing. Without breaking confidentiality - it can be done. Also, for the larger ones, you are sampling anyway if you are using production systems the scale of facebook, because it's cheaper to sample a small percent of the time, and you will get the same coverage/results. So it is not slow all the time, nor slow on all your machines, nor ... See, e.g., https://research.google/pubs/pub45290/ https://research.google/pubs/pub45290/ This obviously does not help with running perf live, but .... Third, if you are concerned about speed, this can also be cached quite well. You can go further too - you can JIT the debug_frame section into executable code if you want, or even compile ahead of time.

ungamedplayer 4y ago

In case anyone following along can't read between the lines, fedora already does this.