5 ms·
Limitations of frame pointer unwinding
- dap 2y agoThis reads to me like FUD. Isn’t the fraction of profile samples in a prologue heavily workload dependent? And whichever way you go on frame pointers, there are winners and losers to including them by default.
- thegeomaster 2y agoThere are no performance winners if you include them by default. There will be an additional >0% overhead when you are executing additional code in the prologue and epilogue, and increasing the register pressure by removing rbp from being ever allocated. There are only "winners" in the sense that people will be able to more easily see why their never-tuned system is so slow. On the other hand, you're punishing all perf-critical usecases with unnecessary overhead. I believe if you have a slow system, it's up to you to profile and optimize it, and that includes even recompiling some software with different flags to enable profiling. It's not the job of upstream to make this easier for you if it means punishing those workloads where teams have diligently profiled and optimized through the years so that there is no, as the author says, low-hanging fruit to find.
- dap 2y agoI’ve been around long enough to have had frame pointers pretty ubiquitously, then lost them, and now starting to have them again. The dark times in the middle were painful. For the software I’ve worked on, the easy dynamic profiling using frame pointers (eg using DTrace) has given far more in performance wins than omitting them would have. (Part of my beef with the article is that while edge cases do break some samples, in practice it’s a very small fraction, and almost by definition not the important ones if you’re trying to find heavy on-CPU code paths.) I get that some use cases may be better without frame pointers. A well-resourced team can always recompile the world, whichever the default is. It’s just that my experience is that most software is not already perfectly tuned and I’d much rather the default be more easily observable.
- thegeomaster 2y agoLook, it's likely we just come from different backgrounds. Most of my perf-sensitive work was optimizing inner loops with SIMD, allowing the compiler to inline hot functions, creating better data structures to make use of the CPU cache, etc. Frame pointer prologue overhead was measurable on most of our use-cases. I have a smaller amount of experience on profiling systems where calls trace across multiple processes, so maybe I haven't felt this pain enough. Though I still think the onus should be on teams to be able to comfortably recompile---not the world---but some part of it. After all, a lot of tuning can only be done through compile flags, such as turning off codepaths/capabilities which are unnecessary.
- dap 2y agoMakes sense. I wasn't exaggerating about recompiling the world, though. Even if we say I'm only interested in profiling my application, a single library compiled without frame pointers makes useless any samples where code in that library was at the top of the stack. I've seen that be libc, openssl, some random Node module or JNI thing, etc. You can't just throw out those samples because they might still be your application's problem. For me in those situations, I would have needed to recompile most of the packages we got from both the OS distro and the supplemental package repo.
- audidude 2y agoI think your viewpoint is valid. My experience is on performance tuning the other side you mention. Cross-application, cross-library, whole-system, daemons, etc. Basically, "the whole OS as it's shipped to users". For my case, I need the whole system setup correctly before it even starts to be useful. For your case, you only need the specific library or application compiled correctly. The rest of the system is negligible and probably not even used. Who would optimize SIMD routines next to function calls anyway?
- audidude 2y agoI added support to Sysprof this weekend for unwinding using libdwfl and DWARF/CFI/eh_frame/etc techniques that Serhei did in eu-stacktrace. The overhead is about 10% of samples. But at least you can unwind on systems without frame-pointers. Personally I'll take the statistical anomalies of frame-pointers which still allow you to know what PID/TID are your cost center even if you don't get perfect unwinds. Everyone seems motivated towards SFrame going forward, which is good. https://blogs.gnome.org/chergert/2024/11/03/profiling-w-o-frame-pointers/ https://blogs.gnome.org/chergert/2024/11/03/profiling-w-o-fr...
- Brian_K_White 2y agoIs this a response to Alma Kitten? In any event I don't understand why frame pointers need to be in by default instead of developers enabling where needed. Having Kitten include pointers by default seems reasonable enough, since Kitten is a devel system.
- ithkuil 2y agoIt's useful to be able to profile on production workloads
- rwmj 2y agoThe real benefit is being able to turn on profiling when a problem is spotted, or in some cases to be able to profile continuously in production (as apparently they do at Netflix).
- thegeomaster 2y agoI get it. This frustrated me to no end. But still I did what I had to do --- recompiled random software throughout the stack, enabled random flags, etc. It was doable and now I can do it much faster. I don't think it's fair for upstream to disable a useful optimization just so I don't have to do this additional work to fix and optimize my system.
- rwmj 2y agoDoing real world, whole system profiling, we've found performance was affected by completely unexpected software running on the system. Recompiling the entire distribution, or even the subset of all software installed, is not realistic for most people. Besides, I have measured the overhead of frame pointers and it's less than 1%, so there's not really any trade-off here. Anyway, soon we'll have SFrame support in the userspace tools and the whole issue will go away.
- thegeomaster 2y agoIn one of my jobs, a 1% perf regression (on a more stable/reproducible system, not PCs) was a reason for a customer raising a ticket, and we'd have to look into it. For dynamically dispatched but short functions, the overhead is easily more than 1% too. So, there is a trade-off, just not one that affects you.
- fooblaster 2y agoI have always had issues with the perf call trace sampling with frame pointers, even when virtually everything in userspace compiled with fno-omit-frame-pointer. It doesn't look like any of the failure modes listed in the article to me though. Shrug. FYI, if you happen to be running on an intel cpu, --call-graph lbr uses some specicalized hardware and often delivers a far superior result, with some notable failure modes. Really looking forward to when AMD implements a similar feature.
- rwmj 2y agoThe problem with Intel LBR (last branch records) is that the depth of the call stack is relatively limited. It depends on the generation of CPU, but LWN has a table here: https://lwn.net/Articles/680985/ https://lwn.net/Articles/680985/ Anything less than 32 is fairly useless for profiling from the kernel through to userspace.
- fooblaster 2y agoYeah, right. I still seem to get far more "comprehensible" traces when using it, even with this limitation. It's often really easy to localize where a trace is coming from, even when truncated. It probably breaks flamegraphs though.
- Sesse__ 2y agoI've tried --call-graph lbr a bunch of times, but often, it… just returns junk? I don't fully understand why, it sometimes returns wild pointers even if you don't have deep stacks.
- fooblaster 2y agoI often get junk when sampling without lbr. Which kernel are you running? The quality of perf and the associated perf_events varies wildly across kernel versions.
- Sesse__ 2y ago
- laserbeam 2y ago"enabling frame pointers is a 1-2% performance loss, which translates to the loss of about 1 or 2 years of compiler improvements" Wait, are we really that close to the maximum of what a compiler can optimize that we're getting barely 1% performance improvements per year with new versions?
- clausecker 2y agoYeah, compilers are already pretty close to the limit of what is possible, unless your code is unusually poorly written.
- londons_explore 2y agoClearly this isn't the case. Plenty of neat C++ "reference implementation" code ends up 5x faster when hand optimized, parallelized, vectorized, etc. There are some transformations that compilers are really bad at. Rearranging data structures, switching out algorithms for equivalent ones with better big-O complexity, generating & using lookup tables, bit-packing things, using caches, hash tables and bloom filters for time/memory trade offs, etc. The spec doesn't prevent such optimizations, but current compilers aren't smart enough to find them.
- adrianN 2y agoImagine the outcry if compilers switched algorithms. How can the compiler know my input size and input distribution? Maybe my dumb algorithm is optimal for my data.
- londons_explore 2y agoCompilers can easily runtime-detect the size and shape of the problem, and run different code for different problem sizes. Many already do for loop-unrolling. Ie. if you memcpy 2 bytes, they won't even branch into the fancy SIMD version. This would just be an extension of that. If the code creates and uses a linked list, yet the list is 1M items long and being accessed entirely by index, branch to a different version of the code which uses an array, etc.
- clausecker 2y agoThe “function prologue is at least 8 bytes long” bit only applies if CET is used. If it is not used, the endbr64 instruction is not emitted and the prologue is only 4 bytes long.
- elteto 2y agoDidn’t really get the point of the post as it just presents something without a conclusion. 9X% of users do not care about a <1% drop in performance. I suspect we get the same variability just by going from one kernel version to another. The impact from all the Intel mitigations that are now enabled by default is much worse. However I do care about nice profiles and stack traces without having to jump through hoops. Asking people to recompile an _entire_ distribution just to get sane defaults is wrong. Those who care about the last drop should build their custom systems as they see fit, and they probably already do.
- yosefk 2y agoit does present a conclusion. once the kernel supports .sframe it will be all-around superior to -fomit-frame-pointer, and a better default for distros to use.
- audidude 2y agoIt does cause more memory pressure because the kernel will have to look at the user-space memory for decoding registers. So yes it will be faster than alternatives to frame-pointers, but it still wont be as fast as frame pointers.
- Brian_K_White 2y agoBut does what you care about matter enough to be the default? Are you the majority? Evaluate "majority" this way: For every/any random binary in a distro, out of all the currently running instances of that binary in the world at any given moment, how many of those need to be profiled? There is no way the answer is "most of them". You have a job where you profile things, and maybe even you profile almost everything you touch. Your whole world has a high quotient of profiling in it. So you want the whole system built for profiling by default. How convenient for you. But your whole world is not the whole world. But it's not just you, there are, zomg thousands, tens of thousands, maybe even hundreds of thousands of developers and ops admins the same as you. Yes and? Is even that most installed instances of any given executable? No way. Or maybe yes. It's possible. Can you show that somehow? But I will guess no way and not even close.
- jeffbee 2y agoComplaining about frame pointers is like complaining about the budget of the Bureau of Labor Statistics. Yes, it's pure overhead, but also yes, it's good to know what is going on.
- ot 2y agoI broadly agree with the thesis of the post, which if I understand correctly is that frame pointers are a temporary compromise until the whole ecosystem gets its act together and manages to agree on some form of out-of-band tracking of frame pointers, and it seems that we'll eventually get there. Some of the statements in the post seem odd to me though. - 5% of system-wide cycles spent in function prologues/epilogues? That is wild, it can't be right. - Is using the whole 8 bytes right for the estimate? Pushing the stack pointer is the first instruction in the prologue and it's literally 1 byte. Epilogue is symmetrical. - Even if we're in the prologue, we know that we're in a leaf call, we can still resolve the instruction pointer to the function, and we can read the return address to find the parent, so what information is lost? When it comes to future alternatives, while frame pointers have their own problems, I think that there are still a few open questions: - Shadow stacks are cool but aren't they limited to a fixed number of entries? What if you have a deeper stack? - Is the memory overhead of lookup tables for very large programs acceptable?
- audidude 2y ago> Shadow stacks are cool but aren't they limited to a fixed number of entries? Current available hardware yes. But I think some of the future Intel stuff was going to allow for much larger depth. > Is the memory overhead of lookup tables for very large programs acceptable? I don't think SFrame is as "dense" as DWARF as a format so you trade a bit of memory size for a much faster unwind experience. But you are definitely right that this adds memory pressure that could otherwise be ignored. Especially if the anomalies are what they sound like, just account for them statistically. You get a PID for cost accounting in the perf_event frame anyway.
- rwmj 2y ago> - Is using the whole 8 bytes right for the estimate? Pushing the stack pointer is the first instruction in the prologue and it's literally 1 byte. Epilogue is symmetrical. I believe it's because of the landing pad for Control Flow Integrity which basically all functions now need. Grabbing main() from a random program on Fedora (which uses frame pointers): 0000000000007000 <main>: 7000: f3 0f 1e fa endbr64 ; landing pad 7004: 55 push %rbp ; set up frame pointer 7005: 48 89 e5 mov %rsp,%rbp It's not much of an issue in practice as the stack trace will still be nearly correct, enough for you to identify the problematic area of the code. > - Shadow stacks are cool but aren't they limited to a fixed number of entries? What if you have a deeper stack? Yes shadow stacks are limited to 32 entries on the most recent Intel CPUs (and as little as 4 entries on very old ones). However they are basically cost free so that's a big advantage. I think SFrame is a sensible middle ground here. It's saner than DWARF and has a long history of use in the kernel so we know it will work.
- tempfile 2y agoThe JS constantly grabbing the anchor and updating it is absolutely appalling UX. It took me something like 11 back button presses to get back to where I was. Borderline malware.
- LoopingBack 2y ago[dead]
- 0908400383 2y ago[flagged]
- 0908400383 2y ago[flagged]
- dingi 2y agoIsn't this just another case of RH (and Canonical to some extent) screwing Fedora users to make their paid offerings better? As far as I understand, this is not gonna be enabled by default on RHEL. If this is a must, it shouldn't be harder to produce two different builds or two different ISOs. People who needs profiling can grab the ISO with profiling enabled. Us regular folks can use the one without.
- michealliam320 2y ago[dead]
- jenda23 2y agoIt was like a dream to the point of selling off my properties in order to invest into binary option. I met these investors online, at first we shared business or investment ideas and it got to a point we started discussing business proposal until i got confused and convinced into investing with them in binary option. I didn't want to believe i was being scammed at first until i wanted to withdraw my profits into my account. The problem got to a point that i was blocked from logging in to communicate with the site. My $120,000 was lost to online scammers with the believe i was investing into binary options in order to make grand profits. I searched for help but none came until i discussed it with my Mom whom got me to know about Rewallet Coin Recovery which had helped her friend with the recovery of her lost funds. I contacted them and to my utmost surprise all my lost Bitcoin was recovered and the funds transferred into my account. I can call them experts in what they do, i got marveled because it was achieved within 72 hours and still grateful to Rewallet. Contact email, rewalletshieldcoinrecovery@aol.com or WhatsApp::+1 (757) 332-1885