17 ms·
Ubuntu 24.04 LTS will enable frame pointers by default
- dmpk2k 3y agoThank god. AMD making the base pointer optional in the x86-64 ABI was foolish.
- brancz 3y agoFrame pointers are such a destructive micro-optimization to omit by default, I am beyond excited about this collaboration with the folks at Canonical to make Ubuntu debuggable by default!
- account42 3y agoIt's a "micro" optimization that can be automatically applied everywhere. And its not destructive because debug info contains all you need to calculate the frame pointers after the fact. Really a no-brainer to have it on unless you are dealing with broken tools.
- brancz 3y agoWe've "fixed"[1] all the tooling that we work on at Polar Signals, and even then I think this is the right thing to do. It enables so much more debuggability aside from profiling. [1] https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-based-stack-walking-using-ebpf https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas...
- duskwuff 3y agoThis option was almost certainly a holdover from the bad old 32-bit x86 days, when disabling frame pointers gave you a seventh valuable general-purpose register. It's no longer beneficial on x86_64 -- even with rbp locked down, you still have fourteen registers there.
- brancz 3y ago100% the way I see it! On 32-bit the performance benefits is very major on just about everything, but not so on 64-bit.
- PaulDavisThe1st 3y agoand 14 registers should be enough for everyone!
- mratsim 3y agoIt's not. 14 is still low compared to other ISAs.
- peterfirefly 3y agoSpilling to an XMM register instead of memory is an option.
- jandrese 3y agoYeah, back in the 32 bit days -fomit-frame-pointer was the only optimization you could count on to really make a difference. It wasn't small either, often 10-15% speedup. No other flag on gcc would make even a full percentage difference in my testing. The Amd64 architecture fixed the underlying problem, so this is pretty much just a holdover. I'm surprised they even enabled it by default.
- grrandalf 3y ago(iirc) I think I used `-fomit-frame-pointer` with DJGPP on DOS on a 486. It was an [unimpressive :)] software-rendered 3D graphics demo, and I was very happy with the substantial free speedup I got.
- cesarb 3y agoTo emphasize this point: on 64-bit x86 with frame pointers, you have twice as many registers as on 32-bit x86 without frame pointers, and these registers are twice as wide. A 64-bit value (more common than you'd expect even when pointers are 32 bits) takes two registers on 32-bit x86, but only a single register on 64-bit x86.
- karmakaze 3y ago> Ubuntu leaps forward with frame pointers by default Yeah it's more like "...will stop omitting frame pointers by default".
- amelius 3y ago> to make Ubuntu debuggable by default! How's that, you'd still need the debug symbols Also has anyone else noticed that running stuff through Valgrind is really only possible if the program was made with Valgrind in mind? For example, Python and its many extensions generate numerous errors and warnings, so many that any real problem becomes hidden. I'd say that modern Linux systems are very far from being debuggable.
- rightbyte 3y agoYou offent need to make excemption file. My tip is to run some thing one program execution, and then run it twice during the same program execution, and diff the reports.
- brancz 3y agoUbuntu hosts a debuginfod server that you can automatically discover debuginfo from any binary from using the binary’s build id. https://sourceware.org/elfutils/Debuginfod.html https://sourceware.org/elfutils/Debuginfod.html
- herodoturtle 3y agoFor the layman, what does this mean?
- supportengineer 3y agoExciting new failure modes
- saagarjha 3y agoBetter stack traces for profiling tools when you don't have debugging symbols, or can't use them for whatever reason
- im3w1l 3y agoI was going to write a long essay about the relation between C and assembly but after thinking for a while, I think there is an easier way to explain it. Stack frames are basically a (single) linked list of information about the call stack. Every frame corresponds to one function call, and says where the local variables are stored, and where the function should return after it has finished. The head of this list is stored in a register (a scarce resource, superfast memory). So to use frame pointers, you have to spend one register, and also every function has to do some work to maintain the linked list. Two instructions worth of work when the function is enterred, and one when it exits. The alternative to doing this explicitly is to keep track of it all implicitly which is faster but a bit more complex.
- brancz 3y agoWe wrote pretty extensively about what was needed to be able to profile things without frame pointers [1]. It's still possible at less than 1% overhead with the right set of technologies, but frame pointers unwinding is virtually free. [1] https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-based-stack-walking-using-ebpf https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas...
- mratsim 3y agoIntel VTune, Apple Instruments and perf have been able to profile without frame pointers.
- pryz 3y agoThe post from Polar Signal blog https://www.polarsignals.com/blog/posts/2023/12/13/embracing-frame-pointers-in-ubuntu-24-04-lts https://www.polarsignals.com/blog/posts/2023/12/13/embracing... Exciting!
- westurner 3y agoCall stack > Structure > Stack and Frame pointers: https://en.wikipedia.org/wiki/Call_stack#Stack_and_frame_pointers https://en.wikipedia.org/wiki/Call_stack#Stack_and_frame_poi... What do the Coding Guidelines listed in e.g. awesome-safety-critical say about Frame pointers? https://awesome-safety-critical.readthedocs.io/en/latest/#coding-guidelines https://awesome-safety-critical.readthedocs.io/en/latest/#co... (Edit) /? "cert" "frame pointer" https://www.google.com/search?q=%22cert%22+%22frame+pointer%22 https://www.google.com/search?q=%22cert%22+%22frame+pointer%... : - Stack buffer overflow > Exploiting stack buffer overflows: https://en.m.wikipedia.org/wiki/Stack_buffer_overflow https://en.m.wikipedia.org/wiki/Stack_buffer_overflow : > In figure C above, when an argument larger than 11 bytes is supplied on the command line foo() overwrites local stack data, the saved frame pointer, and most importantly, the return address What about the Top 25? /? site:cwe.mitre.org "frame pointer" https://www.google.com/search?q=site%3Acwe.mitre.org+%22frame+pointer%22 https://www.google.com/search?q=site%3Acwe.mitre.org+%22fram... : - CWE-121: Stack-based Buffer Overflow https://cwe.mitre.org/data/definitions/121.html https://cwe.mitre.org/data/definitions/121.html This is closer to a better approach for security, debuggability, and performance IMHO: https://news.ycombinator.com/item?id=38138010 https://news.ycombinator.com/item?id=38138010 : > gdb on Fedora auto-installs signed debuginfo packages with debug symbols; Fedora hosts a debuginfod server for their packages (which are built by Koji) and sets `DEBUGINFOD_URLS=` > Without debug symbols, a debugger has to read unlabeled ASM instructions (or VM opcodes (or an LL IR)).
- westurner 3y agoWhen frame pointers are omitted, there are fewer places in memory that can be overwritten to hijack control flow of a program. Someone could easily prepare an demo of a frame pointer buffer overflow exploit to explain?
- daoistmonk 3y agohasn't this already been in fedora for almost a year?
- brancz 3y agoYou're not wrong, but isn't it exciting that it's gaining more traction? Ubuntu is also very widely deployed, so I'll celebrate every distro that makes this the default, be it Ubuntu in this case, or Debian, or Arch, or Suse, or or or...
- bravetraveler 3y agoI believe so, I remember a fair amount of hubbub over it. Can't say it's been useful here, any development I do is miles away from this. It hasn't hurt either so... cool, I guess Call me pessimistic, but I'm not convinced this being the default will lead to more profiling. There's plenty that could be done without this, that isn't, so I'm not buying it.
- brancz 3y agoWhile profiling benefits immensely, imagine all the other ramifications. Cheap tracing of MySQL, postgres, or anything else using bpftrace or any other bcc tools. Getting stack traces of processes that are already core dumping... Profiling is one part, but the debuggability this enables is going to be huge in the long term I predict.
- pavon 3y agoYeah. To be fair there is a stronger argument for it being enabled in fedora than an LTS release (RHEL or Ubuntu) since it has more cutting edge software that needs more frequent debugging, is less likely to be used in production where the (minor, but uneven) performance hits may matter, and has so many upstream developers using it as their daily driver.
- brancz 3y agoI would argue that LTS releases are going to be deployed millions of times and stay around effectively forever with lots of very critical software being deployed on it. Having all processes/binaries be debuggable cheaply and easily in stressful situations is a major improvement that's now here to stay.
- RecycledEle 3y ago[flagged]
- stephendause 3y agoFor anyone else who didn't know what a frame pointer was: https://softwareengineering.stackexchange.com/questions/194339/frame-pointer-explanation https://softwareengineering.stackexchange.com/questions/1943...
- vaylian 3y agoFor x86 that value is typically stored in the stack base pointer register (bp): https://en.wikibooks.org/wiki/X86_Assembly/X86_Architecture https://en.wikibooks.org/wiki/X86_Assembly/X86_Architecture
- amne 3y agoEven in context it is hard to understand this: "The performance wins that these can provide far outweigh the comparatively tiny loss in performance." My guess is that on average the potential performance discovered with the techniques this enables is higher than the guaranteed negligible performance loss.
- saagarjha 3y agoYes, that's correct.
- o11c 3y agoThat is highly optimistic though. Profiling is hard even when you know what you're doing, and doing it wrong can easily lead to pessimization (thinking particularly about if your profiling workload exercises a different set of branches).
- brancz 3y agoHyperscalers have long been doing infrastructure-wide profiling (or "Google-Wide Profiling" as the first whitepaper on the topic calls it [1]). This tech allows Google to reduce infra-resource usage by multiple percentage points per quarter. [1] https://research.google/pubs/google-wide-profiling-a-continuous-profiling-infrastructure-for-data-centers/ https://research.google/pubs/google-wide-profiling-a-continu...
- o11c 3y ago"Software that runs at scale" is a very different story than "Software that is shipped with a distro", unfortunately. Very few distro packages can even define what a normal workload looks like.
- dralley 3y agoThat's precisely why it's nice for users to be able to collect these profiles themselves.
- kapilvt 3y agohmm.. I wonder if the 10% performance regression in python has been resolved. https://discuss.python.org/t/python-3-11-performance-with-frame-pointers/22507 https://discuss.python.org/t/python-3-11-performance-with-fr...
- jonseager 3y agoWe’re waiting on the benchmarks once the archive has been rebuilt to double check if we’ll be affected in that way, but if indeed we do find that sort of regression, we’ll exclude Python from this change (and any other package where there is a substantial hit).
- therealmarv 3y agoPro tip for performance optimisation in Ubuntu (you also gain more RAM): Remove snap (or choose a Ubuntu distro variant like Pop OS without snap)
- solarkraft 3y agoIs that even practical nowadays? My understanding is that snap is deeply integrated. Don't some apt packages point to their snap variants?
- woodruffw 3y agoIt’s still possible, although increasingly annoying. I believe Firefox now points to its snap variant, which I discovered when it broke a bunch of my browser extensions. Switching to the official Mozilla PPA was easily enough, but left a bad taste; if Canonical continues down the route of silently nudging users onto snap, I’ll probably switch to Debian. (I have no particular opinions about snap itself, other than that it seems poorly documented and doesn’t adhere to the “do what I say” philosophy when it’s secretly injected into apt.)
- uxp8u61q 3y ago> if Canonical continues down the route of silently nudging users onto snap, I’ll probably switch to Debian. Serious question: why don't you switch now? I guess I just don't see the point of ubuntu anymore.
- ghotli 3y agoI somewhat painstakingly figured this out the hard way pulling core dumps off of embedded linux devices that gdb had a hard time working with. At the time I was like whew why is omitting frame pointers the default at all in so many places when it didn't seem to make a measurable difference in the performance of the software. I guess it's just vestigial these days and yes please use a compiler flag like this when it measurably makes sense. Making debugging simpler for the rest of us is the way to go.
- o11c 3y agoSomething has to be seriously wrong with the way it was compiled for gdb to have trouble. Debugging (or in-process exception dumps), which only does a reasonable number of backtraces, should always be able to use separate unwind data sections. Including frame pointers should only have a performance effect for sample-based profiling, which does a very large number of backtraces. And the general fact is - people don't profile, and if they do, they don't do it correctly. Omitting frame pointers has significant performance wins on platforms with about 6 registers, like 32-bit intel x86. It's much less of a win on platforms with about 14 registers, like 64-bit x86 or 32-bit ARM, let alone platforms with about 30 registers, like 64-bit ARM. Since modern architectures are strongly trending toward designs that support more registers, a frame pointer isn't unreasonable to choose. But that's still no excuse for all the shitty software that refuses to work correctly without them, rather than merely more slowly. (Note that theoretically it is possible to design an ISA/ABI combo that supports easy and fast unwinding even without frame pointers, but there's always going to be some overhead and to my knowledge this choice hasn't been done.)
- brancz 3y agoFounder of Polar Signals here, the profiling product that's mentioned in the blog post. We've already made it possible to unwind without frame pointers relatively cheaply (<1% overhead), and have written about this extensively [1]. That said, no matter how we spin it, frame pointer unwinding is always going to be cheaper, and while profiling is getting better, I think I'm almost more excited about the other aspects of debuggability this is gaining: out-of-the-box working bpftrace, bcc-tool and anything else that needs to deal with unwinding with just about anything that's running on the box. I think we'll see a huge gain in capabilities over the next few years with frame pointers more prevalent in fedora and ubuntu and I'm sure more will now follow. [1] https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-based-stack-walking-using-ebpf https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas...
- zX41ZdbW 3y agoClickHouse has always-on profiling without frame pointers. But the implementation is very hard - it required patching of LLVM's libunwind to make it 100% async-signal safe. Using frame pointers should be easier and faster.
- ur-whale 3y agoWhat about binary size increase ?
- brenns10 3y agoThis is a good idea for the short-term. As of now, frame pointers are the most reliable way to ensure that software can be profiled by tools like perf*. The core issue is that the kernel must be the one to unwind the userspace stack, and it only knows how to unwind stacks with frame pointers**. The .eh_frame data will never be supported by the kernel, because it involves a turing-complete program that must be executed to compute the necessary unwind info***. For the long term, the more exciting option that's emerging is SFrame[1]. This is a new data section which would be generated by the compiler and contains unwind tables which the kernel will be able to understand. Unlike DWARF/.eh_frame, these tables would remain in the final binary (i.e. not be stripped away), and on exec(), the kernel would store them for use during profiling. Since the format is quite similar to ORC(*), and Steven Rostedt is quite invested in the format, it seems a safe bet that support will land in the kernel. My hope isn't necessarily that a distribution completely disables frame pointers once this format becomes available... though it could be an interesting thing to try. Rather, there can be a conscious choice about whether frame pointers are used, or SFrame, which would be useful for cases like Python, where it's mentioned that frame pointers may still have a significant performance impact. The kernel should be able to fall back to frame pointers when SFrame is unavailable, which means that either will be acceptable. Ideally, in a few years time we'll be able to go back to forgetting about frame pointers for most cases :) --- * Ironically, the kernel itself tends not to use frame pointers! It has its own unwind format called ORC, which gets generated by an in-kernel program called "objtool" which essentially reverse engineers the assembly generated by the compiler. It's x86_64-specific and frequently needs adjustment when the compiler changes code generation. It can't be used for userspace programs. ** it also knows how to unwind kernel stacks with ORC (see above) *** There is an option to allow perf to unwind with DWARF, but it's a total hack (though a very effective one). By passing --call-graph=dwarf, you can instruct the kernel to copy the userspace stack (by default, 8k bytes!) into the perf event buffer with each sample (this can be as many as 100 or 1000 samples per second, per CPU...). Later, the perf userspace program will use that info, along with information about each process's address space, and the debuginfo for each program, to unwind the stacks. This has huge performance overhead, and it requires that you have easy access to debuginfo, which may not be the case, especially for container workloads. [1] https://lwn.net/Articles/940686/ https://lwn.net/Articles/940686/
- londons_explore 3y agoSo the whole world should take a 1-2% performance penalty on everything so some users can maybe run a profiler? Wouldn't it make more sense to just have an 'apt reinstall all --with-frame-pointers' command that power users could run before they wanted to profile something?
- pjmlp 3y agoThey already take performance hits left and right with all those containers running all over the place, parsing JSON.
- alfalfasprout 3y agoThis is very hyperbolic and inaccurate. In 2023 it's not a 1-2% performance penalty anymore and certainly not for most use cases. Only if the 15th register is critical for performance on an x86_64 CPU. Certain workloads might suffer more, but most will certainly suffer less than a 1-2% hit.
- m463 3y agoYou are prematurely optimizing. "can make use of this improved debugging information to diagnose and target performance issues that are orders of magnitude more impactful than the 1-2% upfront cost." Also, can't you get reliable stack dumps when something goes wrong too?
- account42 3y agoYou can get reliable stack dumps without frame pointers. Removing needless instruction and register pressure that 99.99% of users don't rely on in any way and the rest don't need if they fixed their tools is not premature optimization but simple common sense. Which is why its on by default in the first place. Calling 1% or even 0.1% optimizations that apply accross the board "premature optimizing" is a great example of the culture of wastefulness that has made computers less responsive even though hardware has gotten a million times faster. These things do add up.
- m463 3y agointeresting article on frame pointer optimization: https://community.ibm.com/community/user/wasdevops/blogs/kevin-grigorenko1/2023/02/21/lessons-from-the-field-26-frame-pointers-and-why-t https://community.ibm.com/community/user/wasdevops/blogs/kev...
- AndyKelley 3y agoWhy not let the upstream application developer decide, rather than choosing for them? This is one of the downsides of using C/C++ rather than a modern programming language like Rust or Zig. In the former case, the system maintainers reach across the table and change the settings, despite what the actual application developer has chosen. In the latter, the upstream developers' choices are respected more, mainly because the tooling is less standardized. Shoutouts to this NixOS bug which is still ongoing after causing much pain for many years: https://github.com/NixOS/nixpkgs/issues/18995 https://github.com/NixOS/nixpkgs/issues/18995
- comex 3y agoBecause the upstream developer is not the only one who needs to profile the program. For one thing, you might be the upstream developer of a library the program links against, and your stack frames might be hidden under frames from the program that didn’t save the frame pointer. Or if it’s a library that isn’t saving the frame pointer, vice versa. But the more interesting use case is if you’re not the upstream developer of anything, but a skilled user who wants to get a view of your entire system and diagnose problems yourself. Personally I do that all the time, both in my spare time and at work. With respect to performance in particular I have more experience with the macOS tooling than the Linux tooling, but it’s analogous – with tools like Instruments and dtrace I can get a profile of any process I want or of the entire system, and I find that incredibly valuable. And that’s made possible in part by the stock macOS toolchain enabling frame pointers by default. The NixOS case you linked sounds rather annoying, but turning on optimizations and -Werror and PIC by default is very different from just enabling frame pointers by default.
- lathiat 3y agoYeah, I agree with this. Not clearly written in the original article is that "many" (I'm not sure what actual percentage, but vaguely "most") packages already have frame pointers enabled. The problem packages that don't are exactly all of those upstream projects that intentionally compile with -fomit-frame-pointer because of those small performance gains. And those are also usually the exact same projects you end up wanting to profile or otherwise analyse :) I work in the Support organisation at Canonical and the two most frequent projects I run into this with are Ceph and Openvswitch - they both compile with -fomit-frame-pointer by default upstream (and currently in the Ubuntu packages) which makes using perf (which I often need to do with both of those) a pain. While you can do it, you have to record ~8kB of stack extra for every sample (times 1000 per second, times the number of CPUs...) and then unwind it later with the DWARF debug symbols. The resulting perf exports are multiple gigabytes for 0-2 minutes. Compared to maybe 25-200MB for frame-pointer enabled cases where I can usually then easily captured. The problem is the product or upstream project wants to claim the absolute best performance, even 1% better, but the end user rarely needs that last 1% and both them and their support team would like to be able to easily use profiling in production to fix the random much more significant 10-100% performance bugs that inevitably crop up when actually using it and not benchmarking it :)