18 ms·

Fedora now has frame pointers

213 points by LaSombra 4y ago

pengaru 4y ago
The title should really be "Fedora now has frame pointers on x86_64", bringing it inline with ARM, where you already get them.
AshamedCaptain 4y ago
I predicted this would happen when ORC was introduced to the kernel. I'm tired of this eternal wheel of history in computing where everyone feels the need to rewrite stuff that works. If it wasn't for this we'd already have a DWARF unwinder in the kernel and on every other profiling-tool-of-the-week and this would be a non-issue. But no! Let's come up with a new debuginfo format, that should simplify things right? And a couple years later the main distros have to downgrade to the least common denominator (in this case, frame pointers) just because profilers and toolchains can't implement one. single. common. debuginfo. format for once.
- gpderetta 4y ago
  The linux kernel: NIH as an art form.
- quotemstr 4y ago
  > And a couple years later the main distros have to downgrade to the least common denominator (in this case, frame pointers) just because profilers and toolchains can't implement one. single. common. debuginfo. format for once. You don't even need a common debug information format. All you need is a mechanism through which the kernel can ask userspace to collect its own stack. But because the Linux world has elevated shipping the org chart to the highest level of virtue, it's hard to coordinate reasonable solutions.
  - anarazel 4y ago
    Introducing a context switch, performing unwinding in process, with a relatively expensive method, would distort profiles sufficiently to make them not that useful anymore.
    - AshamedCaptain 4y ago
      Kernel-based profiling already involves multiple context switches by definition.
      - anarazel 4y ago
        Sure, but that's not really counter to my point. An interrupt processed in the kernel isn't comparable to sending a signal to a userspace process, scheduling it (might not even be possible immediately!), walking the stack there, returning to the kernel and continuing. For stuff like bpf maps the kernel is even going to need the stacktrace in some form. Aside from the overhead, the delay between sample interrupt and the collected stack will be substantial. And then there's the whole fun of doing the stack walk in userspace in an async signal safe way.
        AshamedCaptain 4y ago
        You're overly exaggerating the impact slow stack unwinding can have on the reliability of profiles. Most people are literally just fine with the poor man's profiler (aka "watch gdb"), which would be significantly slower than anything being suggested here, and doesn't even have kernel help in freezing the traced process (the kernel can freeze it the moment the profiling event happens). The proprietary OS I'm working with right now actually does this, and the only visible limitation is that there are no stack traces on kernel crash. It does CoW the stack frames while the unwinder is going on. I still wouldn't do this, but I also see no reason why a monolithic kernel like Linux that already embeds multiple NTFS implementations can't embed a single decent DWARF unwinder.
        anarazel 4y ago
        Shrug. Our experiences differ drastically then. IME profiling using the gdb style manual sampling method work kinda OK if you have a very skewed profile, but basically not at all after that. Sure, lots of people don't know how to use a profiler with call graphs and everything. But a) partially that's because it's so hard to get a useful call-graph profile b) lots of people don't work on performance sensitive code, but plenty others do. The cost of walking stacks is pretty easy to see if you use perf report on a dwarf profile. A small part of that is just the larger data file, but if you profile perf report you can see that overwhelming the cost is unwinding. In my quick test case at least 70%. I'm sure that libunwind (as used by perf) could be improved by 2-3x - but you'd still be vastly slower.
userbinator 4y ago
Making code less efficient, mainly for the sake of some stupid tool (and I use that term in both meanings...), and one that's supposed to help improve efficiency? That's incredibly ironic. The thing is that DWARF unwinding (the only userspace option that doesn’t use frame pointers) is really unreliable. In fact it has such serious problems that it’s not that usable at all. The real solution to things like this is to improve the debugging information so that return addresses can be tracked more accurately, but no one seems willing to do that work. It reminds me of encountering "value optimised out" when I can clearly see where the value is if I inspect the Asm. Explicit constant frame pointers are almost an anachronism from an era when compilers were far worse at code generation. It seemed like a good idea at the time because it made compilers simpler, and of course on 16-bit x86 bp was the only other register that defaulted to the stack segment and [sp+nn] didn't exist. Those points are not really relevant anymore with 32-bit and 64-bit x86 using a flat memory model. Sometimes the extra byte needed for esp/rsp-relative access is worse, sometimes the extra register consumed by ebp/rbp-relative access is worse. It's really something that needs to be decided case-by-case like a intelligent Asm programmer would.
- vbezhenar 4y ago
  People who build distros and people who fix those low-level tools are different sets of people, I guess. So they have to build distro using tools available today. If tomorrow situation changes, they'll probably revisit this decision.
- justinclift 4y ago
  > The real solution ... Is the "real solution" a case of letting "the perfect" get in the way of "good enough" (sic)?
  - Dylan16807 4y ago
    Just keeping track of the frame pointer is probably not the perfect. In fact it might be the exact opposite. We have very complicated mechanisms that try to keep track of every possible optimization. Maybe that effort is getting in the way of a solution that slightly limits frame pointer optimization but easily figures out the value in all situations. So because the "perfect" mechanism is having implementation problems, and nobody has tried to make the "good" mechanism, we're stuck with the very mediocre "reserve a register".
jart 4y ago
-fomit-frame-pointer is pretty high on the list of most destructive micro-optimizations in the history of open source. When you repurpose the backtrace pointer (RBP) for other things, it totally savages our ability to have native backtraces. All for a pointless ~1% performance boost. I've always thought it was some kind of grand conspiracy to obfuscate the behaviors of closed source software but for some reason every single Linux distro that claims to love freedom has to do this too. I don't want to trust binaries built by strangers on the Internet if they hide their behaviors. Back when I worked at Google I used to read a lot of the company-wide core build configs, and the first thing they always did was make sure -fno-omit-frame-pointer is always passed, no matter what build flags a specific team uses. Don't tell me DWARF is an acceptable alternative. While it can be nice sugar on top with Linux perf report when it actually works, frame pointers let me produce a backtrace in just ten lines of C without any dependencies. If I wanted to do the same with DWARF I'd need to link some GPL library or write 10,000 lines of byzantine code to figure out which function called the one that's crashing. Stuff like this is why everyone uses scripting languages these days, since the main value prop of high-level languages is their ability to reliably print backtraces.
- cwzwarich 4y ago
  > I don't want to trust binaries built by strangers on the Internet if they hide their behaviors. If a binary really wanted to hide its behavior (at least to the same extent that it could with -fomit-frame-pointer), it could just make fake stack frames beyond the current PC.
- catiopatio 4y ago
  DWARF is not just the better alternative, it’s the only correct solution. Not only do you get more complete and correct backtraces with DWARF, you also get preservation of stack-saved registers and the ability to reconstitute a subset of local state in a given frame. libunwind is MIT licensed, not GPL. I’ve also written a full DWARF stack unwinder myself, including a full DWARF expression interpreter — it’s nowhere near 10k lines of code.
  - blinkingled 4y ago
spicyusername 4y ago
Seems like a reasonable solution to a fraught problem. Man hours (spent recompiling the kernel with the right flags in this case) are often undervalued when comparing different solutions. I have found this is especially true when doing cost analysis. It is so easy to look at something that costs $10,000 a month and think it's expensive, despite the fact that it saves twice that much in developer time (salary).
- saurik 4y ago
  How many people are using Fedora? Why must all of them suffer because you want to save a very small amount of your time? Why can't the developers--which include me!!--who want to profile things take the time to custom compile their boxes instead of demanding that all of the poor end users who don't even know what "profile" means have to take a reduction in performance? How many end users would have to exist to make you decide it isn't worth their pain and extra cost for you to have to spend a minutes of your time to help them? (Maybe then further answer that question under the scenario where the users in question by and large never actually install your software.)
  - prmoustache 4y ago
    Fedora End-User vs Fedora Developer edition? What prevents anyone of us to maintain a fork with same SRPMS as fedora but keeping the omition of frame pointers? I guess someone will do that if there really is an observable performance advantage. There are already fedora forks such as Nobara providing some kernel patches and some specific compilation options or patches for some apps.
  - spicyusername 4y ago
    I think many many many users would happily acquiesce, knowing doing so is helping to build a better product. Also 1-2% performance hit is peanuts, given the benefit. It isn't a religious battle of ideology, it's a concrete tradeoff of something small for something large.
    - saurik 4y ago
FullyFunctional 4y ago
I recently built SPECint2k6 for RISC-V (on JH7110) with and without -fomit-frame-pointer and to my surprise the default (with frame pointers) was a tiny bit faster (though so close to the noise floor that I'll settle for "the same"). I haven't dug into why frame pointers seem to cost nothing on RISC-V but it's likely that it's just that there's plenty of registers.
- deleted 4y ago
  [deleted]
- xobs 4y ago
  Unfortunately, it increases the code size by 10%. I was looking into this just last week, and can confirm that it's still a problem on the latest version of Rust nightly: https://github.com/tock/tock/pull/1660 https://github.com/tock/tock/pull/1660 I wish we could have frame pointers, because they would make working in embedded land so much easier and more reliable, but a 10% increase in code size just isn't worth it.
  - andrekandre 4y ago
    perhaps slightly naive question, but whats the actual performance implication 10% increase in code size? (10% slower at runtime?)
    - MobiusHorizons 4y ago
      In the embedded world, you are limited by the amount of storage space available for your code. In a lot of cases the code is in flash or one time programmable rom memory on the microcontroller. Even if you have off-chip storage, you usually have to load the code into ram in order to run, and ram size is also quite limited. For low end microcontrollers it's common for rom / ram sizes to be on the order of 32kb (although they go much lower than that). In this world, the most common optimization level is -Os (or -Oz if available) because running out of storage space means you need a more expensive microcontroller, which increases the cost of each device you are selling, and might require a redesign of the PCB.
nullc 4y ago
I wonder what the total energy footprint of zillions of users using this who will never run a profiler. I imagine it's pretty substantial compared to the cost of putting a DWARF unwinder in the kernel.
- Gigachad 4y ago
  Essentially nothing in a world where almost everyone is running 7 electron apps at once and Windows has lost the ability to suspend properly. Especially when all those apps were built on these platforms exactly because of how annoying it is to debug C programs. The total energy usage could even be negative due to this.
  - nullc 4y ago
    It's a fair point that other things waste unimaginably more, but in terms of debugging -- omitting the frame pointer hasn't been a debugging headache for a long time and even if it were that still wouldn't itself justify running it on every host under the sun including the vast majority that will never have any debugging run on it. I doubt C debugging has had a meaningful role in the use of electron in any case.
    - saagarjha 4y ago
      Not debugging, but profiling. They use similar infrastructure but have different needs when it comes to reliability and overhead.
- rwmj 4y ago
  You have to trade that against the large optimizations that were never made because performance analysis of the whole system was essentially unavailable. This is a very new change in Fedora, but hopefully we'll now see lots of big performance improvements in common workloads. The analysis I was doing when writing this blog post was about understanding where 50% of the time of my program was going. The fix literally doubled the speed of the program: https://bugzilla.redhat.com/show_bug.cgi?id=2166686 https://bugzilla.redhat.com/show_bug.cgi?id=2166686
quotemstr 4y ago
The biggest problem with frame pointer unwinding isn't even the performance hit, but the inability to support unwinding of anything that isn't AOT compiled native code. If you unwind using frame pointers through a Python program, you'll get a bunch of meaningless interpreter frames instead of information about the operation of the program being interpreted. That, and since the FP unwinder can't sleep and can't take page faults, it'll fail to traverse stack frames that happen to be paged out. The right way to solve the kernel unwind of userspace problem is to get the kernel out of the business of unwinding userspace. Concrete proposal here: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/IMBATOSTV7BIX7WQKMUWGDFYJG2TSBIF/ https://lists.fedoraproject.org/archives/list/devel@lists.fe... Unwinding only paged-in, AOT-compiled native code is a step in the wrong direction, especially because it's precisely the people for whom unwinding kinda-sorta works now (kernel, libc, C language diehards, etc.) who would need to approve and coordinate a more robust and comprehensive solution that also, unlike frame pointers, has zero cost when not in use.
- rwmj 4y ago
  You want the kernel to be able to collect the trace as fast as possible. You don't want to add half a million context switches to your trace.
  - quotemstr 4y ago
    Where are the extra "context switches"? Can you count them for me? Under my proposal, we defer stack collection until we're going to return to user space anyway. The user stack of a thread cannot change while that thread is running in user mode.
    - rwmj 4y ago
      So after looking at this, I do now agree [can't edit my original comment]. Are you planning to implement anything like this? I think similar to the suggestion to fix DWARF, enabling frame pointers may not be the best solution, it may be reversed in future when we get something better, but it's the one that actually works right now.
throwawaylinux 4y ago
I wonder how significantly frame pointers damages the stack engine optimization in x86 processors. There's still a bunch of pushes and pops, I guess because that's shorter code than storing into an explicit offset off the stack pointer, but since you've read the stack pointer anyway and synchronized it, the uop sequence and instruction dependencies wouldn't end up being a lot different I would have thought.
- rwmj 4y ago
  It does damage it actually. GCC currently generates slightly pessimised code when frame pointers are enabled, for example always using the frame pointer to access stack variables even if it knows the offset from the stack pointer. I guess this is simply because distributions were all using -fomit-frame-pointer so the frame pointer case was neglected. Hopefully this will improve now that most distros have switched to enabling frame pointers.
javierhonduco 4y ago
I’m looking forward to this change. Ensuring fast, reliable stack traces when debugging under pressure or tracing complex systems is invaluable. Once you try it it’s hard to go back. It’ll take some years until frame pointers become more mainstream. Still, some people might decide to never enable them in libraries they use or their own code. That’s why I’ve been working on a BPF DWARF-based unwinder to fill the gap. It’s fully OSS (Apache 2.0 for user space and GPL for BPF) and should get out of beta in ~1month. Previous HN discussion [0]. Happy to help anybody that wants to integrate it in their tools / systems. - [0]: https://news.ycombinator.com/item?id=33788794 https://news.ycombinator.com/item?id=33788794
peter_retief 4y ago
I have moved my Linux Desktop to Fedora (Recommended by my 26yo son). Been in the Linux space since the beginning so have seen distros come and go. Who remembers Slackware? What a journey Linux has been. Hope the framepointers work out (I think they will squeeze out some more performance) Thanks to all the amazing people who developed the Linux miracle.
- ilyt 4y ago
  >Who remembers Slackware? I still have nightmares about it. What a terribly put together thing. Moving to Debian was such a breath of fresh air, packages that works!
  - peter_retief 4y ago
    Kind of why no one uses it anymore I guess.
  - prmoustache 4y ago
    What didn't work or wasn't well put together on slackware? The only thing that was annoying at the time was having to manually download and compile third party software as well as some of its depencency. In reality it wasn't that much of a big deal. You just had to know how to read a readme and install text file really. But this is kind of a moot point in 2023 with: 1. slackbuilds and its clients, which is pretty much like the AUR for archlinux users or pkgsrc and makes you a lot of additionnal software available very easily. 2. All the software nowadays made installable with flatpak, appimage or via some bash script. 3. podman and docker If anything modern software deployment methods made slackware experience closer to rpm or deb based distros than 20 years ago.
  - dmpk2k 4y ago
    I had the exact opposite experience. Debian always had to add their own variations and special sauce to the configurations and layout. Slackware was as stock as you could get, effectively identical to what you'd get with a configure; make; make install. I much preferred stock.
- gavinray 4y ago
jwilk 4y ago
Related: https://news.ycombinator.com/item?id=34426745 https://news.ycombinator.com/item?id=34426745 ("Fedora's tempest in a stack frame" from lwn.net, 17 days ago, 31 comments)
anarazel 4y ago
Heh. Looking for discussions in debian around this, I found a bug requesting a glibc build with frame pointers enabled. Reported by me. 8 years ago ...
- javierhonduco 4y ago
  That would be great! There’s a precedent in Ubuntu, in case it could help to push this forwards `libc6-prof`.
- jwilk 4y ago
  https://bugs.debian.org/767756 https://bugs.debian.org/767756
AlexeyBelov 4y ago
The author published an update: https://rwmj.wordpress.com/2023/02/06/frame-pointers-an-important-update/ https://rwmj.wordpress.com/2023/02/06/frame-pointers-an-impo...
bravetraveler 4y ago
I can't pretend to know enough about the actual implications of this, but I have to wonder - if the goal is to encourage contributors, why not grease the wheels further on fedpkg/mock? Why must we all have even slower performance? This is more of an aside - I love Fedora, but it already stands out of the pack as a bit slower than others. It's not particularly difficult to rebuild things as-is, is my point - I suppose. It all feels so self-fulfilling. People will want to profile (and benefit from this more)... because things that are context-switch heavy may have slowed down considerably. I'm considering how I may batch my own rebuilds to remove this option with these utilities, mainly to see apples-to-apples what the difference is I know they've published hand-wavy numbers/benchmarks, but I don't trust them for a multitude of reasons
- pxc 4y ago
  Some thorough benchmarks are something I'd love to read about here on HN. If you ever do those benchmarks, post 'em.
  - bravetraveler 4y ago
    I'll be happy to share! I'll probably be waiting for the next release so I can get their pointer-enabled builds. However they do that may give me helpful hints RE: rebuilding the same sources without it. I know the tools to do it, but what I'm imagining right now is a bit convoluted. I'm open to suggestions on things to test. I want to avoid too much reliance on synthetics -- and my 'real world' differs like all of ours My original post has (currently) been down-struck a bit but you're the only reply. I appreciate you taking the time to engage I understand it seems dismissive, perhaps even aggressive, but the Fedora/SuSE people tend to set trends. I just want to measure them. The snark is me finding it ironic that I can (relatively easily) measure this with the same methods they want to avoid... rebuilds with the right options under mock / fedpkg subtext: not that many people find profiling interesting or important... but they will if you start messing with their performance. I find it interesting though so I'm not mad, I just fairly called it self-fulfilling -- I'm going to be profiling because the minority who regularly do, don't want to rebuild (like I will be)
  - 4y ago

213 points by LaSombra 4y ago

The title should really be "Fedora now has frame pointers on x86_64", bringing it inline with ARM, where you already get them.

AshamedCaptain 4y ago

I predicted this would happen when ORC was introduced to the kernel. I'm tired of this eternal wheel of history in computing where everyone feels the need to rewrite stuff that works. If it wasn't for this we'd already have a DWARF unwinder in the kernel and on every other profiling-tool-of-the-week and this would be a non-issue. But no! Let's come up with a new debuginfo format, that should simplify things right? And a couple years later the main distros have to downgrade to the least common denominator (in this case, frame pointers) just because profilers and toolchains can't implement one. single. common. debuginfo. format for once.

gpderetta 4y ago
The linux kernel: NIH as an art form.
quotemstr 4y ago
> And a couple years later the main distros have to downgrade to the least common denominator (in this case, frame pointers) just because profilers and toolchains can't implement one. single. common. debuginfo. format for once. You don't even need a common debug information format. All you need is a mechanism through which the kernel can ask userspace to collect its own stack. But because the Linux world has elevated shipping the org chart to the highest level of virtue, it's hard to coordinate reasonable solutions.
- anarazel 4y ago
  Introducing a context switch, performing unwinding in process, with a relatively expensive method, would distort profiles sufficiently to make them not that useful anymore.
  - AshamedCaptain 4y ago
    Kernel-based profiling already involves multiple context switches by definition.
    - anarazel 4y ago
      Sure, but that's not really counter to my point. An interrupt processed in the kernel isn't comparable to sending a signal to a userspace process, scheduling it (might not even be possible immediately!), walking the stack there, returning to the kernel and continuing. For stuff like bpf maps the kernel is even going to need the stacktrace in some form. Aside from the overhead, the delay between sample interrupt and the collected stack will be substantial. And then there's the whole fun of doing the stack walk in userspace in an async signal safe way.
      - AshamedCaptain 4y ago
        You're overly exaggerating the impact slow stack unwinding can have on the reliability of profiles. Most people are literally just fine with the poor man's profiler (aka "watch gdb"), which would be significantly slower than anything being suggested here, and doesn't even have kernel help in freezing the traced process (the kernel can freeze it the moment the profiling event happens). The proprietary OS I'm working with right now actually does this, and the only visible limitation is that there are no stack traces on kernel crash. It does CoW the stack frames while the unwinder is going on. I still wouldn't do this, but I also see no reason why a monolithic kernel like Linux that already embeds multiple NTFS implementations can't embed a single decent DWARF unwinder.
        anarazel 4y ago
        Shrug. Our experiences differ drastically then. IME profiling using the gdb style manual sampling method work kinda OK if you have a very skewed profile, but basically not at all after that. Sure, lots of people don't know how to use a profiler with call graphs and everything. But a) partially that's because it's so hard to get a useful call-graph profile b) lots of people don't work on performance sensitive code, but plenty others do. The cost of walking stacks is pretty easy to see if you use perf report on a dwarf profile. A small part of that is just the larger data file, but if you profile perf report you can see that overwhelming the cost is unwinding. In my quick test case at least 70%. I'm sure that libunwind (as used by perf) could be improved by 2-3x - but you'd still be vastly slower.

userbinator 4y ago

Making code less efficient, mainly for the sake of some stupid tool (and I use that term in both meanings...), and one that's supposed to help improve efficiency? That's incredibly ironic. The thing is that DWARF unwinding (the only userspace option that doesn’t use frame pointers) is really unreliable. In fact it has such serious problems that it’s not that usable at all. The real solution to things like this is to improve the debugging information so that return addresses can be tracked more accurately, but no one seems willing to do that work. It reminds me of encountering "value optimised out" when I can clearly see where the value is if I inspect the Asm. Explicit constant frame pointers are almost an anachronism from an era when compilers were far worse at code generation. It seemed like a good idea at the time because it made compilers simpler, and of course on 16-bit x86 bp was the only other register that defaulted to the stack segment and [sp+nn] didn't exist. Those points are not really relevant anymore with 32-bit and 64-bit x86 using a flat memory model. Sometimes the extra byte needed for esp/rsp-relative access is worse, sometimes the extra register consumed by ebp/rbp-relative access is worse. It's really something that needs to be decided case-by-case like a intelligent Asm programmer would.

vbezhenar 4y ago
People who build distros and people who fix those low-level tools are different sets of people, I guess. So they have to build distro using tools available today. If tomorrow situation changes, they'll probably revisit this decision.
justinclift 4y ago
> The real solution ... Is the "real solution" a case of letting "the perfect" get in the way of "good enough" (sic)?
- Dylan16807 4y ago
  Just keeping track of the frame pointer is probably not the perfect. In fact it might be the exact opposite. We have very complicated mechanisms that try to keep track of every possible optimization. Maybe that effort is getting in the way of a solution that slightly limits frame pointer optimization but easily figures out the value in all situations. So because the "perfect" mechanism is having implementation problems, and nobody has tried to make the "good" mechanism, we're stuck with the very mediocre "reserve a register".

jart 4y ago

-fomit-frame-pointer is pretty high on the list of most destructive micro-optimizations in the history of open source. When you repurpose the backtrace pointer (RBP) for other things, it totally savages our ability to have native backtraces. All for a pointless ~1% performance boost. I've always thought it was some kind of grand conspiracy to obfuscate the behaviors of closed source software but for some reason every single Linux distro that claims to love freedom has to do this too. I don't want to trust binaries built by strangers on the Internet if they hide their behaviors. Back when I worked at Google I used to read a lot of the company-wide core build configs, and the first thing they always did was make sure -fno-omit-frame-pointer is always passed, no matter what build flags a specific team uses. Don't tell me DWARF is an acceptable alternative. While it can be nice sugar on top with Linux perf report when it actually works, frame pointers let me produce a backtrace in just ten lines of C without any dependencies. If I wanted to do the same with DWARF I'd need to link some GPL library or write 10,000 lines of byzantine code to figure out which function called the one that's crashing. Stuff like this is why everyone uses scripting languages these days, since the main value prop of high-level languages is their ability to reliably print backtraces.

cwzwarich 4y ago
> I don't want to trust binaries built by strangers on the Internet if they hide their behaviors. If a binary really wanted to hide its behavior (at least to the same extent that it could with -fomit-frame-pointer), it could just make fake stack frames beyond the current PC.
catiopatio 4y ago
DWARF is not just the better alternative, it’s the only correct solution. Not only do you get more complete and correct backtraces with DWARF, you also get preservation of stack-saved registers and the ability to reconstitute a subset of local state in a given frame. libunwind is MIT licensed, not GPL. I’ve also written a full DWARF stack unwinder myself, including a full DWARF expression interpreter — it’s nowhere near 10k lines of code.
- blinkingled 4y ago

spicyusername 4y ago

Seems like a reasonable solution to a fraught problem. Man hours (spent recompiling the kernel with the right flags in this case) are often undervalued when comparing different solutions. I have found this is especially true when doing cost analysis. It is so easy to look at something that costs $10,000 a month and think it's expensive, despite the fact that it saves twice that much in developer time (salary).

saurik 4y ago
How many people are using Fedora? Why must all of them suffer because you want to save a very small amount of your time? Why can't the developers--which include me!!--who want to profile things take the time to custom compile their boxes instead of demanding that all of the poor end users who don't even know what "profile" means have to take a reduction in performance? How many end users would have to exist to make you decide it isn't worth their pain and extra cost for you to have to spend a minutes of your time to help them? (Maybe then further answer that question under the scenario where the users in question by and large never actually install your software.)
- prmoustache 4y ago
  Fedora End-User vs Fedora Developer edition? What prevents anyone of us to maintain a fork with same SRPMS as fedora but keeping the omition of frame pointers? I guess someone will do that if there really is an observable performance advantage. There are already fedora forks such as Nobara providing some kernel patches and some specific compilation options or patches for some apps.
- spicyusername 4y ago
  I think many many many users would happily acquiesce, knowing doing so is helping to build a better product. Also 1-2% performance hit is peanuts, given the benefit. It isn't a religious battle of ideology, it's a concrete tradeoff of something small for something large.
  - saurik 4y ago

FullyFunctional 4y ago

I recently built SPECint2k6 for RISC-V (on JH7110) with and without -fomit-frame-pointer and to my surprise the default (with frame pointers) was a tiny bit faster (though so close to the noise floor that I'll settle for "the same"). I haven't dug into why frame pointers seem to cost nothing on RISC-V but it's likely that it's just that there's plenty of registers.

deleted 4y ago
[deleted]
xobs 4y ago
Unfortunately, it increases the code size by 10%. I was looking into this just last week, and can confirm that it's still a problem on the latest version of Rust nightly: https://github.com/tock/tock/pull/1660 https://github.com/tock/tock/pull/1660 I wish we could have frame pointers, because they would make working in embedded land so much easier and more reliable, but a 10% increase in code size just isn't worth it.
- andrekandre 4y ago
  perhaps slightly naive question, but whats the actual performance implication 10% increase in code size? (10% slower at runtime?)
  - MobiusHorizons 4y ago
    In the embedded world, you are limited by the amount of storage space available for your code. In a lot of cases the code is in flash or one time programmable rom memory on the microcontroller. Even if you have off-chip storage, you usually have to load the code into ram in order to run, and ram size is also quite limited. For low end microcontrollers it's common for rom / ram sizes to be on the order of 32kb (although they go much lower than that). In this world, the most common optimization level is -Os (or -Oz if available) because running out of storage space means you need a more expensive microcontroller, which increases the cost of each device you are selling, and might require a redesign of the PCB.

nullc 4y ago

I wonder what the total energy footprint of zillions of users using this who will never run a profiler. I imagine it's pretty substantial compared to the cost of putting a DWARF unwinder in the kernel.

Gigachad 4y ago
Essentially nothing in a world where almost everyone is running 7 electron apps at once and Windows has lost the ability to suspend properly. Especially when all those apps were built on these platforms exactly because of how annoying it is to debug C programs. The total energy usage could even be negative due to this.
- nullc 4y ago
  It's a fair point that other things waste unimaginably more, but in terms of debugging -- omitting the frame pointer hasn't been a debugging headache for a long time and even if it were that still wouldn't itself justify running it on every host under the sun including the vast majority that will never have any debugging run on it. I doubt C debugging has had a meaningful role in the use of electron in any case.
  - saagarjha 4y ago
    Not debugging, but profiling. They use similar infrastructure but have different needs when it comes to reliability and overhead.
rwmj 4y ago
You have to trade that against the large optimizations that were never made because performance analysis of the whole system was essentially unavailable. This is a very new change in Fedora, but hopefully we'll now see lots of big performance improvements in common workloads. The analysis I was doing when writing this blog post was about understanding where 50% of the time of my program was going. The fix literally doubled the speed of the program: https://bugzilla.redhat.com/show_bug.cgi?id=2166686 https://bugzilla.redhat.com/show_bug.cgi?id=2166686

quotemstr 4y ago

The biggest problem with frame pointer unwinding isn't even the performance hit, but the inability to support unwinding of anything that isn't AOT compiled native code. If you unwind using frame pointers through a Python program, you'll get a bunch of meaningless interpreter frames instead of information about the operation of the program being interpreted. That, and since the FP unwinder can't sleep and can't take page faults, it'll fail to traverse stack frames that happen to be paged out. The right way to solve the kernel unwind of userspace problem is to get the kernel out of the business of unwinding userspace. Concrete proposal here: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/IMBATOSTV7BIX7WQKMUWGDFYJG2TSBIF/ https://lists.fedoraproject.org/archives/list/devel@lists.fe... Unwinding only paged-in, AOT-compiled native code is a step in the wrong direction, especially because it's precisely the people for whom unwinding kinda-sorta works now (kernel, libc, C language diehards, etc.) who would need to approve and coordinate a more robust and comprehensive solution that also, unlike frame pointers, has zero cost when not in use.

rwmj 4y ago
You want the kernel to be able to collect the trace as fast as possible. You don't want to add half a million context switches to your trace.
- quotemstr 4y ago
  Where are the extra "context switches"? Can you count them for me? Under my proposal, we defer stack collection until we're going to return to user space anyway. The user stack of a thread cannot change while that thread is running in user mode.
  - rwmj 4y ago
    So after looking at this, I do now agree [can't edit my original comment]. Are you planning to implement anything like this? I think similar to the suggestion to fix DWARF, enabling frame pointers may not be the best solution, it may be reversed in future when we get something better, but it's the one that actually works right now.

throwawaylinux 4y ago

I wonder how significantly frame pointers damages the stack engine optimization in x86 processors. There's still a bunch of pushes and pops, I guess because that's shorter code than storing into an explicit offset off the stack pointer, but since you've read the stack pointer anyway and synchronized it, the uop sequence and instruction dependencies wouldn't end up being a lot different I would have thought.

rwmj 4y ago
It does damage it actually. GCC currently generates slightly pessimised code when frame pointers are enabled, for example always using the frame pointer to access stack variables even if it knows the offset from the stack pointer. I guess this is simply because distributions were all using -fomit-frame-pointer so the frame pointer case was neglected. Hopefully this will improve now that most distros have switched to enabling frame pointers.

javierhonduco 4y ago

I’m looking forward to this change. Ensuring fast, reliable stack traces when debugging under pressure or tracing complex systems is invaluable. Once you try it it’s hard to go back. It’ll take some years until frame pointers become more mainstream. Still, some people might decide to never enable them in libraries they use or their own code. That’s why I’ve been working on a BPF DWARF-based unwinder to fill the gap. It’s fully OSS (Apache 2.0 for user space and GPL for BPF) and should get out of beta in ~1month. Previous HN discussion [0]. Happy to help anybody that wants to integrate it in their tools / systems. - [0]: https://news.ycombinator.com/item?id=33788794 https://news.ycombinator.com/item?id=33788794

peter_retief 4y ago

I have moved my Linux Desktop to Fedora (Recommended by my 26yo son). Been in the Linux space since the beginning so have seen distros come and go. Who remembers Slackware? What a journey Linux has been. Hope the framepointers work out (I think they will squeeze out some more performance) Thanks to all the amazing people who developed the Linux miracle.

ilyt 4y ago
>Who remembers Slackware? I still have nightmares about it. What a terribly put together thing. Moving to Debian was such a breath of fresh air, packages that works!
- peter_retief 4y ago
  Kind of why no one uses it anymore I guess.
- prmoustache 4y ago
  What didn't work or wasn't well put together on slackware? The only thing that was annoying at the time was having to manually download and compile third party software as well as some of its depencency. In reality it wasn't that much of a big deal. You just had to know how to read a readme and install text file really. But this is kind of a moot point in 2023 with: 1. slackbuilds and its clients, which is pretty much like the AUR for archlinux users or pkgsrc and makes you a lot of additionnal software available very easily. 2. All the software nowadays made installable with flatpak, appimage or via some bash script. 3. podman and docker If anything modern software deployment methods made slackware experience closer to rpm or deb based distros than 20 years ago.
- dmpk2k 4y ago
  I had the exact opposite experience. Debian always had to add their own variations and special sauce to the configurations and layout. Slackware was as stock as you could get, effectively identical to what you'd get with a configure; make; make install. I much preferred stock.
gavinray 4y ago

jwilk 4y ago

Related: https://news.ycombinator.com/item?id=34426745 https://news.ycombinator.com/item?id=34426745 ("Fedora's tempest in a stack frame" from lwn.net, 17 days ago, 31 comments)

anarazel 4y ago

Heh. Looking for discussions in debian around this, I found a bug requesting a glibc build with frame pointers enabled. Reported by me. 8 years ago ...

javierhonduco 4y ago
That would be great! There’s a precedent in Ubuntu, in case it could help to push this forwards `libc6-prof`.
jwilk 4y ago
https://bugs.debian.org/767756 https://bugs.debian.org/767756

AlexeyBelov 4y ago

The author published an update: https://rwmj.wordpress.com/2023/02/06/frame-pointers-an-important-update/ https://rwmj.wordpress.com/2023/02/06/frame-pointers-an-impo...

bravetraveler 4y ago

I can't pretend to know enough about the actual implications of this, but I have to wonder - if the goal is to encourage contributors, why not grease the wheels further on fedpkg/mock? Why must we all have even slower performance? This is more of an aside - I love Fedora, but it already stands out of the pack as a bit slower than others. It's not particularly difficult to rebuild things as-is, is my point - I suppose. It all feels so self-fulfilling. People will want to profile (and benefit from this more)... because things that are context-switch heavy may have slowed down considerably. I'm considering how I may batch my own rebuilds to remove this option with these utilities, mainly to see apples-to-apples what the difference is I know they've published hand-wavy numbers/benchmarks, but I don't trust them for a multitude of reasons

pxc 4y ago
Some thorough benchmarks are something I'd love to read about here on HN. If you ever do those benchmarks, post 'em.
- bravetraveler 4y ago
  I'll be happy to share! I'll probably be waiting for the next release so I can get their pointer-enabled builds. However they do that may give me helpful hints RE: rebuilding the same sources without it. I know the tools to do it, but what I'm imagining right now is a bit convoluted. I'm open to suggestions on things to test. I want to avoid too much reliance on synthetics -- and my 'real world' differs like all of ours My original post has (currently) been down-struck a bit but you're the only reply. I appreciate you taking the time to engage I understand it seems dismissive, perhaps even aggressive, but the Fedora/SuSE people tend to set trends. I just want to measure them. The snark is me finding it ironic that I can (relatively easily) measure this with the same methods they want to avoid... rebuilds with the right options under mock / fedpkg subtext: not that many people find profiling interesting or important... but they will if you start messing with their performance. I find it interesting though so I'm not mad, I just fairly called it self-fulfilling -- I'm going to be profiling because the minority who regularly do, don't want to rebuild (like I will be)
- 4y ago