5 ms·

I was at Google in 2005 on the other side of the argument. My view back then was simple: Even if $BIG_COMPANY makes a decision to compile everything with frame

I was at Google in 2005 on the other side of the argument. My view back then was simple: Even if $BIG_COMPANY makes a decision to compile everything with frame pointers, the rest of the community is not. So we'll be stuck fighting an unwinnable argument with a much larger community. Turns out that it was a ~20 year argument. I ended up writing some patches to make libunwind work for gperftools and maintained libunwind for some number of years as a consequence of that work. Having moved on to other areas of computing, I'm now a passive observer. But it's fascinating to read history from the other perspective.

starspangled 3y ago
> So we'll be stuck fighting an unwinnable argument with a much larger community. In what way would you be stuck? What functional problems does adding frame pointers introduce?
- tempay 3y ago
  It “wastes” a register when you’re not actively using them. On x86 that can make a big difference, though with the added registers of x86_64 it much less significant.
  - nlewycky 3y ago
    It caused a problem when building inline assembly heavy code that tried to use all the registers, frame pointer register included.
  - starspangled 3y ago
    Right, but I was asking about functional problems (being "stuck"), which sounded like a big issue for the choice.
  - charleshn 3y ago
    It's not just the loss of an architectural register, it's also the added cost to the prologue/epilogue. Even on x86_64, it can make a difference, in particular for small functions, which might not be inlined for a variety of reasons.
    - Asooka 3y ago
      If your small function is not getting inlined, you should investigate why that is instead of globally breaking performance analysis of your code.
      - Sesse__ 3y ago
        A typical case would be C++ virtual member functions. (They can sometimes be devirtualized, or speculatively partially devirtualized, using LTO+PGO, but there are lots of legitimate cases where they cannot.)
    - kaba0 3y ago
      CPUs spend an enormous amount of time waiting for IO and memory, and push/pop and similar are just insanely well optimized. As the article also claims, I would be very surprised to see any effect, unless that more instructions themselves would spill the I-cache.
      - charleshn 3y ago
        I've seen around 1-3% on non micro benchmarks, real applications. Aee also this benchmark from Phoronix [0]: Of the 100 tests carried out for this article, when taking the geometric mean of all these benchmarks it equated to about a 14% performance penalty of the software with -O2 compared to when adding -fno-omit-frame-pointer. I'm not saying these benchmarks or the workloads I've seen are representative of the "real world", but people keep repeating that frame pointers are basically free, which is just not the case. [0] https://www.phoronix.com/review/fedora-frame-pointer https://www.phoronix.com/review/fedora-frame-pointer
  - inkyoto 3y ago
    Wasting a register on comparatively more modern ISA's (PA-RISC 2.0, MIPS64, POWER, aarch64 etc – they are all more modern and have an abundance of general purpose registers) is not a concern. The actual «wastage» is in having to generate a prologue and an epilogue for each function – 2x instructions to preserve the old frame pointer and set a new one up, and 2x instruction at the point of return – to restore the previous frame pointer. Generally, it is not a big deal with an exception of a pathological case of a very large number of very small functions calling each other frequently where the extra 4x instructions per each such a function will be filling up the L1 instruction cache «unnessarily».
    - weebull 3y ago
      Those pathological cases are really what inlining is for, with the exception of any tiny recursive functions that can't be tail call optimised.
      - inkyoto 3y ago
        Yes, inlining (and LTO can take it a notch or two higher) does away with the problem altogether, however the number of projects that default to «-Os» (or even to «-O2») to build a release product is substantial to large. There is also a significant number of projects that go to great lengths to force-override CFLAGS/CXXFLAGS (usually with «-O2 -g» or even with «-O») or make it extraordinary difficult to change the project's default CFLAGS, for no apparent reasons which eliminates a number of advanced optimisations in builds with default build settings.
- adsharma 3y ago
  I wasn't talking about functional problems. It was a simple observation that big companies were not going to convince Linux distributors to add frame pointers anytime soon and that what those distributors do is relevant. All of the companies involved believed that they were special and decided to build their own (poorly managed) distribution called "third party code" and having to deal with it was not my best experience working at these companies.
  - starspangled 3y ago
    Oh, I just assumed you were talking about Google's Linux distribution and applications it runs on its fleet. I must have mis-assumed. Re-reading... maybe you weren't talking about any builds but just whether or not to oppose kernel and toolchain defaulting to omit frame pointers?
    - adsharma 3y ago
      Google didn't have a Linux distribution for a long time (the one everyone used on the desktop was an outdated rpm based distro, we mostly ignored it for development purposes). What existed was a x86 to x86 cross compilation environment and the libraries involved were manually imported by developers who needed that particular library. My argument was about the cost of ensuring that those libraries were compiled with frame pointers when much of the open source community was defaulting to omit-fp.
      - dooglius 3y ago
        Would it not be easier to patch compilers to always assume the equivalent of -fno-omit-frame-pointer
        adsharma 3y ago
        That was done in 2005. But the task of auditing the supply chain to ensure that every single shared library you ever linked with was compiled a certain way was still hard. Nothing prevented an intern or a new employee from checking in a library without frame pointers into the third-party repo. In 2024, you'd probably create a "build container" that all developers are required to use to build binaries or pay a linux distributor to build that container. But cross compilation was the preferred approach back then. So all binaries had a rpath (run time search path to look for shared library) that ignored the distributor supplied libraries. Having come from a open source background, I found this system hard to digest. But there was a lot of social pressure to work as a bee in a system that thousands of other very competent engineers are using (quite successfully). I remember briefly talking to a chrome OS related group who were using the "build your own custom distro" approach, before deciding to move to another faang.
- deleted 3y ago
  [deleted]
- rwmj 3y ago
  You do get occasional regressions. eg. We found an extremely obscure bug involving enabling frame pointers, valgrind, glibc ifuncs and inlining (all at the same time): https://bugzilla.redhat.com/show_bug.cgi?id=2267598 https://bugzilla.redhat.com/show_bug.cgi?id=2267598 https://github.com/tukaani-project/xz/commit/82ecc538193b380a21622aea02b0ba078e7ade92 https://github.com/tukaani-project/xz/commit/82ecc538193b380...
  - oasisaimlessly 3y ago
    FYI, this "regression" was likely actually a symptom of a purposely-inserted backdoor. See [1] and my comment there [2]. [1]: https://news.ycombinator.com/item?id=39865810 https://news.ycombinator.com/item?id=39865810 [2]: https://news.ycombinator.com/item?id=39867301 https://news.ycombinator.com/item?id=39867301
jart 3y ago
[flagged]
- quotemstr 3y ago
  The clear and obvious win would have been adoption of a universal userspace generic unwind facility, like Windows has --- one that works with multiple languages. Turning on frame pointers is throwing in the towel on the performance tooling ecosystem coordination problem: we can't get people to fix unwind information, so we do this instead? Ugh.
  - rwmj 3y ago
    Yes, although the universal mechanisms that have been proposed so far have been quite ridiculous - for example having every program handle a "frame pointer signal" in userspace, which doesn't account for the reality that we need to do frame unwinding thousands of times a second with the least possible overhead. Frame pointers work for most things, and where they don't work (interpreted code) you're often not that interested in performance.
    - quotemstr 3y ago
      > every program handle a "frame pointer signal" in userspace Yep. That's my proposal. > which doesn't account for the reality that we need to do frame unwinding thousands of times a second with the least possible overhead Yes, it does. The kernel has to return to userspace anyway at some point, and pushing a signal frame during that return is cheap. The cost of signal delivery is the entry into the kernel, and after a perf counter overflow, you've already paid that cost. Why would the actual unwinding be any faster in the kernel than in userspace? Also, so what if a thread enters the kernel and samples the stack multiple times before returning to userspace? While in the kernel, the userspace stack cannot change --- therefore, it's sufficient to delay userspace stack collection until the kernel returns to userspace anyway. You might ask "Don't we have to restore the signal mask after handling the profiling signal?" Not if you don't define the signal to change the signal mask. sigreturn(2) is optional.
brcmthrowaway 3y ago
What area?

starspangled 3y ago

> So we'll be stuck fighting an unwinnable argument with a much larger community. In what way would you be stuck? What functional problems does adding frame pointers introduce?

tempay 3y ago
It “wastes” a register when you’re not actively using them. On x86 that can make a big difference, though with the added registers of x86_64 it much less significant.
- nlewycky 3y ago
  It caused a problem when building inline assembly heavy code that tried to use all the registers, frame pointer register included.
- starspangled 3y ago
  Right, but I was asking about functional problems (being "stuck"), which sounded like a big issue for the choice.
- charleshn 3y ago
  It's not just the loss of an architectural register, it's also the added cost to the prologue/epilogue. Even on x86_64, it can make a difference, in particular for small functions, which might not be inlined for a variety of reasons.
  - Asooka 3y ago
    If your small function is not getting inlined, you should investigate why that is instead of globally breaking performance analysis of your code.
    - Sesse__ 3y ago
      A typical case would be C++ virtual member functions. (They can sometimes be devirtualized, or speculatively partially devirtualized, using LTO+PGO, but there are lots of legitimate cases where they cannot.)
  - kaba0 3y ago
    CPUs spend an enormous amount of time waiting for IO and memory, and push/pop and similar are just insanely well optimized. As the article also claims, I would be very surprised to see any effect, unless that more instructions themselves would spill the I-cache.
    - charleshn 3y ago
      I've seen around 1-3% on non micro benchmarks, real applications. Aee also this benchmark from Phoronix [0]: Of the 100 tests carried out for this article, when taking the geometric mean of all these benchmarks it equated to about a 14% performance penalty of the software with -O2 compared to when adding -fno-omit-frame-pointer. I'm not saying these benchmarks or the workloads I've seen are representative of the "real world", but people keep repeating that frame pointers are basically free, which is just not the case. [0] https://www.phoronix.com/review/fedora-frame-pointer https://www.phoronix.com/review/fedora-frame-pointer
- inkyoto 3y ago
  Wasting a register on comparatively more modern ISA's (PA-RISC 2.0, MIPS64, POWER, aarch64 etc – they are all more modern and have an abundance of general purpose registers) is not a concern. The actual «wastage» is in having to generate a prologue and an epilogue for each function – 2x instructions to preserve the old frame pointer and set a new one up, and 2x instruction at the point of return – to restore the previous frame pointer. Generally, it is not a big deal with an exception of a pathological case of a very large number of very small functions calling each other frequently where the extra 4x instructions per each such a function will be filling up the L1 instruction cache «unnessarily».
  - weebull 3y ago
    Those pathological cases are really what inlining is for, with the exception of any tiny recursive functions that can't be tail call optimised.
    - inkyoto 3y ago
      Yes, inlining (and LTO can take it a notch or two higher) does away with the problem altogether, however the number of projects that default to «-Os» (or even to «-O2») to build a release product is substantial to large. There is also a significant number of projects that go to great lengths to force-override CFLAGS/CXXFLAGS (usually with «-O2 -g» or even with «-O») or make it extraordinary difficult to change the project's default CFLAGS, for no apparent reasons which eliminates a number of advanced optimisations in builds with default build settings.
adsharma 3y ago
I wasn't talking about functional problems. It was a simple observation that big companies were not going to convince Linux distributors to add frame pointers anytime soon and that what those distributors do is relevant. All of the companies involved believed that they were special and decided to build their own (poorly managed) distribution called "third party code" and having to deal with it was not my best experience working at these companies.
- starspangled 3y ago
  Oh, I just assumed you were talking about Google's Linux distribution and applications it runs on its fleet. I must have mis-assumed. Re-reading... maybe you weren't talking about any builds but just whether or not to oppose kernel and toolchain defaulting to omit frame pointers?
  - adsharma 3y ago
    Google didn't have a Linux distribution for a long time (the one everyone used on the desktop was an outdated rpm based distro, we mostly ignored it for development purposes). What existed was a x86 to x86 cross compilation environment and the libraries involved were manually imported by developers who needed that particular library. My argument was about the cost of ensuring that those libraries were compiled with frame pointers when much of the open source community was defaulting to omit-fp.
    - dooglius 3y ago
      Would it not be easier to patch compilers to always assume the equivalent of -fno-omit-frame-pointer
      - adsharma 3y ago
        That was done in 2005. But the task of auditing the supply chain to ensure that every single shared library you ever linked with was compiled a certain way was still hard. Nothing prevented an intern or a new employee from checking in a library without frame pointers into the third-party repo. In 2024, you'd probably create a "build container" that all developers are required to use to build binaries or pay a linux distributor to build that container. But cross compilation was the preferred approach back then. So all binaries had a rpath (run time search path to look for shared library) that ignored the distributor supplied libraries. Having come from a open source background, I found this system hard to digest. But there was a lot of social pressure to work as a bee in a system that thousands of other very competent engineers are using (quite successfully). I remember briefly talking to a chrome OS related group who were using the "build your own custom distro" approach, before deciding to move to another faang.
deleted 3y ago
[deleted]
rwmj 3y ago
You do get occasional regressions. eg. We found an extremely obscure bug involving enabling frame pointers, valgrind, glibc ifuncs and inlining (all at the same time): https://bugzilla.redhat.com/show_bug.cgi?id=2267598 https://bugzilla.redhat.com/show_bug.cgi?id=2267598 https://github.com/tukaani-project/xz/commit/82ecc538193b380a21622aea02b0ba078e7ade92 https://github.com/tukaani-project/xz/commit/82ecc538193b380...
- oasisaimlessly 3y ago
  FYI, this "regression" was likely actually a symptom of a purposely-inserted backdoor. See [1] and my comment there [2]. [1]: https://news.ycombinator.com/item?id=39865810 https://news.ycombinator.com/item?id=39865810 [2]: https://news.ycombinator.com/item?id=39867301 https://news.ycombinator.com/item?id=39867301

jart 3y ago

[flagged]

quotemstr 3y ago
The clear and obvious win would have been adoption of a universal userspace generic unwind facility, like Windows has --- one that works with multiple languages. Turning on frame pointers is throwing in the towel on the performance tooling ecosystem coordination problem: we can't get people to fix unwind information, so we do this instead? Ugh.
- rwmj 3y ago
  Yes, although the universal mechanisms that have been proposed so far have been quite ridiculous - for example having every program handle a "frame pointer signal" in userspace, which doesn't account for the reality that we need to do frame unwinding thousands of times a second with the least possible overhead. Frame pointers work for most things, and where they don't work (interpreted code) you're often not that interested in performance.
  - quotemstr 3y ago
    > every program handle a "frame pointer signal" in userspace Yep. That's my proposal. > which doesn't account for the reality that we need to do frame unwinding thousands of times a second with the least possible overhead Yes, it does. The kernel has to return to userspace anyway at some point, and pushing a signal frame during that return is cheap. The cost of signal delivery is the entry into the kernel, and after a perf counter overflow, you've already paid that cost. Why would the actual unwinding be any faster in the kernel than in userspace? Also, so what if a thread enters the kernel and samples the stack multiple times before returning to userspace? While in the kernel, the userspace stack cannot change --- therefore, it's sufficient to delay userspace stack collection until the kernel returns to userspace anyway. You might ask "Don't we have to restore the signal mask after handling the profiling signal?" Not if you don't define the signal to change the signal mask. sigreturn(2) is optional.

brcmthrowaway 3y ago

What area?