9 ms·

-fomit-frame-pointer is pretty high on the list of most destructive micro-optimizations in the history of open source. When you repurpose the backtrace pointer

by jart 4y ago

-fomit-frame-pointer is pretty high on the list of most destructive micro-optimizations in the history of open source. When you repurpose the backtrace pointer (RBP) for other things, it totally savages our ability to have native backtraces. All for a pointless ~1% performance boost. I've always thought it was some kind of grand conspiracy to obfuscate the behaviors of closed source software but for some reason every single Linux distro that claims to love freedom has to do this too. I don't want to trust binaries built by strangers on the Internet if they hide their behaviors. Back when I worked at Google I used to read a lot of the company-wide core build configs, and the first thing they always did was make sure -fno-omit-frame-pointer is always passed, no matter what build flags a specific team uses. Don't tell me DWARF is an acceptable alternative. While it can be nice sugar on top with Linux perf report when it actually works, frame pointers let me produce a backtrace in just ten lines of C without any dependencies. If I wanted to do the same with DWARF I'd need to link some GPL library or write 10,000 lines of byzantine code to figure out which function called the one that's crashing. Stuff like this is why everyone uses scripting languages these days, since the main value prop of high-level languages is their ability to reliably print backtraces.

cwzwarich 4y ago
> I don't want to trust binaries built by strangers on the Internet if they hide their behaviors. If a binary really wanted to hide its behavior (at least to the same extent that it could with -fomit-frame-pointer), it could just make fake stack frames beyond the current PC.
catiopatio 4y ago
DWARF is not just the better alternative, it’s the only correct solution. Not only do you get more complete and correct backtraces with DWARF, you also get preservation of stack-saved registers and the ability to reconstitute a subset of local state in a given frame. libunwind is MIT licensed, not GPL. I’ve also written a full DWARF stack unwinder myself, including a full DWARF expression interpreter — it’s nowhere near 10k lines of code.
- blinkingled 4y ago
  TFA actually gives an example where DWARF alone doesn't work.
  - Dylan16807 4y ago
    It doesn't give any insight as to why, though, which makes it very weak evidence toward whether DWARF is the real problem.
    - loeg 4y ago
      It's not so much DWARF is the problem as it is that the information is just lost in omit-framepointer mode. You can't reconstruct it from nothing. There's a related problem of compilers generating imperfect debuginfo that fails to reconstruct local variables from registers/memory due to an incomplete DWARF representation, but I don't think that plays a roll in unwinding. (But maybe something similar does.)
      - Dylan16807 4y ago
        What information is lost? The call stack can't be lost or you'd crash immediately.
        saagarjha 4y ago
        It's lost in the translation to DWARF. Of course the program knows how to unwind its own stack correctly, but explaining how to do this for arbitrary programs is untenable without DWARF to help, and it's annoying to parse and compilers don't emit it correctly.
        Dylan16807 4y ago
        If the information is lost by DWARF then I'd say that DWARF is clearly the problem. Unless DWARF can handle it fine and compilers are thoroughly broken, in which case it's a tossup between "DWARF is fundamentally flawed" and "compiler designers are lazy for some unknown reason"?
greesil 4y ago
Information like this is why I stay on this site. This explains so many things.
vlovich123 4y ago
I used gimli-rs to do backtraces and it did a fantastic job at great speed (faster even than LLVM’s symbolizer). It’s MIT-licensed and definitely not 10k lines. It might be a bit Byzantine in the implementation, but it’s all hidden behind a mostly easy-to-use API.
lanza 4y ago
Eh. If you worked at google you'd know that 1% performance improvement for C++ is measured in the 10s-100s of millions of dollars.
- charcircuit 4y ago
  For a minor performance decrease you in return get profiling data which allows you to identify areas where you can improve performance. For the most expensive lines of code it's important that you can optimize on production traffic. This results in a net saving of money over time.
  - ungamedplayer 4y ago
    But you are running code all the time, not fixing performance code all the time.
    - charcircuit 4y ago
      You are also updating the code all of the time. So it makes sense to be profiling all of the new changes.
      - ungamedplayer 4y ago
        I imagine the develper may be, but not even then not 'all' the time.
  - ilyt 4y ago
    If 1% saves you millions you want to recompile the code anyway to get use of optimizing for your current hardware, not generic distro target. Then you can turn on all flags you want.
    - saagarjha 4y ago
      That is exactly what Google does.
jhoechtl 4y ago
> since the main value prop of high-level languages is their ability to reliably print backtraces. Don't tell the golang folks! Besides the inbuilt panic behavior which prints a very helpful backtrace, non-fatal error handling is spinkled all over with fmt.Sprintf "I am here at 134738djfjrsj so you find me in the logs"
- pkulak 4y ago
  I consider Go a high-level language though.
- randomdata 4y ago
  Tell them what? The standard library's debug.PrintStack() reliably prints stack traces.
- ilyt 4y ago
  It takes very little effort to get backtrace together with log using any of the common logging libraries. Don't confuse problems in Go language with your own ineptitude to use it.
khuey 4y ago
Reclaiming %ebp for application use was worth a lot more than 1% back in the day. Now that pretty much everything is on architectures with much less register pressure than 32 bit x86 the tradeoff is more questionable.
- anarazel 4y ago
  From memory, gcc's switch to omitting frame pointers by default on x86 was in the late 2000s - when amd64 was at least clearly on the winning path, if it hadn't already. I understand changing it on 32bit x86, but...
deleted 4y ago
[deleted]
userbinator 4y ago
I've always thought it was some kind of grand conspiracy to obfuscate the behaviors of closed source software but for some reason every single Linux distro that claims to love freedom has to do this too. That's quite a stretch, and I'm saying this as someone who probably believes in things others would consider conspiracy theories. Freedom is irrelevant when you already have the source and the binaries and can do whatever you want with them... and the latter applies even if you're analysing closed-source software. All for a pointless ~1% performance boost. I can rebut with "All for a pointless waste of a register for something that will only be useful the <1% of the time the code is actually run." If I wanted to do the same with DWARF I'd need to link some GPL library or write 10,000 lines of byzantine code to figure out which function called the one that's crashing. Or just learn how to read a memory dump. If your code is somehow crashing so often enough that it becomes such a burden, then I'd say the solution is elsewhere. It's not hard to walk the stack manually.
- kccqzy 4y ago
  > I can rebut with "All for a pointless waste of a register for something that will only be useful the <1% of the time the code is actually run." When I worked at Google, profiling was done so often that it certainly exceeded 1%. There's something called Google-Wide Profiling[1] but teams often opt into much more frequent profiling of important micro services they deploy. It's not just for identifying crashes, but more for performance optimization. It's liberating to debug performance bottlenecks when you can just pull up a flame graph from half an hour ago and compare that against a flame graph from one week ago without worrying about re-deploying the old version to get profiling data. Btw I think this is somewhat of a cultural bias depending on your previous experience. Some people simply value performance at all costs. Some people value more observability and debugability. [1]: https://research.google/pubs/pub36575/ https://research.google/pubs/pub36575/
  - ilyt 4y ago
    > When I worked at Google, profiling was done so often that it certainly exceeded 1%. So a 10% use case for 0.001% of engineers out there. And I'm pretty sure you didn't run stock Fedora distro on servers either ? If you're that high end that profiling code for few % gains is worth millions you can just compile the stuff with whatever flags and optimization you need instead of relying on distro to reduce performance for everyone to appease to few engineers at big corp
jandrese 4y ago
> All for a pointless ~1% performance boost. With 32 bit code this was absolutely not the case. I have not looked so closely at 64 bit code, but in 32 bit land -fomit-frame-pointer was often closer to a 20-30% performance boost. Of all of gcc's optimization options it was the only one that I ever found to make a significant difference in real code.
- jiggawatts 4y ago
  I love these "rules of thumb", "black magic sorcery", and "best practices" that grey-haired old wizards pick up. More often than not, they're invalidated by some advancement in either computer hardware or software. Not just "no longer relevant", but often counterproductive. Worse still, these internalised tricks and rules are often related to Moore's law scaling, making them exponentially wrong over time. Not in the figurative sense, but a literal one. You see this with people claiming that I'm exaggerating when I say all code should be parallel. Meanwhile AMD is about to sell a 128 core / 256 thread processor. You can have two per host for a staggering 512 hardware threads. Single threaded code is capable of utilising just 0.2% of that machine! A similar "tuning best practice" that I still see applied in the field is database servers with dedicated data, log, temp, and templog drives... on cloud VMs where this is guaranteed to have worse performance than simply pooling the same amount of disk into a single logical drive. People pick these things up and hold on to them when they're wrong by orders of magnitude. I can't think of many (any?) other industries where more experienced senior staff can be this wrong about as many things...
  - cleanchit 4y ago
    I wish unzip, tar etc were multithreaded/
    - nix23 4y ago
      -https://zlib.net/pigz/ https://zlib.net/pigz/ -https://manpages.ubuntu.com/manpages/xenial/man1/pxz.1.html https://manpages.ubuntu.com/manpages/xenial/man1/pxz.1.html -https://lzip.nongnu.org/plzip.html https://lzip.nongnu.org/plzip.html And so on, however the Compression-ratio is slightly worse
randomdata 4y ago
> Stuff like this is why everyone uses scripting languages these days Do they? It seems like everyone is moving away from scripting languages for new projects these days, at least where web browser-related requirements doesn't push them into using Javascript. Indeed, before this current pendulum swing, scripting languages had a good run. I'm not convinced it was because of stack traces, though. They won hearts and minds on promises of no longer having to do "XML sit-ups" and whatnot.
- ilyt 4y ago
  Don't look at hackernews or reddit for what most people actually use. "Scripting languges" are doing just fine for majority of CRUD and CRM stuff most software houses produce.
  - throwaway2037 4y ago
    Crap, yes. The whole world is moving towards NodeJS / Web Browser / JavaScript. I dread it, but I cannot deny it.
  - randomdata 4y ago
    Where that language is Javascript due to the browser push towards it, sure. The exception was already addressed.
ilyt 4y ago
The article mentioned shows the worst case being closer yo 10% > Compiling the kernel turned out to be 2.4% slower, and a Blender test case regressed by 2%. The worst case appears to be Python programs, which can see as much as a 10% performance hit. To many, these costs were seen as unacceptable. Sacrificing that much for the absolute minority of people that need to dig that much into performance doesn't seem to be worth it for distro. Would be interesting to see why some cases still get that much regression tho.
- rwmj 4y ago
  That's a very specific case (the Python interpreter main loop), and for that reason the Python interpreter is still using -fomit-frame-pointer. It is specifically called out in the Fedora Change: https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointer#Benchmarking_of_the_performance_impact https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointe...
- mouse_ 4y ago
  While it may be true that only the absolute minority actually push computing forward and optimize things, sidelining that minority will have disastrous consequences for everyone.
- saagarjha 4y ago
  When performance people are telling you to leave some on the table, there's generally a very good reason for it.
- _0w8t 4y ago
  Crash logs for fp-less builds are rather useless. And useful logs are very valuable feedback that the users can provide to developers.
oso2k 4y ago
-fomit-frame-pointer is pretty high on the list of most destructive micro-optimizations in the history of open source. When you repurpose the backtrace pointer (RBP) for other things, it totally savages our ability to have native backtraces Funny. I always learned that the *BP registers stood for Base Pointer and not Backtrace. And then I would use BP to reference function temp variables from that part of the call frame. The Intel “ASM86 Language Reference Manual” calls it that as well. https://community.intel.com/cipcp26785/attachments/cipcp26785/c-compiler/31103/1/121703-003_ASM86_Language_Reference_Manual_Nov83.pdf https://community.intel.com/cipcp26785/attachments/cipcp2678...
- agapon 4y ago
  Yes, and that base pointer gets pushed on the stack when a function call is made. And that allows to accurately restore stack frames to get the backtrace. So, not sure what's funny.
derefr 4y ago
> but for some reason every single Linux distro that claims to love freedom has to do this too ... If I wanted to do the same with DWARF I'd need to link some GPL library or write 10,000 lines of byzantine code to figure out which function called the one that's crashing. I mean, presumably you just answered your own question: while corporate software might not be able to link GPL code, Linux software — that mostly already is GPLed — certainly can. In fact, that's presumably why the library you're referencing (libdwarf?) is under the GPL in the first place — it was created by a Linux developer, for use in their own and others' FOSS Linux software, where it easily solves this problem for them. And because they have this easy solution to this problem on hand, they don't really see the need to turn on frame pointers in their projects' Makefiles.

by jart 4y ago

cwzwarich 4y ago

> I don't want to trust binaries built by strangers on the Internet if they hide their behaviors. If a binary really wanted to hide its behavior (at least to the same extent that it could with -fomit-frame-pointer), it could just make fake stack frames beyond the current PC.

catiopatio 4y ago

DWARF is not just the better alternative, it’s the only correct solution. Not only do you get more complete and correct backtraces with DWARF, you also get preservation of stack-saved registers and the ability to reconstitute a subset of local state in a given frame. libunwind is MIT licensed, not GPL. I’ve also written a full DWARF stack unwinder myself, including a full DWARF expression interpreter — it’s nowhere near 10k lines of code.

blinkingled 4y ago
TFA actually gives an example where DWARF alone doesn't work.
- Dylan16807 4y ago
  It doesn't give any insight as to why, though, which makes it very weak evidence toward whether DWARF is the real problem.
  - loeg 4y ago
    It's not so much DWARF is the problem as it is that the information is just lost in omit-framepointer mode. You can't reconstruct it from nothing. There's a related problem of compilers generating imperfect debuginfo that fails to reconstruct local variables from registers/memory due to an incomplete DWARF representation, but I don't think that plays a roll in unwinding. (But maybe something similar does.)
    - Dylan16807 4y ago
      What information is lost? The call stack can't be lost or you'd crash immediately.
      - saagarjha 4y ago
        It's lost in the translation to DWARF. Of course the program knows how to unwind its own stack correctly, but explaining how to do this for arbitrary programs is untenable without DWARF to help, and it's annoying to parse and compilers don't emit it correctly.
        Dylan16807 4y ago
        If the information is lost by DWARF then I'd say that DWARF is clearly the problem. Unless DWARF can handle it fine and compilers are thoroughly broken, in which case it's a tossup between "DWARF is fundamentally flawed" and "compiler designers are lazy for some unknown reason"?

greesil 4y ago

Information like this is why I stay on this site. This explains so many things.

vlovich123 4y ago

I used gimli-rs to do backtraces and it did a fantastic job at great speed (faster even than LLVM’s symbolizer). It’s MIT-licensed and definitely not 10k lines. It might be a bit Byzantine in the implementation, but it’s all hidden behind a mostly easy-to-use API.

lanza 4y ago

Eh. If you worked at google you'd know that 1% performance improvement for C++ is measured in the 10s-100s of millions of dollars.

charcircuit 4y ago
For a minor performance decrease you in return get profiling data which allows you to identify areas where you can improve performance. For the most expensive lines of code it's important that you can optimize on production traffic. This results in a net saving of money over time.
- ungamedplayer 4y ago
  But you are running code all the time, not fixing performance code all the time.
  - charcircuit 4y ago
    You are also updating the code all of the time. So it makes sense to be profiling all of the new changes.
    - ungamedplayer 4y ago
      I imagine the develper may be, but not even then not 'all' the time.
- ilyt 4y ago
  If 1% saves you millions you want to recompile the code anyway to get use of optimizing for your current hardware, not generic distro target. Then you can turn on all flags you want.
  - saagarjha 4y ago
    That is exactly what Google does.

jhoechtl 4y ago

> since the main value prop of high-level languages is their ability to reliably print backtraces. Don't tell the golang folks! Besides the inbuilt panic behavior which prints a very helpful backtrace, non-fatal error handling is spinkled all over with fmt.Sprintf "I am here at 134738djfjrsj so you find me in the logs"

pkulak 4y ago
I consider Go a high-level language though.
randomdata 4y ago
Tell them what? The standard library's debug.PrintStack() reliably prints stack traces.
ilyt 4y ago
It takes very little effort to get backtrace together with log using any of the common logging libraries. Don't confuse problems in Go language with your own ineptitude to use it.

khuey 4y ago

Reclaiming %ebp for application use was worth a lot more than 1% back in the day. Now that pretty much everything is on architectures with much less register pressure than 32 bit x86 the tradeoff is more questionable.

anarazel 4y ago
From memory, gcc's switch to omitting frame pointers by default on x86 was in the late 2000s - when amd64 was at least clearly on the winning path, if it hadn't already. I understand changing it on 32bit x86, but...

deleted 4y ago

[deleted]

userbinator 4y ago

I've always thought it was some kind of grand conspiracy to obfuscate the behaviors of closed source software but for some reason every single Linux distro that claims to love freedom has to do this too. That's quite a stretch, and I'm saying this as someone who probably believes in things others would consider conspiracy theories. Freedom is irrelevant when you already have the source and the binaries and can do whatever you want with them... and the latter applies even if you're analysing closed-source software. All for a pointless ~1% performance boost. I can rebut with "All for a pointless waste of a register for something that will only be useful the <1% of the time the code is actually run." If I wanted to do the same with DWARF I'd need to link some GPL library or write 10,000 lines of byzantine code to figure out which function called the one that's crashing. Or just learn how to read a memory dump. If your code is somehow crashing so often enough that it becomes such a burden, then I'd say the solution is elsewhere. It's not hard to walk the stack manually.

kccqzy 4y ago
> I can rebut with "All for a pointless waste of a register for something that will only be useful the <1% of the time the code is actually run." When I worked at Google, profiling was done so often that it certainly exceeded 1%. There's something called Google-Wide Profiling[1] but teams often opt into much more frequent profiling of important micro services they deploy. It's not just for identifying crashes, but more for performance optimization. It's liberating to debug performance bottlenecks when you can just pull up a flame graph from half an hour ago and compare that against a flame graph from one week ago without worrying about re-deploying the old version to get profiling data. Btw I think this is somewhat of a cultural bias depending on your previous experience. Some people simply value performance at all costs. Some people value more observability and debugability. [1]: https://research.google/pubs/pub36575/ https://research.google/pubs/pub36575/
- ilyt 4y ago
  > When I worked at Google, profiling was done so often that it certainly exceeded 1%. So a 10% use case for 0.001% of engineers out there. And I'm pretty sure you didn't run stock Fedora distro on servers either ? If you're that high end that profiling code for few % gains is worth millions you can just compile the stuff with whatever flags and optimization you need instead of relying on distro to reduce performance for everyone to appease to few engineers at big corp

jandrese 4y ago

> All for a pointless ~1% performance boost. With 32 bit code this was absolutely not the case. I have not looked so closely at 64 bit code, but in 32 bit land -fomit-frame-pointer was often closer to a 20-30% performance boost. Of all of gcc's optimization options it was the only one that I ever found to make a significant difference in real code.

jiggawatts 4y ago
I love these "rules of thumb", "black magic sorcery", and "best practices" that grey-haired old wizards pick up. More often than not, they're invalidated by some advancement in either computer hardware or software. Not just "no longer relevant", but often counterproductive. Worse still, these internalised tricks and rules are often related to Moore's law scaling, making them exponentially wrong over time. Not in the figurative sense, but a literal one. You see this with people claiming that I'm exaggerating when I say all code should be parallel. Meanwhile AMD is about to sell a 128 core / 256 thread processor. You can have two per host for a staggering 512 hardware threads. Single threaded code is capable of utilising just 0.2% of that machine! A similar "tuning best practice" that I still see applied in the field is database servers with dedicated data, log, temp, and templog drives... on cloud VMs where this is guaranteed to have worse performance than simply pooling the same amount of disk into a single logical drive. People pick these things up and hold on to them when they're wrong by orders of magnitude. I can't think of many (any?) other industries where more experienced senior staff can be this wrong about as many things...
- cleanchit 4y ago
  I wish unzip, tar etc were multithreaded/
  - nix23 4y ago
    -https://zlib.net/pigz/ https://zlib.net/pigz/ -https://manpages.ubuntu.com/manpages/xenial/man1/pxz.1.html https://manpages.ubuntu.com/manpages/xenial/man1/pxz.1.html -https://lzip.nongnu.org/plzip.html https://lzip.nongnu.org/plzip.html And so on, however the Compression-ratio is slightly worse

randomdata 4y ago

> Stuff like this is why everyone uses scripting languages these days Do they? It seems like everyone is moving away from scripting languages for new projects these days, at least where web browser-related requirements doesn't push them into using Javascript. Indeed, before this current pendulum swing, scripting languages had a good run. I'm not convinced it was because of stack traces, though. They won hearts and minds on promises of no longer having to do "XML sit-ups" and whatnot.

ilyt 4y ago
Don't look at hackernews or reddit for what most people actually use. "Scripting languges" are doing just fine for majority of CRUD and CRM stuff most software houses produce.
- throwaway2037 4y ago
  Crap, yes. The whole world is moving towards NodeJS / Web Browser / JavaScript. I dread it, but I cannot deny it.
- randomdata 4y ago
  Where that language is Javascript due to the browser push towards it, sure. The exception was already addressed.

ilyt 4y ago

The article mentioned shows the worst case being closer yo 10% > Compiling the kernel turned out to be 2.4% slower, and a Blender test case regressed by 2%. The worst case appears to be Python programs, which can see as much as a 10% performance hit. To many, these costs were seen as unacceptable. Sacrificing that much for the absolute minority of people that need to dig that much into performance doesn't seem to be worth it for distro. Would be interesting to see why some cases still get that much regression tho.

rwmj 4y ago
That's a very specific case (the Python interpreter main loop), and for that reason the Python interpreter is still using -fomit-frame-pointer. It is specifically called out in the Fedora Change: https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointer#Benchmarking_of_the_performance_impact https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointe...
mouse_ 4y ago
While it may be true that only the absolute minority actually push computing forward and optimize things, sidelining that minority will have disastrous consequences for everyone.
saagarjha 4y ago
When performance people are telling you to leave some on the table, there's generally a very good reason for it.
_0w8t 4y ago
Crash logs for fp-less builds are rather useless. And useful logs are very valuable feedback that the users can provide to developers.

oso2k 4y ago

-fomit-frame-pointer is pretty high on the list of most destructive micro-optimizations in the history of open source. When you repurpose the backtrace pointer (RBP) for other things, it totally savages our ability to have native backtraces Funny. I always learned that the *BP registers stood for Base Pointer and not Backtrace. And then I would use BP to reference function temp variables from that part of the call frame. The Intel “ASM86 Language Reference Manual” calls it that as well. https://community.intel.com/cipcp26785/attachments/cipcp26785/c-compiler/31103/1/121703-003_ASM86_Language_Reference_Manual_Nov83.pdf https://community.intel.com/cipcp26785/attachments/cipcp2678...

agapon 4y ago
Yes, and that base pointer gets pushed on the stack when a function call is made. And that allows to accurately restore stack frames to get the backtrace. So, not sure what's funny.

derefr 4y ago

> but for some reason every single Linux distro that claims to love freedom has to do this too ... If I wanted to do the same with DWARF I'd need to link some GPL library or write 10,000 lines of byzantine code to figure out which function called the one that's crashing. I mean, presumably you just answered your own question: while corporate software might not be able to link GPL code, Linux software — that mostly already is GPLed — certainly can. In fact, that's presumably why the library you're referencing (libdwarf?) is under the GPL in the first place — it was created by a Linux developer, for use in their own and others' FOSS Linux software, where it easily solves this problem for them. And because they have this easy solution to this problem on hand, they don't really see the need to turn on frame pointers in their projects' Makefiles.