12 ms·

The return of the frame pointers

684 points by mfiguiere 3y ago

sroussey 3y ago
-fomit-frame-pointer Had serious performance improvements back in the day. I would always compile my own MySQL and PHP and save money on hardware. But those days has 32bit processors with few registers. Times change.
teleforce 3y ago
I'm surprised that most the comments are concerned about the telemetry and observabilty per se rather the potential of frame pointers to the programmers at large. This can provide beneficial insight for hard to debug programming constructs for examples delegates where C# and D have but not Rust, and also CTFE where D and Zig have but not Rust. In the new future I can foresee is that seamless frame pointers integration with programming language constructs can ease debugging of these extremely useful higher order function constructs like delegates. It also has the potential to enable futuristic programming paradigm as proposed by Bret Victor's presentation [2]. [1] D Language Reference: Function Pointers, Delegates and Closures: https://dlang.org/spec/function.html#function-pointers-delegates https://dlang.org/spec/function.html#function-pointers-deleg... [2] Bret Victor The Future of Programming: https://youtu.be/8pTEmbeENF4 https://youtu.be/8pTEmbeENF4
- deleted 3y ago
  [deleted]
deleted 3y ago
[deleted]
simscitizen 3y ago
All this talk about frame pointers enabled on various Linux distros and we still haven’t talked about the biggest one, which would be Android. Have they decided to compile with frame pointers yet? The last time I looked in to this many years ago, some perf folks on the Android team dismissed it as a perf regression (and also ART was already using the FP register for its own purposes).
matheusmoreira 3y ago
That's really interesting. I disabled frame pointer omission in my project because I read that it hurts debugging and code introspection and provided only minimal performance benefits. I had no idea it had caused so much pain over decades.
adsharma 3y ago
I was at Google in 2005 on the other side of the argument. My view back then was simple: Even if $BIG_COMPANY makes a decision to compile everything with frame pointers, the rest of the community is not. So we'll be stuck fighting an unwinnable argument with a much larger community. Turns out that it was a ~20 year argument. I ended up writing some patches to make libunwind work for gperftools and maintained libunwind for some number of years as a consequence of that work. Having moved on to other areas of computing, I'm now a passive observer. But it's fascinating to read history from the other perspective.
- starspangled 3y ago
  > So we'll be stuck fighting an unwinnable argument with a much larger community. In what way would you be stuck? What functional problems does adding frame pointers introduce?
  - tempay 3y ago
    It “wastes” a register when you’re not actively using them. On x86 that can make a big difference, though with the added registers of x86_64 it much less significant.
    - nlewycky 3y ago
      It caused a problem when building inline assembly heavy code that tried to use all the registers, frame pointer register included.
    - starspangled 3y ago
      Right, but I was asking about functional problems (being "stuck"), which sounded like a big issue for the choice.
    - charleshn 3y ago
      It's not just the loss of an architectural register, it's also the added cost to the prologue/epilogue. Even on x86_64, it can make a difference, in particular for small functions, which might not be inlined for a variety of reasons.
pajko 3y ago
There's another option: https://lesenechal.fr/en/linux/unwinding-the-stack-the-hard-way https://lesenechal.fr/en/linux/unwinding-the-stack-the-hard-...
- loeg 3y ago
  Brendan mentions DWARF unwinding, actually, and briefly mentions why he considers it insufficient.
  - haberman 3y ago
    The biggest objection seems to be the Java/JIT case. eh_frame supports a "personality function" which is AIUI basically a callback for performing custom unwinding. If the personality function could also support custom logic for producing backtraces, then the profiling sampler could effectively read the JVM's own metadata about the JIT'ted code, which I assume it must have in order to produce backtraces for the JVM itself.
    - loeg 3y ago
      This also seems like a big objection: > The overhead to walk DWARF is also too high, as it was designed for non-realtime use.
      - kouteiheika 3y ago
        Not a problem in practice. The way you solve it is to just translate DWARF into a simpler representation that doesn't require you to walk anything. (But I understand why people don't want to do it. DWARF is insanely complex and annoying to deal with.) Source: I wrote multiple profilers.
        brendangregg 3y ago
        For a busy 64-CPU production JVM, I tested Google's Java symbol logging agent that just logged timestamp, symbol, address, size. The c2 compiler was so busy, constantly, that the overhead of this was too high to be practical (beyond startup analysis). And all this was generating was a timestamp log to do symbol lookup. For DWARF to walk stacks there's a lot more steps, so while I could see it work for light workloads I doubt it's practical for the heavy production workloads I typically analyze. What do you think? Have you tested on a large production server where c2 is a measurable portion of CPU constantly, the code cache is >1Gbyte and under heavy load?
dap 3y ago
Good post! > Profiling has been broken for 20 years and we've only now just fixed it. It was a shame when they went away. Lots of people, certainly on other systems and probably Linux too, have found the absence of frame pointers painful this whole time and tried to keep them available in as many environments as possible. It’s validating (if also kind of frustrating) to see mainstream Linux bring them back.
- trws 3y ago
  I’m sincerely curious. While I realize that using dwarf for unwinding is annoying, why is it so bad that it’s worth pessimizing all code on the system? It’s slow on Debian derivatives because they package only the slow unwinding path for perf for example, for license reasons, but with decent tooling I barely notice the difference. What am I missing?
  - deleted 3y ago
    [deleted]
  - BeeOnRope 3y ago
    Which is the slow unwinding path? The one from libbfd?
  - brenns10 3y ago
    Using DWARF is annoying with perf because the kernel doesn't support stack unwinding with DWARF, and never will. The kernel has to be the one which unwinds the user space stacks, because it is the one managing the perf counters and handling the interrupts. Since it can't unwind the user space stacks, the kernel has to copy the entire stack (8192 bytes by default) into the output file (perf.data) and then the perf user space program will unwind it later. It does this for each sample, which is usually hundreds of times per second, per CPU. Though it depends how you configured the collection. That does have a significant overhead: first, runtime overhead: copying 8k bytes, hundreds of times per second, and writing it to disk, all don't come for free. You spend quite a bit of CPU time doing the memcpy operation which consumes memory bandwidth too. You also frequently need to increase the size of the perf memory buffer to accommodate all this data while it waits for user space to write it to disk. Second, disk space overhead, since the 8k stack bytes per sample are far larger than the stack trace would be. And third, it does require that you install debuginfo packages to get the DWARF info, which is usually a pain to do on production machines, and they consume a lot of disk space on their own. Many of these overheads aren't too bad in simple cases (lower sample rates, fewer CPUs, or targeting a single task). But with larger machines with hundreds of CPUs, full system collections, and higher frequencies, the overhead can increase exponentially. I'm not certain I know what you mean by the "slow unwinding path for perf", as there is no faster path for user space when frame pointers are disabled (except Intel LBR as outlined in the blog).
ngcc_hk 3y ago
It said gcc. I noted the default of llvm said to default with framepounter from 2011. Is this mainly a gcc issue?
- bawolff 3y ago
  It doesn't really matter what the default of the compiler is, but what distros chose.
userbinator 3y ago
[flagged]
- dap 3y ago
  I’m not sure what use case you’re coming from but it sounds like you’re saying something like: most end users don’t use a profiler or debugger so why should they pay the cost of debuggability? That’s fine I guess if you’re throwing software over a wall to users and taking no responsibility for their experience. But a lot of people who build software do take some responsibility for bugs and performance problems that their users experience. This stuff is invaluable for them. End users benefit (tremendously) from software being debuggable even if the users themselves never run a profiler or debugger that uses the frame pointers (because the developers are able to find and fix problems reported by other users or the developers themselves).
  - userbinator 3y ago
    [flagged]
    - ivlad 3y ago
      While I somewhat support the idea of avoiding debug-mode applications shipped to end customers, it seems in this particular case Brendan argues about the need to debug binaries in runtime on the server side. If there is a binary component running in production, the choice of running it with debug enabled (and/or with an ability to activate debug in runtime) is purely choice of the system owners (who nowadays are both the developers and the operators of components). “Observability” primarily refers to ability to view system state beyond of what blackbox monitoring does. Again, the term primarily refers to server side operations and not to software shipped to end users. As much as spying on users is ugly, it’s not related to debugging server side.
- supriyo-biswas 3y ago
  https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html > Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes. > Please don't fulminate. Please don't sneer, including at the rest of the community.
WalterBright 3y ago
Guess I'll add it back in to the DMD code generator!
Joker_vD 3y ago
Of course, if you cede RBP to be a frame pointer, you may as well have two stacks, one which is pointed into by RBP and stores the activation frames, and the other one which is pointed into by RSP and stores the return addresses only. At this point, you don't even need to "walk the stack" because the call stack is literally just a flat array of return addresses. Why do we normally store the return addresses near to the local variables in the first place, again? There are so many downsides.
- naasking 3y ago
  It simplifies storage management. A stack frame is a simple bump pointer which is always in cache and only one guard page for overflow, in your proposal you need two guard pages and double the stack manipulations and doubling the chance of a cache miss.
  - Joker_vD 3y ago
    Yes, two guard pages are needed. No, the stack management stays the same: it's just "CALL func" at the call site, "SUB RBP, <frame_size>" at the prologue and "ADD RBP, <frame_size>; RET" at the epilogue. As for chances of a cache miss... probably, but I guess you also double them up when you enable CFET/Shadow Stack so eh. In exchange, it becomes very difficult for the stack smashing to corrupt the return address.
  - imtringued 3y ago
    The reduceron had five stacks and it was faster because of it.
- dan-robertson 3y ago
  Note the ‘shadow stacks’ CPU feature mentioned briefly in the article, though it’s more for security reasons. It’s pretty similar to what you describe.
  - rwmj 3y ago
    Shadow stacks have been proposed as an alternative, although it's my understanding that in current CPUs they hold only a limited number of frames, like 16 or 32?
claytonwramsey 3y ago
That's very interesting to me - I had seen the `[unknown]` mountain in my profiles but never knew why. I think it's a tough thing to justify: 2% performance is actually a pretty big difference. It would be really nice to have fine-grained control over frame pointer inclusion: provided fine-grained profiling, we could determine whether we needed the frame pointers for a given function or compilation unit. I wouldn't be surprised if we see that only a handful of operations are dramatically slowed by frame pointer inclusion while the rest don't really care.
- naasking 3y ago
  > 2% performance is actually a pretty big difference. No it's not, particularly when it can help you identify hotspots via profiling that can net you improvements of 10% or more.
  - pm215 3y ago
    Sure, but how many of the people running distro compiled code do perf analysis? And how many of the people who need to do perf analysis are unable to use a with-frame-pointers version when they need to? And how many of those 10% perf improvements are in common distro code that get upstreamed to improve general user experience, as opposed to being in private application code? If you're netflix then "enable frame pointers" is a no-brainer. But if you're a distro who's building code for millions of users, many of whom will likely never need to fire up a profiler, I think the question is at least a little trickier. The overall best tradeoff might end up being still to enable frame pointers, but I can see the other side too.
    - jart 3y ago
      It's not a technical tradeoff, it's a refusal to compromise. Lack of frame pointers prevents many groups from using software built by distros altogether. If a distro decides that they'd rather make things go 1% faster for grandma, at the cost of alienating thousands of engineers at places like Netflix and Google who simply want to volunteer millions of dollars of their employers resources helping distros to find 10x performance improvements, then the distros are doing a great disservice to both grandma and themselves.
tdullien 3y ago
As much as the return of frame pointers is a good thing, it's largely unnecessary -- it arrives at a point where multiple eBPF-based profilers are available that do fine using .eh_frame and also manually unwinding high level language runtime stacks: Both Parca from PolarSignals as well the artist formerly known as Prodfiler (now Elastic Universal Profiling) do fine. So this is a solution for a problem, and it arrives just at the moment that people have solved the problem more generically ;) (Prodfiler coauthor here, we had solved all of this by the time we launched in Summer 2021)
- Tomte 3y ago
  You mean we don‘t need accessible profiling in free software because there are companies selling it to us. Cool.
  - brancz 3y ago
    Parca's user-space code is apache2 and the eBPF code is GPL.
  - tdullien 3y ago
    Parca is open-source, Prodfiler's eBPF code is GPL, and the rest of Prodfiler is currently going through OTel donation, so my point is: There's now multiple FOSS implementations of a more generic and powerful technique.
- int_19h 3y ago
  PolarSignals is specifically discussed in the linked threads, and they conclude that their approach is not good enough for perf reasons.
  - tdullien 3y ago
    Oh nice, I can't find that - can you post a link?
  - javierhonduco 3y ago
    Curious to hear more about this. Full disclosure: I designed and implemented .eh_frame unwinding when I worked at Polar Signals.
5- 3y ago
so what is the downside to using e.g. dwarf-based stack walking (supported by perf) for libc, which was the original stated problem? in the discussion the issue gets conflated with jit-ted languages, but that has nothing to do with the crusade to enable frame pointer for system libraries. and if you care that much for dwarf overhead... just cache the unwind information in your system-level profiler? no need to rebuild everything.
- yxhuvud 3y ago
  The article explains why DWARF is not an option.
  - menaerus 3y ago
    Extremely light on the details, and also conflates it with the JIT which makes it harder to understand the point, so I was wondering about the same thing as well.
- brancz 3y ago
  The way perf does it is slow, as the entire stack is copied into user-space and is then asynchronously unwound. This is solvable as Brendan calls out, we’ve created an eBPF-based profiler at Polar Signals, that essentially does what you said, it optimized the unwind tables, caches them in bpf maps, and then synchronously unwinds as opposed to copying the whole stack into user-space.
  - stefan_ 3y ago
    This conveniently sidesteps the whole issue of getting DWARF data in the first place, which is also still a broken disjointed mess on Linux. Hell, Windows solved this many many years ago.
    - bregma 3y ago
      You'd need a pretty special distro to have enabled -fno-asynchronous-unwind-tables by default in its toolchain. By default on most Linux distros the frame tables are built into all the binaries, and end up in the GNU_EH_FRAME segment, which is always available in any running process. Doesn't sound a broken and disjointed mess to me. Sounds more like a smoothly running solved problem.
ReleaseCandidat 3y ago
That's one thing Apple did do right on ARM: > The frame pointer register (x29) must always address a valid frame record. Some functions — such as leaf functions or tail calls — may opt not to create an entry in this list. As a result, stack traces are always meaningful, even without debug information. https://developer.apple.com/documentation/xcode/writing-arm64-code-for-apple-platforms https://developer.apple.com/documentation/xcode/writing-arm6...
- microtherion 3y ago
  On Apple platforms, there is often an interpretability problem of another kind: Because of the prevalence of deeply nested blocks / closures, backtraces for Objective C / Swift apps are often spread across numerous threads. I don't know of a good solution for that yet.
  - felixge 3y ago
    I'm not very familiar with Objective C and Swift, so this might not make sense. But JS used to have a similar problem with async/await. The v8 engine solved it by walking the chain of JS promises to recover the "logical stack" developers are interested in [1]. [1] https://v8.dev/blog/fast-async https://v8.dev/blog/fast-async
    - astrange 3y ago
      Swift concurrency does a similar thing. For the older dispatch blocks, Xcode injects a library that records backtraces over thread hops.
eqvinox 3y ago
This doesn't detract from the content at all but the register counts are off; SI and DI count as GPRs on i686 bringing it to 6+BP (not 4+BP) meanwhile x86_64 has 14+BP (not 16+BP).
- cesarb 3y ago
  > [...] on i686 bringing it to 6+BP (not 4+BP) meanwhile x86_64 has 14+BP (not 16+BP). That is, on i686 you have 7 GPRs without frame pointers, while on x86_64 you have 14 GPRs even with frame pointers. Copying a comment of mine from an older related discussion (https://news.ycombinator.com/item?id=38632848 https://news.ycombinator.com/item?id=38632848): "To emphasize this point: on 64-bit x86 with frame pointers, you have twice as many registers as on 32-bit x86 without frame pointers, and these registers are twice as wide. A 64-bit value (more common than you'd expect even when pointers are 32 bits) takes two registers on 32-bit x86, but only a single register on 64-bit x86."
  - brendangregg 3y ago
    Thanks!
benreesman 3y ago
Brendan is such a treasure to the community (buy his book it’s great). I wasn’t doing extreme performance stuff when -fomit-frame-pointer became the norm, so maybe it was a big win for enough people to be a sane default, but even that seems dubious: “just works” profiling is how you figure out when you’re in an extreme performance scenario (if you’re an SG14 WG type, you know it and are used to all the defaults being wrong for you). I’m deeply grateful for all the legends who have worked on libunwind, gperf stuff, perftool, DTrace, eBPF: these are the too-often-unsung heroes of software that is still fast after decades of Moore’s law free-riding. But they’ve been fighting an uphill battle against a weird alliance of people trying to game compiler benchmarks and the really irresponsible posture that “developer time is more expensive” which is only sometimes true and never true if you care about people on low-spec gear, which is the community of users who that is already the least-resourced part of the global community. I’m fortunate enough to have a fairly modern desktop, laptop, and phone: for me it’s merely annoying that chat applications and music players and windowing systems offer nothing new except enshittification in terms of features while needing 10-100x the resources they did a decade ago. But for half of my career and 2/3rds of my time coding, I was on low-spec gear most of the time, and I would have been largely excluded if people didn’t care a lot about old computers back then. I’m trying to help a couple of aspiring hackers get started right now it’s a real struggle to get their environments set up with limitations like Intel Macs and WSL2 as the Linux option (WSL2 is very cool but it’s not loved up enough by e.g. yarn projects). If you want new hackers, you need to make things work well on older computers. Thanks again Brendan et al!
sesm 3y ago
glibc is only 2 MB, why Chrome relies on system glibc instead of statically linking their own version with frame pointers enabled?
- wruza 3y ago
  https://stackoverflow.com/questions/57476533/why-is-statically-linking-glibc-discouraged https://stackoverflow.com/questions/57476533/why-is-statical... I guess the similar situation with msvcrt’s.
- nolist_policy 3y ago
  At the very least Chrome needs to link to the system libGL.so and friends for gpu acceleration, libva.so for video acceleration, and so on. And these are linked against glibc of course.
  - dooglius 3y ago
    having/omitting frame pointers doesn't change the ABI; it will work if you compile against glibc-nofp and link against glibc-withfp
dsign 3y ago
I remember when the omission of stack frame pointers started spreading at the beginning of the 2000s. I was in college at the time, studying computer sciences in a very poor third-world country. Our computers were old and far from powerful. So, for most course projects, we would eschew interprets and use compilers. Mind you, what my college lacked in money it compensated by having interesting course work. We studied and implemented low level data-structures, compilers, assembly-code numerical routines and even a device driver for Minix. During my first two years in college, if one of our programs did something funny, I would attach gdb and see what was happening at assembly level. I got used to "walking the stack" manually, though the debugger often helped a lot. Happy times, until all of the sudden, "-fomit-frame-pointer" was all the rage, and stack traces stopped making sense. Just like that, debugging that segfault or illegal instruction became exponentially harder. A short time later, I started using Python for almost everything to avoid broken debugging sessions. So, I lost an order of magnitude or two with "-fomit-frame-pointer". But learning Python served me well for other adventures.
- nextaccountic 3y ago
  Didn't you know about -fno-omit-frame-pointer?
  - e40 3y ago
    Perhaps they were talking about libraries they didn’t compile, and with languages with callbacks this can be a real problem.
  - noobermin 3y ago
    You unfortunately do not control everything on your machine and how it's compiled. I think source vased distros like gentoo are the only exception but have their own warts.
- POVKING_89 3y ago
  [flagged]
  - 3y ago
rwmj 3y ago
I'm glad he mentioned Fedora because it's been a tiresome battle to keep frame pointers enabled in the whole distribution (eg https://pagure.io/fesco/issue/3084 https://pagure.io/fesco/issue/3084). There's a persistent myth that frame pointers have a huge overhead, because there was a single Python case that had a +10% slow down (now fixed). The actual measured overhead is under 1%, which is far outweighed by the benefits we've been able to make in certain applications.
- sidewndr46 3y ago
  I'm still baffled by this attitude. That "under 1%" overhead is why computers are measurably slower than 30 years ago to use. All those "under 1%" overhead add up
- brendangregg 3y ago
  Thanks; what was the Python fix?
  - rwmj 3y ago
    This was the investigation: https://discuss.python.org/t/python-3-11-performance-with-frame-pointers/22507 https://discuss.python.org/t/python-3-11-performance-with-fr... Initially we just turned off frame pointers for the Python 3.9 interpreter in Fedora. They are back on in Python 3.12 where it seems the upstream bug has been fixed, although I can't find the actual fix right now. Fedora tracking bug: https://bugzilla.redhat.com/2158729 https://bugzilla.redhat.com/2158729 Fedora change in Python 3.9 to disable frame pointers: https://src.fedoraproject.org/rpms/python3.9/c/9b71f8369141c1840c8d87db1655fcefc576ddff?branch=rawhide https://src.fedoraproject.org/rpms/python3.9/c/9b71f8369141c...
    - brendangregg 3y ago
      Ah right, thanks, I remember I saw Andrii's analysis in the other thread. https://pagure.io/fesco/issue/2817#comment-826636 https://pagure.io/fesco/issue/2817#comment-826636
- 3y ago
zzbn00 3y ago
NiX (and I assume Guix) are very convenient for this as it is fairly easy to turn frame pointers on or off for parts or whole of the system.
tkiolp4 3y ago
Are his books (the one about Systems Performance and eBPF) relevant for normal software engineers who want to improve performance in normal services? I don’t work for faang, and our usual performance issues are solved by adding indexes here and there, caching, and simple code analysis. Tools like Datadog help a lot already.
- wavemode 3y ago
  Diving into flame graphs being worthwhile for optimization, assumes that your workload is CPU-bound. Most business software does not have such workloads, and rather (as you yourself have noted) spend most of their time waiting for I/O (database, network, filesystem, etc). And so, (as you again have noted), your best bet is to just use plain old logging and tracing (like what datadog provides) to find out where the waiting is happening.
- polio 3y ago
  Profiling is a pretty basic technique that is applicable to all software engineering. I'm not sure what a "normal" service is here, but I think we all have an obligation to understand what's happening in the systems we own. Some people may believe that 100ms latency is acceptable for a CLI tool, but what if it could be 3ms? On some aesthetic level, it also feels good to be able to eliminate excess. Finally, you should learn it because you won't necessarily have that job forever.
mgaunard 3y ago
You don't need frame pointers, all the relevant info is stored in dwarf debug data.
DaveFlater 3y ago
GCC optimization causes the frame pointer push to move around, resulting in wrong call stacks. "Wontfix" https://news.ycombinator.com/item?id=38896343 https://news.ycombinator.com/item?id=38896343
- rwmj 3y ago
  That was in 2012. Does it still occur on modern GCC? There definitely have been regressions with frame pointers being enabled, although we've fixed all the ones we've found in current (2024) Fedora.
  - ndesaulniers 3y ago
    Pretty sure unwinding thumb generated by GCC is still non-unwindable via FPs. That's been a pain.
  - jart 3y ago
    I think so and I vaguely seem to recall -fno-schedule-insns2 being the only thing that fixes it. To get the full power of frame pointers and hackable binary, what I use is: -fno-schedule-insns2 -fno-omit-frame-pointers -fno-optimize-sibling-calls -mno-omit-leaf-frame-pointer -fpatchable-function-entry=18,16 -fno-inline-functions-called-once The only flag that's potentially problematic is -fno-optimize-sibling-calls since it breaks the optimal approach to writing interpreters and slows down code that's written in a more mathematical style.
tzot 3y ago
I am not sure, but I believe -fomit-frame-pointer in x86-64 allows the compiler to use a _thirteenth_ register, not a _seventeenth_ .
cesarb 3y ago
I disagree with this sentence of the article: "I could say that times have changed and now the original 2004 reasons for omitting frame pointers are no longer valid in 2024." The original 2004 reason for omitting frame pointers is still valid in 2024: it's still a big performance win on the register-starved 32-bit x86 architecture. What has changed is that the 32-bit x86 architecture is much less relevant nowadays (other than legacy software, for most people it's only used for a small instant while starting up the firmware), and other common 32-bit architectures (like embedded 32-bit ARM) are not as register-starved as the 32-bit x86.
- IshKebab 3y ago
  That's exactly what they were saying. You're not disagreeing at all.
deleted 3y ago
[deleted]
shaggie76 3y ago
I thought we'd been using /Oy (Frame-Pointer Omission) for years on Windows and that there was a pdata section on x64 that was used for stack-walking however to my great surprise I just read on MSDN that "In x64 compilers, /Oy and /Oy- are not available." Does this mean Microsoft decided they weren't going to support breaking profilers and debuggers OR is there some magic in the pdata section that makes it work even if you omit the frame-pointer?
- Iwan-Zotow 3y ago
  MS had unwinding support done right for a LONG time, 32 and 64, starting with structured C exceptions
- MarkSweep 3y ago
  Some Google found this: https://devblogs.microsoft.com/oldnewthing/20130906-00/?p=3303 https://devblogs.microsoft.com/oldnewthing/20130906-00/?p=33... “Recovering a broken stack on x64 machines on Windows is trickier because the x64 uses unwind codes for stack walking rather than a frame pointer chain.” More details are here: https://learn.microsoft.com/en-us/cpp/build/exception-handling-x64?view=msvc-170 https://learn.microsoft.com/en-us/cpp/build/exception-handli...
- quotemstr 3y ago
  Microsoft has had excellent universal unwinding support for decades now. I'm disappointed to see someone as prominent as this article's author present as infeasible what Microsoft has had working for so long.
  - Iwan-Zotow 3y ago
    exactly! starting with C structured exceptions
- musjleman 3y ago
  > In x64 compilers The default is omission. If you have a Windows machine, in all likelihood almost no 64 bit code running on it has frame pointers. > OR is there some magic in the pdata section that makes it work even if you omit the frame-pointer You haven't ever needed frame pointers to unwind using ... unwind information. The same thing exists for linux as `.eh_frame` section.
javierhonduco 3y ago
Overall, I am for frame pointers, but after some years working in this space, I thought I would share some thoughts: * Many frame pointer unwinders don't account for a problem they have that DWARF unwind info doesn't have: the fact that the frame set-up is not atomic, it's done in two instructions, `push $rbp` and `mov $rsp $rbp`, and if when a snapshot is taken we are in the `push`, we'll miss the parent frame. I think this might be able to be fired by inspecting the code, but I think this might only be as good as a heuristic as there could be other `push %rbp` unrelated to the stack frame. I would love to hear if there's a better approach! * I developed the solution Brendan mentions which allows faster, in-kernel unwinding without frame pointers using BPF [0]. This doesn't use DWARF CFI (the unwind info) as-is but converts it into a random-access format that we can use in BPF. He mentions not supporting JVM languages, and while it's true that right now it only supports JIT sections that have frame pointers, I planned to implement a full JVM interpreter unwinder. I have left Polar Signals since and shifted priorities but it's feasible to get a JVM unwinder to work in lockstep with the native unwinder. * In an ideal world, enabling frame pointers should be done on a case-by-case. Benchmarking is key, and the tradeoffs that you make might change a lot depending on the industry you are in, and what your software is doing. In the past I have seen large projects enabling/disabling frame pointers not doing an in-depth assessment of losses/gains of performance, observability, and how they connect to business metrics. The Fedora folks have done a superb and rigorous job here. * Related to the previous point, having a build system that enables you to change this system-wide, including libraries your software depends on can be awesome to not only test these changes but also put them in production. * Lastly, I am quite excited about SFrame that Indu is working on. It's going to solve a lot of the problems we are facing right now while letting users decide whether they use frame pointers. I can't wait for it, but I am afraid it might take several years until all the infrastructure is in place and everybody upgrades to it. - [0]: https://web.archive.org/web/20231222054207/https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-based-stack-walking-using-ebpf https://web.archive.org/web/20231222054207/https://www.polar...
- spc476 3y ago
  It could be one instruction: ENTER N,0 (where N is the amount of stack space to reserve for locals)---this is the same as: PUSH EBP MOV ESP,ESP SUB SP,N (I don't recall if ENTER is x86-64 or not). But even with this, the frame setup isn't atomic with respect to CALL, and if the snapshot is taken after the CALL but before the ENTER, we still don't get the fame setup. As for the reason why ENTER isn't used, it was deemed too slow. LEAVE (MOV SP,BP; POP BP) is used as it's just as fast as, if not faster, than the sequence it replaces. If ENTER were just the PUSH/MOV/SUB sequence, it probably would be used, but it's that other operand (which is 0 above in my example) that kills it performance wise (it's for nested functions to gain access to upper stack frames and is every expensive to use).
secondcoming 3y ago
Can they not be disabled on a per-function basis?
weebull 3y ago
Just as a general comment on this topic... The fact that people complain about the performance of the mechanism that enables the system to be profiled, and so performance problems be identified, is beyond ironic. Surely the epitome of premature optimisation.
- account42 3y ago
  It's not moronic when the people profiling and the people having performance issues are different groups. That some developer benefits from having frame pointers in libc does not mean that all users of that software also need to have frame pointers enabled.
- im3w1l 3y ago
  It goes both ways. I see people with ultra-bloated applications trying to add even more bloat and force it on the rest of us too. Like the guy saying that DWARF unwinding is impractical when you have 1Gbyte code cache. I don't see the people writing hyperfast low-level code asking for this. My experience with profiling is anyway that you don't need that fine-grained profiling. It's main use is finding stuff like "we spend 90% of our time reallocating a string over and over to add characters one at a time". After a few of those it's just "it's a little bit slow everywhere".
  - Sesse__ 3y ago
    > After a few of those it's just "it's a little bit slow everywhere". And after that, you need fine-grained profiling to find multiple 1% wins and apply them repeatedly. (I do this for a living, basically)
    - im3w1l 3y ago
      Like disabling frame pointers
- doubloon 3y ago
  im sure in ancient mesopotamia there was somebody arguing about you could brew beer faster if you stop measuring the hops so carefully but then someone else was saying yes but if you dont measure the hops carefully then you dont know the efficiency of your overall beer making process so you cant isolate the bottlenecks. the funny thing is i am not sure if the world would actually work properly if we didn't have both of these kinds of people.
Cold_Miserable 3y ago
Not interesting. Enter/leave also does the same thing as your save/restore rbp. Far more interesting I recall there might be an instruction where rbp isn't allowed.
titzer 3y ago
Virgil doesn't use frame pointers. If you don't have dynamic stack allocation, the frame of a given function has a fixed size can be found with a simple (binary-search) table lookup. Virgil's technique uses an additional page-indexed range that further restricts the lookup to be a few comparisons on average (O(log(# retpoints per page)). It combines the unwind info with stackmaps for GC. It takes very little space. The main driver is in (https://github.com/titzer/virgil/blob/master/rt/native/NativeStackWalker.v3); https://github.com/titzer/virgil/blob/master/rt/native/Nativ... the rest of the code in the directory implements the decoding of metadata. I think frame pointers only make sense if frames are dynamically-sized (i.e. have stack allocation of data). Otherwise it seems weird to me that a dynamic mechanism is used when a static mechanism would suffice; mostly because no one agreed on an ABI for the metadata encoding, or an unwind routine. I believe the 1-2% measurement number. That's in the same ballpark as pervasive checks for array bounds checks. It's weird that the odd debugging and profiling task gets special pleading for a 1% cost but adding a layer of security gets the finger. Very bizarre priorities.
- moonchild 3y ago
  You can add bounds checks to c, but that costs a hell of a lot more than 1-2%. C++ has them off by default for std::vector because c++ is designed by and for the utterly insane. Other than that, I can't off the top of my head think of a language that doesn't have them.
  - shrimp_emoji 3y ago
    > C++ has them off by default for std::vector because c++ is designed by and for the utterly insane. And for those who value performance and don't want to pay the cost of "a lot more than 1-2%" ;p
    - moonchild 3y ago
      std::vector falls into the category of things that are easy to bounds check, st the cost, even under today's primitive compilers, is low. It's direct pointer accesses—which are common in c but not in c++ or most other languages—that are hard to and therefore cost more to bounds check.
codeflo 3y ago
All of this information is static, there's no need to sacrifice a whole CPU register only to store data that's already known. A simple lookup data structure that maps an instruction address range to the stack offset of the return address should be enough to recover the stack layout. On Windows, you'd precompute that from PDB files, I'm sure you can do the same thing with whatever the equivalent debug data structure is on Linux.
- fsmv 3y ago
  [deleted]
  - deleted 3y ago
    [deleted]
- loeg 3y ago
  It isn't entirely static because of alloca().
  - account42 3y ago
    Banning alloca() would make more sense than re-enabling frame pointers.
    - loeg 3y ago
      At least re-enabling FPs is possible.
mikewarot 3y ago
I started programming in 1979, and I can't believe I've managed to avoid learning about stack frames all those EBP register tricks until now. I always had parameters to functions in registers, not on the stack, for the most part. The compiler hid a lot of things from me. Is it because I avoided Linux and C most of my life? Perhaps it's because I used debug, and Periscope before that... and never gdb?
boulos 3y ago
JIT'ed code is sadly poorly supported, but LLVM has had great hooks for noting each method that is produced and its address. So you can build a simple mixed-mode unwinder, pretty easily, but mostly in process. I think Intel's DNN things dump their info out to some common file that perf can read instead, but because the *kernels* themselves reuse rbp throughout oneDNN, it's totally useless. Finally, can any JVM folks explain this claim about DWARF info from the article: > Doesn't exist for JIT'd runtimes like the Java JVM that just sounds surprising to me. Is it off by default or literally not available? (Google searches have mostly pointed to people wanting to include the JNI/C side of a JVM stack, like https://github.com/async-profiler/async-profiler/issues/215 https://github.com/async-profiler/async-profiler/issues/215).
POVKING_89 3y ago
[flagged]
POVKING_89 3y ago
[flagged]
olliej 3y ago
Have compilers (or I guess x86?) gotten better at dealing with the frame pointer? Or are we just saying that taking a significant perf hit is acceptable if it lets you find other tiny perf problems? Because I recall -fomit-frame-pointer being a significant performance win, bigger than most of the things that you need a perfect profiler to spot.
vlovich123 3y ago
To this day I still believe that there should be a dedicated protected separate stack region for the call stack that only the CPU can write to/read from. Walking the stack then becomes trivially fast because you just need to do a very small memcpy. And stack memory overflows can never overwrite the return address.
- ndesaulniers 3y ago
  This is a thing; it's called shadow call stack. Both ARM and now Intel have extensions for it.
  - vlovich123 3y ago
    But the shadow stack concept seems much dumber to me. Why write the address to the regular stack and the shadow stack and then compare? Why not only use the shadow stack and not put return addresses on the main stack at all.
    - BeeOnRope 3y ago
      ABI compatibility.
      - vlovich123 3y ago
        When you're compiling from scratch, especially when you're not working on a shared library, ABI compatability doesn't matter as much. Doesn't explain why there's no -fshadow-stack-only option to pass in.
        BeeOnRope 3y ago
        > Doesn't explain why there's no -fshadow-stack-only option to pass in. I thought you were asking about the design of the hardware: it's designed that way because compatibility means that the vast majority of people want something backwards compatible. Very little is fully "compiled from scratch" when you consider for example that libc is in the set of things that must be recompiled.
BinaryRage 3y ago
I remember talking to Brendan about the PreserveFramePointer patch during my first months at Netflix in 2015. As of JDK 21, unfortunately it is no longer a general purpose solution for the JVM, because it prevents a fast path being taken for stack thawing for virtual threads: https://github.com/openjdk/jdk/blob/d32ce65781c1d7815a69ceac720cdf3ae39caa9e/src/hotspot/share/runtime/continuationFreezeThaw.cpp#L1807 https://github.com/openjdk/jdk/blob/d32ce65781c1d7815a69ceac...

684 points by mfiguiere 3y ago

-fomit-frame-pointer Had serious performance improvements back in the day. I would always compile my own MySQL and PHP and save money on hardware. But those days has 32bit processors with few registers. Times change.

teleforce 3y ago

I'm surprised that most the comments are concerned about the telemetry and observabilty per se rather the potential of frame pointers to the programmers at large. This can provide beneficial insight for hard to debug programming constructs for examples delegates where C# and D have but not Rust, and also CTFE where D and Zig have but not Rust. In the new future I can foresee is that seamless frame pointers integration with programming language constructs can ease debugging of these extremely useful higher order function constructs like delegates. It also has the potential to enable futuristic programming paradigm as proposed by Bret Victor's presentation [2]. [1] D Language Reference: Function Pointers, Delegates and Closures: https://dlang.org/spec/function.html#function-pointers-delegates https://dlang.org/spec/function.html#function-pointers-deleg... [2] Bret Victor The Future of Programming: https://youtu.be/8pTEmbeENF4 https://youtu.be/8pTEmbeENF4

deleted 3y ago
[deleted]

deleted 3y ago

[deleted]

simscitizen 3y ago

All this talk about frame pointers enabled on various Linux distros and we still haven’t talked about the biggest one, which would be Android. Have they decided to compile with frame pointers yet? The last time I looked in to this many years ago, some perf folks on the Android team dismissed it as a perf regression (and also ART was already using the FP register for its own purposes).

matheusmoreira 3y ago

That's really interesting. I disabled frame pointer omission in my project because I read that it hurts debugging and code introspection and provided only minimal performance benefits. I had no idea it had caused so much pain over decades.

adsharma 3y ago

I was at Google in 2005 on the other side of the argument. My view back then was simple: Even if $BIG_COMPANY makes a decision to compile everything with frame pointers, the rest of the community is not. So we'll be stuck fighting an unwinnable argument with a much larger community. Turns out that it was a ~20 year argument. I ended up writing some patches to make libunwind work for gperftools and maintained libunwind for some number of years as a consequence of that work. Having moved on to other areas of computing, I'm now a passive observer. But it's fascinating to read history from the other perspective.

starspangled 3y ago
> So we'll be stuck fighting an unwinnable argument with a much larger community. In what way would you be stuck? What functional problems does adding frame pointers introduce?
- tempay 3y ago
  It “wastes” a register when you’re not actively using them. On x86 that can make a big difference, though with the added registers of x86_64 it much less significant.
  - nlewycky 3y ago
    It caused a problem when building inline assembly heavy code that tried to use all the registers, frame pointer register included.
  - starspangled 3y ago
    Right, but I was asking about functional problems (being "stuck"), which sounded like a big issue for the choice.
  - charleshn 3y ago
    It's not just the loss of an architectural register, it's also the added cost to the prologue/epilogue. Even on x86_64, it can make a difference, in particular for small functions, which might not be inlined for a variety of reasons.

pajko 3y ago

There's another option: https://lesenechal.fr/en/linux/unwinding-the-stack-the-hard-way https://lesenechal.fr/en/linux/unwinding-the-stack-the-hard-...

loeg 3y ago
Brendan mentions DWARF unwinding, actually, and briefly mentions why he considers it insufficient.
- haberman 3y ago
  The biggest objection seems to be the Java/JIT case. eh_frame supports a "personality function" which is AIUI basically a callback for performing custom unwinding. If the personality function could also support custom logic for producing backtraces, then the profiling sampler could effectively read the JVM's own metadata about the JIT'ted code, which I assume it must have in order to produce backtraces for the JVM itself.
  - loeg 3y ago
    This also seems like a big objection: > The overhead to walk DWARF is also too high, as it was designed for non-realtime use.
    - kouteiheika 3y ago
      Not a problem in practice. The way you solve it is to just translate DWARF into a simpler representation that doesn't require you to walk anything. (But I understand why people don't want to do it. DWARF is insanely complex and annoying to deal with.) Source: I wrote multiple profilers.
      - brendangregg 3y ago
        For a busy 64-CPU production JVM, I tested Google's Java symbol logging agent that just logged timestamp, symbol, address, size. The c2 compiler was so busy, constantly, that the overhead of this was too high to be practical (beyond startup analysis). And all this was generating was a timestamp log to do symbol lookup. For DWARF to walk stacks there's a lot more steps, so while I could see it work for light workloads I doubt it's practical for the heavy production workloads I typically analyze. What do you think? Have you tested on a large production server where c2 is a measurable portion of CPU constantly, the code cache is >1Gbyte and under heavy load?

dap 3y ago

Good post! > Profiling has been broken for 20 years and we've only now just fixed it. It was a shame when they went away. Lots of people, certainly on other systems and probably Linux too, have found the absence of frame pointers painful this whole time and tried to keep them available in as many environments as possible. It’s validating (if also kind of frustrating) to see mainstream Linux bring them back.

trws 3y ago
I’m sincerely curious. While I realize that using dwarf for unwinding is annoying, why is it so bad that it’s worth pessimizing all code on the system? It’s slow on Debian derivatives because they package only the slow unwinding path for perf for example, for license reasons, but with decent tooling I barely notice the difference. What am I missing?
- deleted 3y ago
  [deleted]
- BeeOnRope 3y ago
  Which is the slow unwinding path? The one from libbfd?
- brenns10 3y ago
  Using DWARF is annoying with perf because the kernel doesn't support stack unwinding with DWARF, and never will. The kernel has to be the one which unwinds the user space stacks, because it is the one managing the perf counters and handling the interrupts. Since it can't unwind the user space stacks, the kernel has to copy the entire stack (8192 bytes by default) into the output file (perf.data) and then the perf user space program will unwind it later. It does this for each sample, which is usually hundreds of times per second, per CPU. Though it depends how you configured the collection. That does have a significant overhead: first, runtime overhead: copying 8k bytes, hundreds of times per second, and writing it to disk, all don't come for free. You spend quite a bit of CPU time doing the memcpy operation which consumes memory bandwidth too. You also frequently need to increase the size of the perf memory buffer to accommodate all this data while it waits for user space to write it to disk. Second, disk space overhead, since the 8k stack bytes per sample are far larger than the stack trace would be. And third, it does require that you install debuginfo packages to get the DWARF info, which is usually a pain to do on production machines, and they consume a lot of disk space on their own. Many of these overheads aren't too bad in simple cases (lower sample rates, fewer CPUs, or targeting a single task). But with larger machines with hundreds of CPUs, full system collections, and higher frequencies, the overhead can increase exponentially. I'm not certain I know what you mean by the "slow unwinding path for perf", as there is no faster path for user space when frame pointers are disabled (except Intel LBR as outlined in the blog).

ngcc_hk 3y ago

It said gcc. I noted the default of llvm said to default with framepounter from 2011. Is this mainly a gcc issue?

bawolff 3y ago
It doesn't really matter what the default of the compiler is, but what distros chose.

userbinator 3y ago

[flagged]

dap 3y ago
I’m not sure what use case you’re coming from but it sounds like you’re saying something like: most end users don’t use a profiler or debugger so why should they pay the cost of debuggability? That’s fine I guess if you’re throwing software over a wall to users and taking no responsibility for their experience. But a lot of people who build software do take some responsibility for bugs and performance problems that their users experience. This stuff is invaluable for them. End users benefit (tremendously) from software being debuggable even if the users themselves never run a profiler or debugger that uses the frame pointers (because the developers are able to find and fix problems reported by other users or the developers themselves).
- userbinator 3y ago
  [flagged]
  - ivlad 3y ago
    While I somewhat support the idea of avoiding debug-mode applications shipped to end customers, it seems in this particular case Brendan argues about the need to debug binaries in runtime on the server side. If there is a binary component running in production, the choice of running it with debug enabled (and/or with an ability to activate debug in runtime) is purely choice of the system owners (who nowadays are both the developers and the operators of components). “Observability” primarily refers to ability to view system state beyond of what blackbox monitoring does. Again, the term primarily refers to server side operations and not to software shipped to end users. As much as spying on users is ugly, it’s not related to debugging server side.
supriyo-biswas 3y ago
https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html > Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes. > Please don't fulminate. Please don't sneer, including at the rest of the community.

WalterBright 3y ago

Guess I'll add it back in to the DMD code generator!

Joker_vD 3y ago

Of course, if you cede RBP to be a frame pointer, you may as well have two stacks, one which is pointed into by RBP and stores the activation frames, and the other one which is pointed into by RSP and stores the return addresses only. At this point, you don't even need to "walk the stack" because the call stack is literally just a flat array of return addresses. Why do we normally store the return addresses near to the local variables in the first place, again? There are so many downsides.

naasking 3y ago
It simplifies storage management. A stack frame is a simple bump pointer which is always in cache and only one guard page for overflow, in your proposal you need two guard pages and double the stack manipulations and doubling the chance of a cache miss.
- Joker_vD 3y ago
  Yes, two guard pages are needed. No, the stack management stays the same: it's just "CALL func" at the call site, "SUB RBP, <frame_size>" at the prologue and "ADD RBP, <frame_size>; RET" at the epilogue. As for chances of a cache miss... probably, but I guess you also double them up when you enable CFET/Shadow Stack so eh. In exchange, it becomes very difficult for the stack smashing to corrupt the return address.
- imtringued 3y ago
  The reduceron had five stacks and it was faster because of it.
dan-robertson 3y ago
Note the ‘shadow stacks’ CPU feature mentioned briefly in the article, though it’s more for security reasons. It’s pretty similar to what you describe.
- rwmj 3y ago
  Shadow stacks have been proposed as an alternative, although it's my understanding that in current CPUs they hold only a limited number of frames, like 16 or 32?

claytonwramsey 3y ago

That's very interesting to me - I had seen the `[unknown]` mountain in my profiles but never knew why. I think it's a tough thing to justify: 2% performance is actually a pretty big difference. It would be really nice to have fine-grained control over frame pointer inclusion: provided fine-grained profiling, we could determine whether we needed the frame pointers for a given function or compilation unit. I wouldn't be surprised if we see that only a handful of operations are dramatically slowed by frame pointer inclusion while the rest don't really care.

naasking 3y ago
> 2% performance is actually a pretty big difference. No it's not, particularly when it can help you identify hotspots via profiling that can net you improvements of 10% or more.
- pm215 3y ago
  Sure, but how many of the people running distro compiled code do perf analysis? And how many of the people who need to do perf analysis are unable to use a with-frame-pointers version when they need to? And how many of those 10% perf improvements are in common distro code that get upstreamed to improve general user experience, as opposed to being in private application code? If you're netflix then "enable frame pointers" is a no-brainer. But if you're a distro who's building code for millions of users, many of whom will likely never need to fire up a profiler, I think the question is at least a little trickier. The overall best tradeoff might end up being still to enable frame pointers, but I can see the other side too.
  - jart 3y ago
    It's not a technical tradeoff, it's a refusal to compromise. Lack of frame pointers prevents many groups from using software built by distros altogether. If a distro decides that they'd rather make things go 1% faster for grandma, at the cost of alienating thousands of engineers at places like Netflix and Google who simply want to volunteer millions of dollars of their employers resources helping distros to find 10x performance improvements, then the distros are doing a great disservice to both grandma and themselves.

tdullien 3y ago

As much as the return of frame pointers is a good thing, it's largely unnecessary -- it arrives at a point where multiple eBPF-based profilers are available that do fine using .eh_frame and also manually unwinding high level language runtime stacks: Both Parca from PolarSignals as well the artist formerly known as Prodfiler (now Elastic Universal Profiling) do fine. So this is a solution for a problem, and it arrives just at the moment that people have solved the problem more generically ;) (Prodfiler coauthor here, we had solved all of this by the time we launched in Summer 2021)

Tomte 3y ago
You mean we don‘t need accessible profiling in free software because there are companies selling it to us. Cool.
- brancz 3y ago
  Parca's user-space code is apache2 and the eBPF code is GPL.
- tdullien 3y ago
  Parca is open-source, Prodfiler's eBPF code is GPL, and the rest of Prodfiler is currently going through OTel donation, so my point is: There's now multiple FOSS implementations of a more generic and powerful technique.
int_19h 3y ago
PolarSignals is specifically discussed in the linked threads, and they conclude that their approach is not good enough for perf reasons.
- tdullien 3y ago
  Oh nice, I can't find that - can you post a link?
- javierhonduco 3y ago
  Curious to hear more about this. Full disclosure: I designed and implemented .eh_frame unwinding when I worked at Polar Signals.

5- 3y ago

so what is the downside to using e.g. dwarf-based stack walking (supported by perf) for libc, which was the original stated problem? in the discussion the issue gets conflated with jit-ted languages, but that has nothing to do with the crusade to enable frame pointer for system libraries. and if you care that much for dwarf overhead... just cache the unwind information in your system-level profiler? no need to rebuild everything.

yxhuvud 3y ago
The article explains why DWARF is not an option.
- menaerus 3y ago
  Extremely light on the details, and also conflates it with the JIT which makes it harder to understand the point, so I was wondering about the same thing as well.
brancz 3y ago
The way perf does it is slow, as the entire stack is copied into user-space and is then asynchronously unwound. This is solvable as Brendan calls out, we’ve created an eBPF-based profiler at Polar Signals, that essentially does what you said, it optimized the unwind tables, caches them in bpf maps, and then synchronously unwinds as opposed to copying the whole stack into user-space.
- stefan_ 3y ago
  This conveniently sidesteps the whole issue of getting DWARF data in the first place, which is also still a broken disjointed mess on Linux. Hell, Windows solved this many many years ago.
  - bregma 3y ago
    You'd need a pretty special distro to have enabled -fno-asynchronous-unwind-tables by default in its toolchain. By default on most Linux distros the frame tables are built into all the binaries, and end up in the GNU_EH_FRAME segment, which is always available in any running process. Doesn't sound a broken and disjointed mess to me. Sounds more like a smoothly running solved problem.

ReleaseCandidat 3y ago

That's one thing Apple did do right on ARM: > The frame pointer register (x29) must always address a valid frame record. Some functions — such as leaf functions or tail calls — may opt not to create an entry in this list. As a result, stack traces are always meaningful, even without debug information. https://developer.apple.com/documentation/xcode/writing-arm64-code-for-apple-platforms https://developer.apple.com/documentation/xcode/writing-arm6...

microtherion 3y ago
On Apple platforms, there is often an interpretability problem of another kind: Because of the prevalence of deeply nested blocks / closures, backtraces for Objective C / Swift apps are often spread across numerous threads. I don't know of a good solution for that yet.
- felixge 3y ago
  I'm not very familiar with Objective C and Swift, so this might not make sense. But JS used to have a similar problem with async/await. The v8 engine solved it by walking the chain of JS promises to recover the "logical stack" developers are interested in [1]. [1] https://v8.dev/blog/fast-async https://v8.dev/blog/fast-async
  - astrange 3y ago
    Swift concurrency does a similar thing. For the older dispatch blocks, Xcode injects a library that records backtraces over thread hops.

eqvinox 3y ago

This doesn't detract from the content at all but the register counts are off; SI and DI count as GPRs on i686 bringing it to 6+BP (not 4+BP) meanwhile x86_64 has 14+BP (not 16+BP).

cesarb 3y ago
> [...] on i686 bringing it to 6+BP (not 4+BP) meanwhile x86_64 has 14+BP (not 16+BP). That is, on i686 you have 7 GPRs without frame pointers, while on x86_64 you have 14 GPRs even with frame pointers. Copying a comment of mine from an older related discussion (https://news.ycombinator.com/item?id=38632848 https://news.ycombinator.com/item?id=38632848): "To emphasize this point: on 64-bit x86 with frame pointers, you have twice as many registers as on 32-bit x86 without frame pointers, and these registers are twice as wide. A 64-bit value (more common than you'd expect even when pointers are 32 bits) takes two registers on 32-bit x86, but only a single register on 64-bit x86."
- brendangregg 3y ago
  Thanks!

benreesman 3y ago

Brendan is such a treasure to the community (buy his book it’s great). I wasn’t doing extreme performance stuff when -fomit-frame-pointer became the norm, so maybe it was a big win for enough people to be a sane default, but even that seems dubious: “just works” profiling is how you figure out when you’re in an extreme performance scenario (if you’re an SG14 WG type, you know it and are used to all the defaults being wrong for you). I’m deeply grateful for all the legends who have worked on libunwind, gperf stuff, perftool, DTrace, eBPF: these are the too-often-unsung heroes of software that is still fast after decades of Moore’s law free-riding. But they’ve been fighting an uphill battle against a weird alliance of people trying to game compiler benchmarks and the really irresponsible posture that “developer time is more expensive” which is only sometimes true and never true if you care about people on low-spec gear, which is the community of users who that is already the least-resourced part of the global community. I’m fortunate enough to have a fairly modern desktop, laptop, and phone: for me it’s merely annoying that chat applications and music players and windowing systems offer nothing new except enshittification in terms of features while needing 10-100x the resources they did a decade ago. But for half of my career and 2/3rds of my time coding, I was on low-spec gear most of the time, and I would have been largely excluded if people didn’t care a lot about old computers back then. I’m trying to help a couple of aspiring hackers get started right now it’s a real struggle to get their environments set up with limitations like Intel Macs and WSL2 as the Linux option (WSL2 is very cool but it’s not loved up enough by e.g. yarn projects). If you want new hackers, you need to make things work well on older computers. Thanks again Brendan et al!

sesm 3y ago

glibc is only 2 MB, why Chrome relies on system glibc instead of statically linking their own version with frame pointers enabled?

wruza 3y ago
https://stackoverflow.com/questions/57476533/why-is-statically-linking-glibc-discouraged https://stackoverflow.com/questions/57476533/why-is-statical... I guess the similar situation with msvcrt’s.
nolist_policy 3y ago
At the very least Chrome needs to link to the system libGL.so and friends for gpu acceleration, libva.so for video acceleration, and so on. And these are linked against glibc of course.
- dooglius 3y ago
  having/omitting frame pointers doesn't change the ABI; it will work if you compile against glibc-nofp and link against glibc-withfp

dsign 3y ago

I remember when the omission of stack frame pointers started spreading at the beginning of the 2000s. I was in college at the time, studying computer sciences in a very poor third-world country. Our computers were old and far from powerful. So, for most course projects, we would eschew interprets and use compilers. Mind you, what my college lacked in money it compensated by having interesting course work. We studied and implemented low level data-structures, compilers, assembly-code numerical routines and even a device driver for Minix. During my first two years in college, if one of our programs did something funny, I would attach gdb and see what was happening at assembly level. I got used to "walking the stack" manually, though the debugger often helped a lot. Happy times, until all of the sudden, "-fomit-frame-pointer" was all the rage, and stack traces stopped making sense. Just like that, debugging that segfault or illegal instruction became exponentially harder. A short time later, I started using Python for almost everything to avoid broken debugging sessions. So, I lost an order of magnitude or two with "-fomit-frame-pointer". But learning Python served me well for other adventures.

nextaccountic 3y ago
Didn't you know about -fno-omit-frame-pointer?
- e40 3y ago
  Perhaps they were talking about libraries they didn’t compile, and with languages with callbacks this can be a real problem.
- noobermin 3y ago
  You unfortunately do not control everything on your machine and how it's compiled. I think source vased distros like gentoo are the only exception but have their own warts.
POVKING_89 3y ago
[flagged]
- 3y ago

rwmj 3y ago

I'm glad he mentioned Fedora because it's been a tiresome battle to keep frame pointers enabled in the whole distribution (eg https://pagure.io/fesco/issue/3084 https://pagure.io/fesco/issue/3084). There's a persistent myth that frame pointers have a huge overhead, because there was a single Python case that had a +10% slow down (now fixed). The actual measured overhead is under 1%, which is far outweighed by the benefits we've been able to make in certain applications.

sidewndr46 3y ago
I'm still baffled by this attitude. That "under 1%" overhead is why computers are measurably slower than 30 years ago to use. All those "under 1%" overhead add up
brendangregg 3y ago
Thanks; what was the Python fix?
- rwmj 3y ago
  This was the investigation: https://discuss.python.org/t/python-3-11-performance-with-frame-pointers/22507 https://discuss.python.org/t/python-3-11-performance-with-fr... Initially we just turned off frame pointers for the Python 3.9 interpreter in Fedora. They are back on in Python 3.12 where it seems the upstream bug has been fixed, although I can't find the actual fix right now. Fedora tracking bug: https://bugzilla.redhat.com/2158729 https://bugzilla.redhat.com/2158729 Fedora change in Python 3.9 to disable frame pointers: https://src.fedoraproject.org/rpms/python3.9/c/9b71f8369141c1840c8d87db1655fcefc576ddff?branch=rawhide https://src.fedoraproject.org/rpms/python3.9/c/9b71f8369141c...
  - brendangregg 3y ago
    Ah right, thanks, I remember I saw Andrii's analysis in the other thread. https://pagure.io/fesco/issue/2817#comment-826636 https://pagure.io/fesco/issue/2817#comment-826636
3y ago

zzbn00 3y ago

NiX (and I assume Guix) are very convenient for this as it is fairly easy to turn frame pointers on or off for parts or whole of the system.

tkiolp4 3y ago

Are his books (the one about Systems Performance and eBPF) relevant for normal software engineers who want to improve performance in normal services? I don’t work for faang, and our usual performance issues are solved by adding indexes here and there, caching, and simple code analysis. Tools like Datadog help a lot already.

wavemode 3y ago
Diving into flame graphs being worthwhile for optimization, assumes that your workload is CPU-bound. Most business software does not have such workloads, and rather (as you yourself have noted) spend most of their time waiting for I/O (database, network, filesystem, etc). And so, (as you again have noted), your best bet is to just use plain old logging and tracing (like what datadog provides) to find out where the waiting is happening.
polio 3y ago
Profiling is a pretty basic technique that is applicable to all software engineering. I'm not sure what a "normal" service is here, but I think we all have an obligation to understand what's happening in the systems we own. Some people may believe that 100ms latency is acceptable for a CLI tool, but what if it could be 3ms? On some aesthetic level, it also feels good to be able to eliminate excess. Finally, you should learn it because you won't necessarily have that job forever.

mgaunard 3y ago

You don't need frame pointers, all the relevant info is stored in dwarf debug data.

DaveFlater 3y ago

GCC optimization causes the frame pointer push to move around, resulting in wrong call stacks. "Wontfix" https://news.ycombinator.com/item?id=38896343 https://news.ycombinator.com/item?id=38896343

rwmj 3y ago
That was in 2012. Does it still occur on modern GCC? There definitely have been regressions with frame pointers being enabled, although we've fixed all the ones we've found in current (2024) Fedora.
- ndesaulniers 3y ago
  Pretty sure unwinding thumb generated by GCC is still non-unwindable via FPs. That's been a pain.
- jart 3y ago
  I think so and I vaguely seem to recall -fno-schedule-insns2 being the only thing that fixes it. To get the full power of frame pointers and hackable binary, what I use is: -fno-schedule-insns2 -fno-omit-frame-pointers -fno-optimize-sibling-calls -mno-omit-leaf-frame-pointer -fpatchable-function-entry=18,16 -fno-inline-functions-called-once The only flag that's potentially problematic is -fno-optimize-sibling-calls since it breaks the optimal approach to writing interpreters and slows down code that's written in a more mathematical style.

tzot 3y ago

I am not sure, but I believe -fomit-frame-pointer in x86-64 allows the compiler to use a _thirteenth_ register, not a _seventeenth_ .

cesarb 3y ago

I disagree with this sentence of the article: "I could say that times have changed and now the original 2004 reasons for omitting frame pointers are no longer valid in 2024." The original 2004 reason for omitting frame pointers is still valid in 2024: it's still a big performance win on the register-starved 32-bit x86 architecture. What has changed is that the 32-bit x86 architecture is much less relevant nowadays (other than legacy software, for most people it's only used for a small instant while starting up the firmware), and other common 32-bit architectures (like embedded 32-bit ARM) are not as register-starved as the 32-bit x86.

IshKebab 3y ago
That's exactly what they were saying. You're not disagreeing at all.

deleted 3y ago

[deleted]

shaggie76 3y ago

I thought we'd been using /Oy (Frame-Pointer Omission) for years on Windows and that there was a pdata section on x64 that was used for stack-walking however to my great surprise I just read on MSDN that "In x64 compilers, /Oy and /Oy- are not available." Does this mean Microsoft decided they weren't going to support breaking profilers and debuggers OR is there some magic in the pdata section that makes it work even if you omit the frame-pointer?

Iwan-Zotow 3y ago
MS had unwinding support done right for a LONG time, 32 and 64, starting with structured C exceptions
MarkSweep 3y ago
Some Google found this: https://devblogs.microsoft.com/oldnewthing/20130906-00/?p=3303 https://devblogs.microsoft.com/oldnewthing/20130906-00/?p=33... “Recovering a broken stack on x64 machines on Windows is trickier because the x64 uses unwind codes for stack walking rather than a frame pointer chain.” More details are here: https://learn.microsoft.com/en-us/cpp/build/exception-handling-x64?view=msvc-170 https://learn.microsoft.com/en-us/cpp/build/exception-handli...
quotemstr 3y ago
Microsoft has had excellent universal unwinding support for decades now. I'm disappointed to see someone as prominent as this article's author present as infeasible what Microsoft has had working for so long.
- Iwan-Zotow 3y ago
  exactly! starting with C structured exceptions
musjleman 3y ago
> In x64 compilers The default is omission. If you have a Windows machine, in all likelihood almost no 64 bit code running on it has frame pointers. > OR is there some magic in the pdata section that makes it work even if you omit the frame-pointer You haven't ever needed frame pointers to unwind using ... unwind information. The same thing exists for linux as `.eh_frame` section.

javierhonduco 3y ago

Overall, I am for frame pointers, but after some years working in this space, I thought I would share some thoughts: * Many frame pointer unwinders don't account for a problem they have that DWARF unwind info doesn't have: the fact that the frame set-up is not atomic, it's done in two instructions, `push $rbp` and `mov $rsp $rbp`, and if when a snapshot is taken we are in the `push`, we'll miss the parent frame. I think this might be able to be fired by inspecting the code, but I think this might only be as good as a heuristic as there could be other `push %rbp` unrelated to the stack frame. I would love to hear if there's a better approach! * I developed the solution Brendan mentions which allows faster, in-kernel unwinding without frame pointers using BPF [0]. This doesn't use DWARF CFI (the unwind info) as-is but converts it into a random-access format that we can use in BPF. He mentions not supporting JVM languages, and while it's true that right now it only supports JIT sections that have frame pointers, I planned to implement a full JVM interpreter unwinder. I have left Polar Signals since and shifted priorities but it's feasible to get a JVM unwinder to work in lockstep with the native unwinder. * In an ideal world, enabling frame pointers should be done on a case-by-case. Benchmarking is key, and the tradeoffs that you make might change a lot depending on the industry you are in, and what your software is doing. In the past I have seen large projects enabling/disabling frame pointers not doing an in-depth assessment of losses/gains of performance, observability, and how they connect to business metrics. The Fedora folks have done a superb and rigorous job here. * Related to the previous point, having a build system that enables you to change this system-wide, including libraries your software depends on can be awesome to not only test these changes but also put them in production. * Lastly, I am quite excited about SFrame that Indu is working on. It's going to solve a lot of the problems we are facing right now while letting users decide whether they use frame pointers. I can't wait for it, but I am afraid it might take several years until all the infrastructure is in place and everybody upgrades to it. - [0]: https://web.archive.org/web/20231222054207/https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-based-stack-walking-using-ebpf https://web.archive.org/web/20231222054207/https://www.polar...

spc476 3y ago
It could be one instruction: ENTER N,0 (where N is the amount of stack space to reserve for locals)---this is the same as: PUSH EBP MOV ESP,ESP SUB SP,N (I don't recall if ENTER is x86-64 or not). But even with this, the frame setup isn't atomic with respect to CALL, and if the snapshot is taken after the CALL but before the ENTER, we still don't get the fame setup. As for the reason why ENTER isn't used, it was deemed too slow. LEAVE (MOV SP,BP; POP BP) is used as it's just as fast as, if not faster, than the sequence it replaces. If ENTER were just the PUSH/MOV/SUB sequence, it probably would be used, but it's that other operand (which is 0 above in my example) that kills it performance wise (it's for nested functions to gain access to upper stack frames and is every expensive to use).

secondcoming 3y ago

Can they not be disabled on a per-function basis?

weebull 3y ago

Just as a general comment on this topic... The fact that people complain about the performance of the mechanism that enables the system to be profiled, and so performance problems be identified, is beyond ironic. Surely the epitome of premature optimisation.

account42 3y ago
It's not moronic when the people profiling and the people having performance issues are different groups. That some developer benefits from having frame pointers in libc does not mean that all users of that software also need to have frame pointers enabled.
im3w1l 3y ago
It goes both ways. I see people with ultra-bloated applications trying to add even more bloat and force it on the rest of us too. Like the guy saying that DWARF unwinding is impractical when you have 1Gbyte code cache. I don't see the people writing hyperfast low-level code asking for this. My experience with profiling is anyway that you don't need that fine-grained profiling. It's main use is finding stuff like "we spend 90% of our time reallocating a string over and over to add characters one at a time". After a few of those it's just "it's a little bit slow everywhere".
- Sesse__ 3y ago
  > After a few of those it's just "it's a little bit slow everywhere". And after that, you need fine-grained profiling to find multiple 1% wins and apply them repeatedly. (I do this for a living, basically)
  - im3w1l 3y ago
    Like disabling frame pointers
doubloon 3y ago
im sure in ancient mesopotamia there was somebody arguing about you could brew beer faster if you stop measuring the hops so carefully but then someone else was saying yes but if you dont measure the hops carefully then you dont know the efficiency of your overall beer making process so you cant isolate the bottlenecks. the funny thing is i am not sure if the world would actually work properly if we didn't have both of these kinds of people.

Cold_Miserable 3y ago

Not interesting. Enter/leave also does the same thing as your save/restore rbp. Far more interesting I recall there might be an instruction where rbp isn't allowed.

titzer 3y ago

Virgil doesn't use frame pointers. If you don't have dynamic stack allocation, the frame of a given function has a fixed size can be found with a simple (binary-search) table lookup. Virgil's technique uses an additional page-indexed range that further restricts the lookup to be a few comparisons on average (O(log(# retpoints per page)). It combines the unwind info with stackmaps for GC. It takes very little space. The main driver is in (https://github.com/titzer/virgil/blob/master/rt/native/NativeStackWalker.v3); https://github.com/titzer/virgil/blob/master/rt/native/Nativ... the rest of the code in the directory implements the decoding of metadata. I think frame pointers only make sense if frames are dynamically-sized (i.e. have stack allocation of data). Otherwise it seems weird to me that a dynamic mechanism is used when a static mechanism would suffice; mostly because no one agreed on an ABI for the metadata encoding, or an unwind routine. I believe the 1-2% measurement number. That's in the same ballpark as pervasive checks for array bounds checks. It's weird that the odd debugging and profiling task gets special pleading for a 1% cost but adding a layer of security gets the finger. Very bizarre priorities.

moonchild 3y ago
You can add bounds checks to c, but that costs a hell of a lot more than 1-2%. C++ has them off by default for std::vector because c++ is designed by and for the utterly insane. Other than that, I can't off the top of my head think of a language that doesn't have them.
- shrimp_emoji 3y ago
  > C++ has them off by default for std::vector because c++ is designed by and for the utterly insane. And for those who value performance and don't want to pay the cost of "a lot more than 1-2%" ;p
  - moonchild 3y ago
    std::vector falls into the category of things that are easy to bounds check, st the cost, even under today's primitive compilers, is low. It's direct pointer accesses—which are common in c but not in c++ or most other languages—that are hard to and therefore cost more to bounds check.

codeflo 3y ago

All of this information is static, there's no need to sacrifice a whole CPU register only to store data that's already known. A simple lookup data structure that maps an instruction address range to the stack offset of the return address should be enough to recover the stack layout. On Windows, you'd precompute that from PDB files, I'm sure you can do the same thing with whatever the equivalent debug data structure is on Linux.

fsmv 3y ago
[deleted]
- deleted 3y ago
  [deleted]
loeg 3y ago
It isn't entirely static because of alloca().
- account42 3y ago
  Banning alloca() would make more sense than re-enabling frame pointers.
  - loeg 3y ago
    At least re-enabling FPs is possible.

mikewarot 3y ago

I started programming in 1979, and I can't believe I've managed to avoid learning about stack frames all those EBP register tricks until now. I always had parameters to functions in registers, not on the stack, for the most part. The compiler hid a lot of things from me. Is it because I avoided Linux and C most of my life? Perhaps it's because I used debug, and Periscope before that... and never gdb?

boulos 3y ago

JIT'ed code is sadly poorly supported, but LLVM has had great hooks for noting each method that is produced and its address. So you can build a simple mixed-mode unwinder, pretty easily, but mostly in process. I think Intel's DNN things dump their info out to some common file that perf can read instead, but because the *kernels* themselves reuse rbp throughout oneDNN, it's totally useless. Finally, can any JVM folks explain this claim about DWARF info from the article: > Doesn't exist for JIT'd runtimes like the Java JVM that just sounds surprising to me. Is it off by default or literally not available? (Google searches have mostly pointed to people wanting to include the JNI/C side of a JVM stack, like https://github.com/async-profiler/async-profiler/issues/215 https://github.com/async-profiler/async-profiler/issues/215).

POVKING_89 3y ago

[flagged]

POVKING_89 3y ago

[flagged]

olliej 3y ago

Have compilers (or I guess x86?) gotten better at dealing with the frame pointer? Or are we just saying that taking a significant perf hit is acceptable if it lets you find other tiny perf problems? Because I recall -fomit-frame-pointer being a significant performance win, bigger than most of the things that you need a perfect profiler to spot.

vlovich123 3y ago

To this day I still believe that there should be a dedicated protected separate stack region for the call stack that only the CPU can write to/read from. Walking the stack then becomes trivially fast because you just need to do a very small memcpy. And stack memory overflows can never overwrite the return address.

ndesaulniers 3y ago
This is a thing; it's called shadow call stack. Both ARM and now Intel have extensions for it.
- vlovich123 3y ago
  But the shadow stack concept seems much dumber to me. Why write the address to the regular stack and the shadow stack and then compare? Why not only use the shadow stack and not put return addresses on the main stack at all.
  - BeeOnRope 3y ago
    ABI compatibility.
    - vlovich123 3y ago
      When you're compiling from scratch, especially when you're not working on a shared library, ABI compatability doesn't matter as much. Doesn't explain why there's no -fshadow-stack-only option to pass in.
      - BeeOnRope 3y ago
        > Doesn't explain why there's no -fshadow-stack-only option to pass in. I thought you were asking about the design of the hardware: it's designed that way because compatibility means that the vast majority of people want something backwards compatible. Very little is fully "compiled from scratch" when you consider for example that libc is in the set of things that must be recompiled.

BinaryRage 3y ago

I remember talking to Brendan about the PreserveFramePointer patch during my first months at Netflix in 2015. As of JDK 21, unfortunately it is no longer a general purpose solution for the JVM, because it prevents a fast path being taken for stack thawing for virtual threads: https://github.com/openjdk/jdk/blob/d32ce65781c1d7815a69ceac720cdf3ae39caa9e/src/hotspot/share/runtime/continuationFreezeThaw.cpp#L1807 https://github.com/openjdk/jdk/blob/d32ce65781c1d7815a69ceac...