12 ms·

The Sad State of Debug Performance in C++

146 points by SuperV1234 4y ago

chubot 4y ago
It's also annoying for tools -- like debuggers, and uftrace (user space function tracing), which will start showing all these tiny functions that you don't really care about I prefer to avoid all the operator overloading and smart pointers, etc.
glandium 4y ago
> Even if -Og was ubiquitous, it is still suboptimal to -O0: it can still inline code a bit too aggressively for an effective debugging session. IMO, the problem is not so much the inlining, but the sad state of the optimization passes losing track of many things and producing useless debug info.
- DannyBee 4y ago
  Eons ago, i wrote a lot of the first version of debug location tracking/location list tracking in GCC, and the support for evaluating them in GDB (plenty of others came along and improved it nearly instantly - it thankfully did not last long alone ;P). I have spent, over the years, an amazing amount of time on trying to improve optimized-code debug info. You could and should improve the optimization passes, but at various points in life they were actually very good. It is always a dance between advancing optimization and debug info, and i certainly will not claim that debug info is thought about as much as it should be - it is often fixed later, as i'm sure you expect. But at the same time, what the author amounts to ends up reducing to the following: 1. The output executable is pretty well optimized 2. The output debug info is a recreation of the program as-written that can be executed by a debugger 3. The mapping between #2 and #1 has no holes. IE #2 is basically "you've output the entire program using dwarf5 as a target processor". This is never going to be a thing that regular users strive for, for lots of reasons. So if the game companies or whoever want this to be the case, they will actually have to contribute more than just bug reports - because it's not something that drives compiler usage for any language, whether it's C++ or anything else (IE it's hubris to believe that other languages won't end up the same way as they get better at optimizing. C++ just has a long head start) At the same time I am sure that just one good compiler person paid for a year working on any of the compilers to improve optimized debug info would get to a really good place. I know - i was once paid to do it, and when i stopped, it was, back then, in a pretty darn good place compared to where we started[1]. Rather than throwing away every abstraction, it's hard to believe this entire industry can't afford this. Yet I have not seen them try before (perhaps they did after my time, but my time covered 20 years, and it didn't really happen during then) So it is 100% reasonable to blame game developers or whoever as a group - because as a group, they aren't (again, AFAIK) doing anything about it other than writing blog posts about how the language is mean to them, and filing bugs that compilers are mean to them. [1] I'll note, in practice, the most effective route over time is likely actually to just store enough info to do deoptimize during debugging as-needed, which is the path various VM based languages take and works well. The traditional path to doing this is to JIT the language along the way (which is totally possible for C++), but it's not strictly necessary to do it that way. The harder part is the debugger/compiler cooperation, not the runtime mechanism.
  - khuey 4y ago
    Thanks for doing that work. Even many years later gcc is in a noticeably better place than clang when it comes to how much debug info is preserved by the optimizer. https://robert.ocallahan.org/2018/11/comparing-quality-of-debug-information.html https://robert.ocallahan.org/2018/11/comparing-quality-of-de...
  - dannas 4y ago
    [gcc/gdb/LLVM outsider here] If someone is interested in what's being done to improve LLVMs debuginfo, here's a talk by Djordje Todorovic from two years ago: https://www.youtube.com/watch?v=GpMLt1oecOk https://www.youtube.com/watch?v=GpMLt1oecOk. It's about tracking dwarf locations for variables by keeping around information from the parent frame. According to that presentation, at least that work has been progressing.
    - dannas 4y ago
      As for contributions from the gamedev industry, here's a EuroLLVM 2022 presentation from Orlando Cazalet-Hyams working at Sony: https://www.snsystems.com/technology/tech-blog/improving-debug-locations-for-variables-in-memory https://www.snsystems.com/technology/tech-blog/improving-deb...
  - tialaramex 4y ago
    > they aren't (again, AFAIK) doing anything about it other than writing blog posts about how the language is mean to them, and filing bugs that compilers are mean to them. There are people from that industry or adjacent industries writing other languages. Most obviously Jai (Jonathan Blow's language, Jonathan did "Braid" and "The Witness") but you could also argue for Ginger Bill's Odin. I suppose in this context it will be interesting to see whether the debug story is any better in these languages (neither is yet finished).
  - jmorse2 4y ago
    > they aren't (again, AFAIK) doing anything about it Person slightly doing things to LLVM checking in -- the three points you list are compelling, because it's always going to be difficult to describe one program (source) in terms of a vastly different (optimised) program. Do you think there's mileage in showing the developer a partially optimised program [0] instead of trying to perfectly describe the source? It'd be easier to describe in debug-info, and possibly easier for developers to identify unexpected changes to the code they wrote. [0] "Non-Transparent Debugging of Optimized Code", 1999
- usefulcat 4y ago
  It's not just inlining. With -Og (clang or gcc), the debugger frequently can't even tell me the value of function arguments because they've been "optimized out".
  - adrian_b 4y ago
    While I agree with you and with the OP and I have also been sometimes annoyed by this kind of debugger messages, the more universal solution than wanting an improved debugger is to rely for debugging on some form of progress and error logging in your program, which will work at any optimization level. Adding such debugging aids in your program may need some extra work, but it also functions in all cases when using the debugger is hopeless, e.g. for concurrent programs or other programs that depend on real-time interactions which prevent the use of breakpoints.
    - usefulcat 4y ago
      Yes and no. I use logging too, but that's only useful when I know (or can guess) ahead of time what locations and what things at each location I'll need to know in order to diagnose the problem. Of course I can always go back and add some logging, but it's not a given that I'll want to keep it around after the problem is fixed. For example if it's in a performance-critical section of code, even the cost of checking to see whether logging is enabled may be too great. And in any case every log statement does add visual noise to the code.
  - SuperV1234 4y ago
    Just want to remind you that `-Og` in Clang is not representative here, as it's exactly the same as `-O1` -- they implemented support for the flag just to ease transitions between GCC and Clang without having to filter out flags, but it doesn't provide any debugging-specific advantage.
    - not2b 4y ago
      Yes, those who think -Og is no good because they are clang users should perhaps file bug reports or contribute improvements instead.
alternatetwo 4y ago
MSVC has /Ob1 [1] which only inlines specifically marked functions, or functions defined in a header. Shouldn't this inline all those trivial calls yet keep most debug info if you disable all the other optimizations? I haven't experimented with it yet *or what I rather mean is use Debug builds, but enable /Ob1 in them. [1]: https://learn.microsoft.com/en-us/cpp/build/reference/ob-inline-function-expansion?view=msvc-170 https://learn.microsoft.com/en-us/cpp/build/reference/ob-inl...
BenFrantzDale 4y ago
It’s always nice to understand more about why game developers do the weird things they do. I always buried and run in “RelWithDebInfo”, then if I really need it, I’ll throw `#pragma clang optimize off` around the code that the debugger can’t see into.
- cyber_kinetist 4y ago
  Really nice tip to know! I've also struggled a lot with slow debug performance, so this will be very handy when I know there's a bug somewhere in a specific region but can't pinpoint exactly where.
  - BenFrantzDale 4y ago
    Yeah, I lost lots of time to switching to a debug build and having to rebuild the whole project since my Debug-build ccache wasn’t warm. The `#pragma` version lets me get my work done quickly and easily.
  - jotux 4y ago
    GCC has something similar: >#pragma GCC optimize ("O3") https://gcc.gnu.org/onlinedocs/gcc-7.2.0/gcc/Function-Specific-Option-Pragmas.html#Function-Specific-Option-Pragmas https://gcc.gnu.org/onlinedocs/gcc-7.2.0/gcc/Function-Specif...
halayli 4y ago
The std::move -> call issue has been fixed in clang-15.
- edflsafoiewq 4y ago
  Yes, the article mentions. > Clang 15.x, also motivated by my #53689 issue, also introduced a similar folding pass for the same functions chosen by GCC (plus std::move_if_noexcept, which I assume GCC maintainers forgot about). This one seems to be enabled by default – see a comparison between Clang 14.x and Clang 15.x on Compiler Explorer.
humanrebar 4y ago
Have any multibillion dollar game (etc.) companies invested any resources into debug build modes in the open source compilers they develop with? Seems like it should be trivial to tie team productivity rates to quality of the debug experience.
- fartsucker69 4y ago
  That proposition seems risky. If the compiler owners don't accept your pull requests, you risk having to maintain a fork with your changes indefinitely. Also, I suspect like 95%+ of development at game dev studios happens in MSVC. Nothing to change there. Unfortunately pestering the compiler devs about this seems like the optimal approach lol
  - deleted 4y ago
    [deleted]
  - humanrebar 4y ago
    Maybe? If each AAA studio was putting 6-7 figures into the relevant projects, their change requests should be especially well explained and well formed. A lot of ignored PRs aren't well communicated. Also, it's kind of moot. For platforms that use GCC, Clang, etc., the maintainers are OSS communities, so that's how you influence the product in a healthy direction for your projects. To sit back and let things happen is also an option, but then debugging support might not meet your needs, for instance. Seems like mobile and wasm based platforms exist and don't generally use MSVC. Folks around here might be surprised how effective $1MM/yr in engineer salaries can be in the relatively under-resourced OSS compiler development space.
- Decabytes 4y ago
  Not exactly what you were asking, but RAD tried to build a debugger for Linux^1, but the project died. I’ve heard the people at RAD described as hard core old school programmers, so if they struggled with this it must just be a hard problem 1. http://www.radgametools.com/debug.htm http://www.radgametools.com/debug.htm
323 4y ago
I worked on a CAD/CAM product, debug mode was literally unusable, it was 100-1000 times slower than optimized mode.
bluGill 4y ago
I'm with Kernigham and Pike: the debugger is for getting a stack trace out of core dumps. Printf is for debugging.
- jjtheblunt 4y ago
  oh where is that quote? that's great!
  - bluGill 4y ago
    In the 'practice of programming'. https://www.cs.princeton.edu/~bwk/tpop.webpage/ https://www.cs.princeton.edu/~bwk/tpop.webpage/ I last read it some years ago so I can't tell you exactly what they said. I do know that when pointed out I realized I suffer from the same sins they hate about debuggers: I would step through code mindlessly, changing variables and the like, but never stopping to think. While printf is more painful it forces me to think why is the code in that state, which is key to fixing bugs.
    - jjtheblunt 4y ago
      i totally agree
- properparity 4y ago
  Why limit yourself to crusty text when you can have live visualizations of data structures in various formats, graph dx/dy over time of some variable, stop and inspect data on some condition without having to modify code, quickly start execution on a certain line, quickly drill down into struct members, etc. I mean it does everything a printf() can do - except just better.
  - Night_Thastus 4y ago
    For a couple months I had no functioning debugger and was left with only Printf's. I do not ever desire to go back to that. It is far too painful. I think my time to find issues likely was 4x+ longer than before. Some programs simply have too much state, too complex of structures (for performance or memory reasons), or very long runtimes to reasonably debug purely with prints.
Deganta 4y ago
The problem is that C++ is full of so called "zero-cost-abstractions" that are only zero cost if you ignore: 1. Compile time 2. Non-optimized builds Stuffing everything into the Standard Library using lots of template magic does has its upside, but debug performance is one of the big downsides. Another one are really clunky interfaces, like std::variant. This should really have been a language feature, not a library features.
- titzer 4y ago
  The phrase "zero-cost abstraction" is a lie anyway. It's all relative to how much compiler optimization happens. If you actually executed the C++ specification step by step, the code would be dog slow, because there are so many implicit conversions, constructions, inlining, constant folding, etc. We could quibble about how much static resolution qualifies (template expansion, overload resolution, etc--that's executing an inflated in-memory IR, and not the original source code.) Not to mention that all the rest of the compiler backend is working hard to get good machine code, and cannot possibly produce "optimal" machine code, so there is an unmeasurable cost just because of the limitations of any one particular compiler backend. (And those things matter, too!)
  - olliej 4y ago
    If you implemented any language following the spec in a step by step manner they would be dog slow, because any usable spec has to be very explicit in every step required for every operation. All those conversions that you believe only happen if you implement the spec “step by step” always happen. Actual implementations of languages then implement the explicitly described semantics, in whatever way they feel is most efficient.
    - titzer 4y ago
      Bah. Compiling Java to bytecode consists of a lot of static resolution steps and then a fairly straightforward, simple translation. Single-stepping the bytecodes will absolutely produce the same results as the optimized JIT code and there are few expressions that generate more than 1 to a handful of bytecodes.
StellarScience 4y ago
MSVC lacks -Og because it instead treats inlining, optimization, and inclusion of debug information as separate, orthogonal choices. These days I rarely build a whole MSVC project as "Debug" without optimization. Instead I enable both optimization and debug information, so I always get call stacks and line-by-line stepping ability. When I end up debugging some file where the optimizations inhibit debuggability, I'll recompile just that compilation unit or library without optimization or inlining. That said, I agree with the gist of the article that both "zero cost abstraction" and debuggability need to be constantly improved.
- plorkyeran 4y ago
  I think you misunderstand what -Og is. All of the major compilers treat optimization and inclusion of debug information as separate choices, and generating debug symbols for optimized release builds is a very normal thing to do. GCC's -Og is just an attempt at selecting a set of optimizations which don't cause too many problems for debuggability.
  - mort96 4y ago
    It's worth noting that it does a pretty bad job at it. I've had to go back and remove -Og any time I've used it because it just makes so many variables "optimised out".
_gabe_ 4y ago
I ran into these problems with std::unordered_map. While developing my game, I would always run it in debug mode because I need to debug it. Loading a game world was taking around 12 seconds, and it was beginning to grate on me because of the increased iteration times. So I decided to profile it and see what was taking so long. I compiled in release mode to begin debugging, and it magically took the world load time down to 2 seconds. I wondered what code I had written that caused such a massive performance hit in debug mode. So I went to profile my code in a debug build to find out. Lo and behold, something like 90% of the CPU time was wasted doing lookups from std::unordered_map because of debug iterators. I tried everything I could to just turn debug iterators off, and eventually gave up and switched to robin_hood::unordered_map which runs great in debug and release. Now I have an unwarranted aversion to the std lib, even though it really is great so long as you're running it in release mode.
- foonathan 4y ago
  To be fair, std::unordered_map is also very slow in release mode.
- leni536 4y ago
  > debug iterators MSVC then, I presume.
- gernb 4y ago
  It'd maybe be nice to be able to mark which code gets optimized. If I could make all std as "don't debug this and keep it optimized" I would. I know some debuggers have the option to at least not step into certain code that that doesn't solve the optimization issue
  - TillE 4y ago
    As others have mentioned, you can accomplish the equivalent by using a release build with pragmas to turn off optimizations in certain sections of code.
  - gumby 4y ago
    You don't have to compile every file with the same flags. You can vary them with each invocation of the compiler, so trivially get what you want at the translation unit level.
staticassertion 4y ago
With our rust project we actually compile with `01` because the cost of that optimization pass is less than the cost to our tests running 100x slower. There is definitely a need for a 'more optimal' debug mode, and it is one of those things that feels zero cost until it isn't. Similarly, I try to at least consider compile times when writing code these days.
PaulDavisThe1st 4y ago
I've been using C++ since 1993. I still don't understand how I've written all this code and never once found myself using, or wishing I could use std::move. Presumably my code has been less than optimally efficient or something, because it seems that most people talking about "modern" C++ view it as absolutely central to the language.
- the-lazy-guy 4y ago
  std::move is not only about performance. It is about being able to disallow destructive copies of certain types. I.e. unique_ptr can not be copied, and auto_ptr can. Can you write perfectly functioning code without these features? Sure! You just have to be accurate. Same with performance. Where nowadays simple std::move() on a return value suffices you had previously explicitly to pass the object/vector with which you swapped as a parameter. Life with move semantics is a tad easier.
  - MauranKilom 4y ago
    Note that you don't even have to std::move a return value in almost all cases. (The very much non-language-lawyer explanation is that local variables are automatically moved from when you return them.)
    - dryanau 4y ago
      Also, if you write > return std::move(ret); it can interfere with return value optimisation (RVO). There might be cases where the move makes sense but to my understanding it's enough to just use return. It's very much possible to take advantage of modern C++ move semantics without an explicit call to move. Maybe you return a unique_ptr created by make_unique in a factory method, or call swap() on a moveable object, etc. I rarely need to write std::move to take advantage of modern C++ although it took some self-education about rvalues etc to learn how to do it.
    - olliej 4y ago
      As u/dryanau says returning std::move breaks the return value optimization such that clang at least will warn about it by default.
not2b 4y ago
The author needs to learn about the -Og switch, which can be used together with debugging. It can be thought of as "do those optimizations that don't interfere with debugging". I verified with Godbolt that this suffices to eliminate the call overhead.
- SuperV1234 4y ago
  Did you even read the article? I mentioned '-Og' several times and its shortcomings.
  - not2b 4y ago
    I read it, but too quickly. Sorry. As you say, the main problem is that only gcc has a usable implementation; the other issues, for me, are an acceptable tradeoff (reasonable speed, but sometimes difficulty because inlining hid something).
olliej 4y ago
The urge to have basic language features not be part of the language, but instead be part of the standard library is responsible for a lot of the less than stellar parts of of C++. Std::move/forward are particularly obvious cases, but you get similar accessing members of tuples, etc. Wanting to keep things in the standard library rather than the language itself means that you have to compile large amounts of template hell for a wide array of basic things which hurts build time even in debug modes, and then as this article says you end up with no-op operations that become calls. You also can’t easily debug any of this because you end up with absurd layers of template nonsense for what is again basic functionality. You can compile with -O1, but that then inlines things that aren’t part of the standard library that makes debugging of the actually relevant code annoying (this is what the debug llvm and clang debug builds do), as it inlines your own code and also means that you lose variables all over the place.
andrewmcwatters 4y ago
You know, at least in C++ you get some really poorly performing standard containers. Every time I reach for C, it drives me nuts that the best you get is GLib. And BOTH pale in comparison to newer language standard libraries like Go's.
phaedrus 4y ago
This dovetails with something I was thinking about recently in the context of Reproducible Builds (or the lack thereof). For most of its history Computer Science (at least as practiced by industry) has gone all-in on "functionally equivalent" transformations, and done almost nothing in the space of "output invariant" guarantees. In my day job we sometimes we need to reproduce the build of a firmware ROM or executable, sometimes decades after the engineer who last built it left. Getting a match is easier (or even possible) for older tech only because of the relative unsophistication of the compilers used - even then it's only reproducible "by accident," and not because the compiler vendor made any guarantees about X language construct reliably produces Y machine code and data layout. But we need that! For getting accurate baselines. For security verification. And there's no reason in principle we should have to forego updating compilers, IDEs, and OS environments in the indirect hope of not disturbing anything. Those are two separate things: if we had a through-line of higher level language construct -> semantically defined transformation (irregardless of optimization settings) -> machine code, vendors could continue to update their IDEs and compilers while just making sure they still respect the invariants. C++'s so-called zero-cost abstractions are poor substitute for this: header (library) writers and C++ gurus write code as if they worked like this "guaranteed output transform" I describe, but no compiler actually has to respect it (nevermind that the fine details of what the transformation actually is isn't nailed down) and it differs between Debug and Release build which is particularly bad for game development as TFA makes clear.

146 points by SuperV1234 4y ago

It's also annoying for tools -- like debuggers, and uftrace (user space function tracing), which will start showing all these tiny functions that you don't really care about I prefer to avoid all the operator overloading and smart pointers, etc.

glandium 4y ago

> Even if -Og was ubiquitous, it is still suboptimal to -O0: it can still inline code a bit too aggressively for an effective debugging session. IMO, the problem is not so much the inlining, but the sad state of the optimization passes losing track of many things and producing useless debug info.

DannyBee 4y ago
Eons ago, i wrote a lot of the first version of debug location tracking/location list tracking in GCC, and the support for evaluating them in GDB (plenty of others came along and improved it nearly instantly - it thankfully did not last long alone ;P). I have spent, over the years, an amazing amount of time on trying to improve optimized-code debug info. You could and should improve the optimization passes, but at various points in life they were actually very good. It is always a dance between advancing optimization and debug info, and i certainly will not claim that debug info is thought about as much as it should be - it is often fixed later, as i'm sure you expect. But at the same time, what the author amounts to ends up reducing to the following: 1. The output executable is pretty well optimized 2. The output debug info is a recreation of the program as-written that can be executed by a debugger 3. The mapping between #2 and #1 has no holes. IE #2 is basically "you've output the entire program using dwarf5 as a target processor". This is never going to be a thing that regular users strive for, for lots of reasons. So if the game companies or whoever want this to be the case, they will actually have to contribute more than just bug reports - because it's not something that drives compiler usage for any language, whether it's C++ or anything else (IE it's hubris to believe that other languages won't end up the same way as they get better at optimizing. C++ just has a long head start) At the same time I am sure that just one good compiler person paid for a year working on any of the compilers to improve optimized debug info would get to a really good place. I know - i was once paid to do it, and when i stopped, it was, back then, in a pretty darn good place compared to where we started[1]. Rather than throwing away every abstraction, it's hard to believe this entire industry can't afford this. Yet I have not seen them try before (perhaps they did after my time, but my time covered 20 years, and it didn't really happen during then) So it is 100% reasonable to blame game developers or whoever as a group - because as a group, they aren't (again, AFAIK) doing anything about it other than writing blog posts about how the language is mean to them, and filing bugs that compilers are mean to them. [1] I'll note, in practice, the most effective route over time is likely actually to just store enough info to do deoptimize during debugging as-needed, which is the path various VM based languages take and works well. The traditional path to doing this is to JIT the language along the way (which is totally possible for C++), but it's not strictly necessary to do it that way. The harder part is the debugger/compiler cooperation, not the runtime mechanism.
- khuey 4y ago
  Thanks for doing that work. Even many years later gcc is in a noticeably better place than clang when it comes to how much debug info is preserved by the optimizer. https://robert.ocallahan.org/2018/11/comparing-quality-of-debug-information.html https://robert.ocallahan.org/2018/11/comparing-quality-of-de...
- dannas 4y ago
  [gcc/gdb/LLVM outsider here] If someone is interested in what's being done to improve LLVMs debuginfo, here's a talk by Djordje Todorovic from two years ago: https://www.youtube.com/watch?v=GpMLt1oecOk https://www.youtube.com/watch?v=GpMLt1oecOk. It's about tracking dwarf locations for variables by keeping around information from the parent frame. According to that presentation, at least that work has been progressing.
  - dannas 4y ago
    As for contributions from the gamedev industry, here's a EuroLLVM 2022 presentation from Orlando Cazalet-Hyams working at Sony: https://www.snsystems.com/technology/tech-blog/improving-debug-locations-for-variables-in-memory https://www.snsystems.com/technology/tech-blog/improving-deb...
- tialaramex 4y ago
  > they aren't (again, AFAIK) doing anything about it other than writing blog posts about how the language is mean to them, and filing bugs that compilers are mean to them. There are people from that industry or adjacent industries writing other languages. Most obviously Jai (Jonathan Blow's language, Jonathan did "Braid" and "The Witness") but you could also argue for Ginger Bill's Odin. I suppose in this context it will be interesting to see whether the debug story is any better in these languages (neither is yet finished).
- jmorse2 4y ago
  > they aren't (again, AFAIK) doing anything about it Person slightly doing things to LLVM checking in -- the three points you list are compelling, because it's always going to be difficult to describe one program (source) in terms of a vastly different (optimised) program. Do you think there's mileage in showing the developer a partially optimised program [0] instead of trying to perfectly describe the source? It'd be easier to describe in debug-info, and possibly easier for developers to identify unexpected changes to the code they wrote. [0] "Non-Transparent Debugging of Optimized Code", 1999
usefulcat 4y ago
It's not just inlining. With -Og (clang or gcc), the debugger frequently can't even tell me the value of function arguments because they've been "optimized out".
- adrian_b 4y ago
  While I agree with you and with the OP and I have also been sometimes annoyed by this kind of debugger messages, the more universal solution than wanting an improved debugger is to rely for debugging on some form of progress and error logging in your program, which will work at any optimization level. Adding such debugging aids in your program may need some extra work, but it also functions in all cases when using the debugger is hopeless, e.g. for concurrent programs or other programs that depend on real-time interactions which prevent the use of breakpoints.
  - usefulcat 4y ago
    Yes and no. I use logging too, but that's only useful when I know (or can guess) ahead of time what locations and what things at each location I'll need to know in order to diagnose the problem. Of course I can always go back and add some logging, but it's not a given that I'll want to keep it around after the problem is fixed. For example if it's in a performance-critical section of code, even the cost of checking to see whether logging is enabled may be too great. And in any case every log statement does add visual noise to the code.
- SuperV1234 4y ago
  Just want to remind you that `-Og` in Clang is not representative here, as it's exactly the same as `-O1` -- they implemented support for the flag just to ease transitions between GCC and Clang without having to filter out flags, but it doesn't provide any debugging-specific advantage.
  - not2b 4y ago
    Yes, those who think -Og is no good because they are clang users should perhaps file bug reports or contribute improvements instead.

alternatetwo 4y ago

MSVC has /Ob1 [1] which only inlines specifically marked functions, or functions defined in a header. Shouldn't this inline all those trivial calls yet keep most debug info if you disable all the other optimizations? I haven't experimented with it yet *or what I rather mean is use Debug builds, but enable /Ob1 in them. [1]: https://learn.microsoft.com/en-us/cpp/build/reference/ob-inline-function-expansion?view=msvc-170 https://learn.microsoft.com/en-us/cpp/build/reference/ob-inl...

BenFrantzDale 4y ago

It’s always nice to understand more about why game developers do the weird things they do. I always buried and run in “RelWithDebInfo”, then if I really need it, I’ll throw `#pragma clang optimize off` around the code that the debugger can’t see into.

cyber_kinetist 4y ago
Really nice tip to know! I've also struggled a lot with slow debug performance, so this will be very handy when I know there's a bug somewhere in a specific region but can't pinpoint exactly where.
- BenFrantzDale 4y ago
  Yeah, I lost lots of time to switching to a debug build and having to rebuild the whole project since my Debug-build ccache wasn’t warm. The `#pragma` version lets me get my work done quickly and easily.
- jotux 4y ago
  GCC has something similar: >#pragma GCC optimize ("O3") https://gcc.gnu.org/onlinedocs/gcc-7.2.0/gcc/Function-Specific-Option-Pragmas.html#Function-Specific-Option-Pragmas https://gcc.gnu.org/onlinedocs/gcc-7.2.0/gcc/Function-Specif...

halayli 4y ago

The std::move -> call issue has been fixed in clang-15.

edflsafoiewq 4y ago
Yes, the article mentions. > Clang 15.x, also motivated by my #53689 issue, also introduced a similar folding pass for the same functions chosen by GCC (plus std::move_if_noexcept, which I assume GCC maintainers forgot about). This one seems to be enabled by default – see a comparison between Clang 14.x and Clang 15.x on Compiler Explorer.

humanrebar 4y ago

Have any multibillion dollar game (etc.) companies invested any resources into debug build modes in the open source compilers they develop with? Seems like it should be trivial to tie team productivity rates to quality of the debug experience.

fartsucker69 4y ago
That proposition seems risky. If the compiler owners don't accept your pull requests, you risk having to maintain a fork with your changes indefinitely. Also, I suspect like 95%+ of development at game dev studios happens in MSVC. Nothing to change there. Unfortunately pestering the compiler devs about this seems like the optimal approach lol
- deleted 4y ago
  [deleted]
- humanrebar 4y ago
  Maybe? If each AAA studio was putting 6-7 figures into the relevant projects, their change requests should be especially well explained and well formed. A lot of ignored PRs aren't well communicated. Also, it's kind of moot. For platforms that use GCC, Clang, etc., the maintainers are OSS communities, so that's how you influence the product in a healthy direction for your projects. To sit back and let things happen is also an option, but then debugging support might not meet your needs, for instance. Seems like mobile and wasm based platforms exist and don't generally use MSVC. Folks around here might be surprised how effective $1MM/yr in engineer salaries can be in the relatively under-resourced OSS compiler development space.
Decabytes 4y ago
Not exactly what you were asking, but RAD tried to build a debugger for Linux^1, but the project died. I’ve heard the people at RAD described as hard core old school programmers, so if they struggled with this it must just be a hard problem 1. http://www.radgametools.com/debug.htm http://www.radgametools.com/debug.htm

323 4y ago

I worked on a CAD/CAM product, debug mode was literally unusable, it was 100-1000 times slower than optimized mode.

bluGill 4y ago

I'm with Kernigham and Pike: the debugger is for getting a stack trace out of core dumps. Printf is for debugging.

jjtheblunt 4y ago
oh where is that quote? that's great!
- bluGill 4y ago
  In the 'practice of programming'. https://www.cs.princeton.edu/~bwk/tpop.webpage/ https://www.cs.princeton.edu/~bwk/tpop.webpage/ I last read it some years ago so I can't tell you exactly what they said. I do know that when pointed out I realized I suffer from the same sins they hate about debuggers: I would step through code mindlessly, changing variables and the like, but never stopping to think. While printf is more painful it forces me to think why is the code in that state, which is key to fixing bugs.
  - jjtheblunt 4y ago
    i totally agree
properparity 4y ago
Why limit yourself to crusty text when you can have live visualizations of data structures in various formats, graph dx/dy over time of some variable, stop and inspect data on some condition without having to modify code, quickly start execution on a certain line, quickly drill down into struct members, etc. I mean it does everything a printf() can do - except just better.
- Night_Thastus 4y ago
  For a couple months I had no functioning debugger and was left with only Printf's. I do not ever desire to go back to that. It is far too painful. I think my time to find issues likely was 4x+ longer than before. Some programs simply have too much state, too complex of structures (for performance or memory reasons), or very long runtimes to reasonably debug purely with prints.

Deganta 4y ago

The problem is that C++ is full of so called "zero-cost-abstractions" that are only zero cost if you ignore: 1. Compile time 2. Non-optimized builds Stuffing everything into the Standard Library using lots of template magic does has its upside, but debug performance is one of the big downsides. Another one are really clunky interfaces, like std::variant. This should really have been a language feature, not a library features.

titzer 4y ago
The phrase "zero-cost abstraction" is a lie anyway. It's all relative to how much compiler optimization happens. If you actually executed the C++ specification step by step, the code would be dog slow, because there are so many implicit conversions, constructions, inlining, constant folding, etc. We could quibble about how much static resolution qualifies (template expansion, overload resolution, etc--that's executing an inflated in-memory IR, and not the original source code.) Not to mention that all the rest of the compiler backend is working hard to get good machine code, and cannot possibly produce "optimal" machine code, so there is an unmeasurable cost just because of the limitations of any one particular compiler backend. (And those things matter, too!)
- olliej 4y ago
  If you implemented any language following the spec in a step by step manner they would be dog slow, because any usable spec has to be very explicit in every step required for every operation. All those conversions that you believe only happen if you implement the spec “step by step” always happen. Actual implementations of languages then implement the explicitly described semantics, in whatever way they feel is most efficient.
  - titzer 4y ago
    Bah. Compiling Java to bytecode consists of a lot of static resolution steps and then a fairly straightforward, simple translation. Single-stepping the bytecodes will absolutely produce the same results as the optimized JIT code and there are few expressions that generate more than 1 to a handful of bytecodes.

StellarScience 4y ago

MSVC lacks -Og because it instead treats inlining, optimization, and inclusion of debug information as separate, orthogonal choices. These days I rarely build a whole MSVC project as "Debug" without optimization. Instead I enable both optimization and debug information, so I always get call stacks and line-by-line stepping ability. When I end up debugging some file where the optimizations inhibit debuggability, I'll recompile just that compilation unit or library without optimization or inlining. That said, I agree with the gist of the article that both "zero cost abstraction" and debuggability need to be constantly improved.

plorkyeran 4y ago
I think you misunderstand what -Og is. All of the major compilers treat optimization and inclusion of debug information as separate choices, and generating debug symbols for optimized release builds is a very normal thing to do. GCC's -Og is just an attempt at selecting a set of optimizations which don't cause too many problems for debuggability.
- mort96 4y ago
  It's worth noting that it does a pretty bad job at it. I've had to go back and remove -Og any time I've used it because it just makes so many variables "optimised out".

_gabe_ 4y ago

I ran into these problems with std::unordered_map. While developing my game, I would always run it in debug mode because I need to debug it. Loading a game world was taking around 12 seconds, and it was beginning to grate on me because of the increased iteration times. So I decided to profile it and see what was taking so long. I compiled in release mode to begin debugging, and it magically took the world load time down to 2 seconds. I wondered what code I had written that caused such a massive performance hit in debug mode. So I went to profile my code in a debug build to find out. Lo and behold, something like 90% of the CPU time was wasted doing lookups from std::unordered_map because of debug iterators. I tried everything I could to just turn debug iterators off, and eventually gave up and switched to robin_hood::unordered_map which runs great in debug and release. Now I have an unwarranted aversion to the std lib, even though it really is great so long as you're running it in release mode.

foonathan 4y ago
To be fair, std::unordered_map is also very slow in release mode.
leni536 4y ago
> debug iterators MSVC then, I presume.
gernb 4y ago
It'd maybe be nice to be able to mark which code gets optimized. If I could make all std as "don't debug this and keep it optimized" I would. I know some debuggers have the option to at least not step into certain code that that doesn't solve the optimization issue
- TillE 4y ago
  As others have mentioned, you can accomplish the equivalent by using a release build with pragmas to turn off optimizations in certain sections of code.
- gumby 4y ago
  You don't have to compile every file with the same flags. You can vary them with each invocation of the compiler, so trivially get what you want at the translation unit level.

staticassertion 4y ago

With our rust project we actually compile with `01` because the cost of that optimization pass is less than the cost to our tests running 100x slower. There is definitely a need for a 'more optimal' debug mode, and it is one of those things that feels zero cost until it isn't. Similarly, I try to at least consider compile times when writing code these days.

PaulDavisThe1st 4y ago

I've been using C++ since 1993. I still don't understand how I've written all this code and never once found myself using, or wishing I could use std::move. Presumably my code has been less than optimally efficient or something, because it seems that most people talking about "modern" C++ view it as absolutely central to the language.

the-lazy-guy 4y ago
std::move is not only about performance. It is about being able to disallow destructive copies of certain types. I.e. unique_ptr can not be copied, and auto_ptr can. Can you write perfectly functioning code without these features? Sure! You just have to be accurate. Same with performance. Where nowadays simple std::move() on a return value suffices you had previously explicitly to pass the object/vector with which you swapped as a parameter. Life with move semantics is a tad easier.
- MauranKilom 4y ago
  Note that you don't even have to std::move a return value in almost all cases. (The very much non-language-lawyer explanation is that local variables are automatically moved from when you return them.)
  - dryanau 4y ago
    Also, if you write > return std::move(ret); it can interfere with return value optimisation (RVO). There might be cases where the move makes sense but to my understanding it's enough to just use return. It's very much possible to take advantage of modern C++ move semantics without an explicit call to move. Maybe you return a unique_ptr created by make_unique in a factory method, or call swap() on a moveable object, etc. I rarely need to write std::move to take advantage of modern C++ although it took some self-education about rvalues etc to learn how to do it.
  - olliej 4y ago
    As u/dryanau says returning std::move breaks the return value optimization such that clang at least will warn about it by default.

not2b 4y ago

The author needs to learn about the -Og switch, which can be used together with debugging. It can be thought of as "do those optimizations that don't interfere with debugging". I verified with Godbolt that this suffices to eliminate the call overhead.

SuperV1234 4y ago
Did you even read the article? I mentioned '-Og' several times and its shortcomings.
- not2b 4y ago
  I read it, but too quickly. Sorry. As you say, the main problem is that only gcc has a usable implementation; the other issues, for me, are an acceptable tradeoff (reasonable speed, but sometimes difficulty because inlining hid something).

olliej 4y ago

The urge to have basic language features not be part of the language, but instead be part of the standard library is responsible for a lot of the less than stellar parts of of C++. Std::move/forward are particularly obvious cases, but you get similar accessing members of tuples, etc. Wanting to keep things in the standard library rather than the language itself means that you have to compile large amounts of template hell for a wide array of basic things which hurts build time even in debug modes, and then as this article says you end up with no-op operations that become calls. You also can’t easily debug any of this because you end up with absurd layers of template nonsense for what is again basic functionality. You can compile with -O1, but that then inlines things that aren’t part of the standard library that makes debugging of the actually relevant code annoying (this is what the debug llvm and clang debug builds do), as it inlines your own code and also means that you lose variables all over the place.

andrewmcwatters 4y ago

You know, at least in C++ you get some really poorly performing standard containers. Every time I reach for C, it drives me nuts that the best you get is GLib. And BOTH pale in comparison to newer language standard libraries like Go's.

phaedrus 4y ago

This dovetails with something I was thinking about recently in the context of Reproducible Builds (or the lack thereof). For most of its history Computer Science (at least as practiced by industry) has gone all-in on "functionally equivalent" transformations, and done almost nothing in the space of "output invariant" guarantees. In my day job we sometimes we need to reproduce the build of a firmware ROM or executable, sometimes decades after the engineer who last built it left. Getting a match is easier (or even possible) for older tech only because of the relative unsophistication of the compilers used - even then it's only reproducible "by accident," and not because the compiler vendor made any guarantees about X language construct reliably produces Y machine code and data layout. But we need that! For getting accurate baselines. For security verification. And there's no reason in principle we should have to forego updating compilers, IDEs, and OS environments in the indirect hope of not disturbing anything. Those are two separate things: if we had a through-line of higher level language construct -> semantically defined transformation (irregardless of optimization settings) -> machine code, vendors could continue to update their IDEs and compilers while just making sure they still respect the invariants. C++'s so-called zero-cost abstractions are poor substitute for this: header (library) writers and C++ gurus write code as if they worked like this "guaranteed output transform" I describe, but no compiler actually has to respect it (nevermind that the fine details of what the transformation actually is isn't nailed down) and it differs between Debug and Release build which is particularly bad for game development as TFA makes clear.