3 ms·

Having debugged a lot of optimized code I would strongly recommend against it unless you are in a context where performance of your build is paramount (games?)

Having debugged a lot of optimized code I would strongly recommend against it unless you are in a context where performance of your build is paramount (games?) or you cannot reproduce the bug when compiled without optimizations. Compilers really do a terrible job at preserving useful debug info when you turn them on. It’s a massive pain to have everything be marked as “optimized out” and reassemble the things you want from other variables or by using a disassembler to manually track which register the value is hiding in.

SleepyMyroslav 2y ago
It's not only games. Anything sizeable that needs to run to repro will crumble under 20-100x times slowdown. Multithreaded behaviors will be just different. All those wonderful templated abstractions do not come for free in -O0. Ranges are especially egregious example. Debug build is truly dead outside of unit testing (imho). Realistic scenario that gamedev uses: deoptimize translation units you are interested in finding or reproducing bugs.
- senkora 2y ago
  > Realistic scenario that gamedev uses: deoptimize translation units you are interested in finding or reproducing bugs. Yep, looks like that’s this bullet point: https://dhashe.com/category/blog.html#partition-your-tus-into-debuggable-and-fast https://dhashe.com/category/blog.html#partition-your-tus-int...
- flohofwoe 2y ago
  > 20-100x This would be really unusual though right? For reference, in plain C code I see a 2x slowdown, in Zig a 4x slowdown (which I still need to investigate why exactly that's the case), and in C++ (even with heavy stdlib usage and on MSVC) at most 10x - which is the absolute worst case I've seen yet. My C++ info is a bit outdated though, have things gotten much worse in "modern" C++? Or in other words: if you see a slowdown of 100x in debug mode, I would be really concerned about why the performance is so heavily dependent on the optimizer doing it's thing and would start investigating what's the reason for such a massive slowdown.
  - Cu3PO42 2y ago
    At work, I have a large C++ codebase in which I can observe slowdowns of one order of magnitude or more with -O0 compared to -O3. And for many bugs it still takes 10 minutes of runtime to hit them, so not using optimizations just isn't tenable. As far as I can tell, this is worse than for the average C++ codebase. I have some ideas for why the optimizer is able to achieve such large improvements with this particular code and some of that could surely be done with better source, but it's a huge code base and a refactoring on that scale just isn't going to happen.
  - TinkersW 2y ago
    At least with MSVC if you use SIMD heavily the debug build will be 100x slower easily. I believe it creates debugable SIMD by storing each operation to memory, though I'm not entirely sure how they managed such a massive slowdown even so:) Anyway the way I deal with it is to mark functions I want to debug with a macro that disables optimizations just for that function.
  - SleepyMyroslav 2y ago
    Does memory of such C++ that can be used in debug build exists somewhere? Yeah, one can still write that. Example from open source: tracy_force_inline void swap( Vector& other ) { uint8_t tmp[sizeof( Vector<T> )]; memcpy( (char*)tmp, &other, sizeof( Vector<T> ) ); memcpy( (char*)&other, this, sizeof( Vector<T> ) ); memcpy( (char*)this, tmp, sizeof( Vector<T> ) ); } This is quite fast in debug. Will people who write C++ in 202x gamedev do it? Nah, I don't think so. Just stop even maintaining that pesky 'Debug' build and it will be fine. Build configurations grow in other direction. There are optimized configurations already that do not need local build step - LTO/PGO and friends. People just do not build that locally and real world performance is there. YMMV of course.
mark_undoio 2y ago
> It’s a massive pain to have everything be marked as “optimized out” and reassemble the things you want from other variables or by using a disassembler to manually track which register the value is hiding in. If you've got a time traveling / reversible debugger than you can (sometimes) go back to a point where the value was being written / used, at which point it'll often reappear in scope and be accessible. I believe DWARF's built-in virtual machine should be able to recompute missing values in many cases but I don't think compilers are great at putting the relevant info in, even where it should be possible to compute the right value fairly easily.
- mark_undoio 2y ago
  The other trick I've found really helpful for "optimized out" values is to find places where they cross boundaries that block optimizations (e.g. procedure calls to another translation unit, so long as you're not doing some kind of link-time optimisation). e.g. if the value you're interested in is being passed to / returned from a function then inspecting it around the call / return site should have the value available.
  - o11c 2y ago
    Two particular notes around this (exact commands assuming gdb, but other debuggers should have equivalents): Pass various arguments to `backtrace` rather than just relying the default. Chances are it will have some non-optimized-out variables, which you can use to figure out what's going on. Use `info registers` and see what looks like a pointer, then cast it to a type you suspect it is. Note that this can be done for any stack frame.
  - account42 2y ago
    Even with LTO or inside translation units, compilers are (sadly) extremely conservative about changing function boundaries.
    - mark_undoio 2y ago
      I've seen gcc do an enthusiastic job on `static` functions within a C compilation unit - specifically where they only have one call site and so can be fully inlined into the caller. In that case, the code did become pretty hard to debug due to the extensive inlining and reordering it had allowed. Unfortunate because the only reason such functions exist is to make the structure of the code more apparent! Maybe that's an exception (and / or maybe it's easier with C than for C++). Maybe the less is that it's still always worth trying function call boundaries, in case the compiler has been conservative!
DyslexicAtheist 2y ago
feels strange still to see complaints about debugging in production being inconvenient when we should have caught these issues in test/staging. secondly I think not having debug tools and debug data in production is a security feature.
- saagarjha 2y ago
  You’re trolling, right?

SleepyMyroslav 2y ago

It's not only games. Anything sizeable that needs to run to repro will crumble under 20-100x times slowdown. Multithreaded behaviors will be just different. All those wonderful templated abstractions do not come for free in -O0. Ranges are especially egregious example. Debug build is truly dead outside of unit testing (imho). Realistic scenario that gamedev uses: deoptimize translation units you are interested in finding or reproducing bugs.

senkora 2y ago
> Realistic scenario that gamedev uses: deoptimize translation units you are interested in finding or reproducing bugs. Yep, looks like that’s this bullet point: https://dhashe.com/category/blog.html#partition-your-tus-into-debuggable-and-fast https://dhashe.com/category/blog.html#partition-your-tus-int...
flohofwoe 2y ago
> 20-100x This would be really unusual though right? For reference, in plain C code I see a 2x slowdown, in Zig a 4x slowdown (which I still need to investigate why exactly that's the case), and in C++ (even with heavy stdlib usage and on MSVC) at most 10x - which is the absolute worst case I've seen yet. My C++ info is a bit outdated though, have things gotten much worse in "modern" C++? Or in other words: if you see a slowdown of 100x in debug mode, I would be really concerned about why the performance is so heavily dependent on the optimizer doing it's thing and would start investigating what's the reason for such a massive slowdown.
- Cu3PO42 2y ago
  At work, I have a large C++ codebase in which I can observe slowdowns of one order of magnitude or more with -O0 compared to -O3. And for many bugs it still takes 10 minutes of runtime to hit them, so not using optimizations just isn't tenable. As far as I can tell, this is worse than for the average C++ codebase. I have some ideas for why the optimizer is able to achieve such large improvements with this particular code and some of that could surely be done with better source, but it's a huge code base and a refactoring on that scale just isn't going to happen.
- TinkersW 2y ago
  At least with MSVC if you use SIMD heavily the debug build will be 100x slower easily. I believe it creates debugable SIMD by storing each operation to memory, though I'm not entirely sure how they managed such a massive slowdown even so:) Anyway the way I deal with it is to mark functions I want to debug with a macro that disables optimizations just for that function.
- SleepyMyroslav 2y ago
  Does memory of such C++ that can be used in debug build exists somewhere? Yeah, one can still write that. Example from open source: tracy_force_inline void swap( Vector& other ) { uint8_t tmp[sizeof( Vector<T> )]; memcpy( (char*)tmp, &other, sizeof( Vector<T> ) ); memcpy( (char*)&other, this, sizeof( Vector<T> ) ); memcpy( (char*)this, tmp, sizeof( Vector<T> ) ); } This is quite fast in debug. Will people who write C++ in 202x gamedev do it? Nah, I don't think so. Just stop even maintaining that pesky 'Debug' build and it will be fine. Build configurations grow in other direction. There are optimized configurations already that do not need local build step - LTO/PGO and friends. People just do not build that locally and real world performance is there. YMMV of course.

mark_undoio 2y ago

> It’s a massive pain to have everything be marked as “optimized out” and reassemble the things you want from other variables or by using a disassembler to manually track which register the value is hiding in. If you've got a time traveling / reversible debugger than you can (sometimes) go back to a point where the value was being written / used, at which point it'll often reappear in scope and be accessible. I believe DWARF's built-in virtual machine should be able to recompute missing values in many cases but I don't think compilers are great at putting the relevant info in, even where it should be possible to compute the right value fairly easily.

mark_undoio 2y ago
The other trick I've found really helpful for "optimized out" values is to find places where they cross boundaries that block optimizations (e.g. procedure calls to another translation unit, so long as you're not doing some kind of link-time optimisation). e.g. if the value you're interested in is being passed to / returned from a function then inspecting it around the call / return site should have the value available.
- o11c 2y ago
  Two particular notes around this (exact commands assuming gdb, but other debuggers should have equivalents): Pass various arguments to `backtrace` rather than just relying the default. Chances are it will have some non-optimized-out variables, which you can use to figure out what's going on. Use `info registers` and see what looks like a pointer, then cast it to a type you suspect it is. Note that this can be done for any stack frame.
- account42 2y ago
  Even with LTO or inside translation units, compilers are (sadly) extremely conservative about changing function boundaries.
  - mark_undoio 2y ago
    I've seen gcc do an enthusiastic job on `static` functions within a C compilation unit - specifically where they only have one call site and so can be fully inlined into the caller. In that case, the code did become pretty hard to debug due to the extensive inlining and reordering it had allowed. Unfortunate because the only reason such functions exist is to make the structure of the code more apparent! Maybe that's an exception (and / or maybe it's easier with C than for C++). Maybe the less is that it's still always worth trying function call boundaries, in case the compiler has been conservative!

DyslexicAtheist 2y ago

feels strange still to see complaints about debugging in production being inconvenient when we should have caught these issues in test/staging. secondly I think not having debug tools and debug data in production is a security feature.

saagarjha 2y ago
You’re trolling, right?