8 ms·

How to build highly-debuggable C++ binaries

218 points by synergy20 2y ago

mark_undoio 2y ago
Good to see this discussed - debuggability is not talked about enough but, done right, it could be a superpower. Setting the build for an old x64 machine (https://dhashe.com/how-to-build-highly-debuggable-c-binaries.html#set-the-build-architecture-to-base-x86_64 https://dhashe.com/how-to-build-highly-debuggable-c-binaries...) for reversible / time travel debuggers seems unnecessarily restrictive to me. I'd expect a modern time travel debug tool (e.g. either rr or Undo - disclaimer, which I work on) to cope fine with most modern instructions (I believe GDB's built-in record / replay debugging tends to be further behind the curve on new CPU instructions - but if you're doing anything at scale it's not the right choice anyhow). Regarding compilation (https://dhashe.com/how-to-build-highly-debuggable-c-binaries.html#partition-your-tus-into-debuggable-and-fast https://dhashe.com/how-to-build-highly-debuggable-c-binaries...) - we generally advise customers to use -Og rather than -O0. As the article states, this will still optimise out some code but should be a good trade-off without being too slow. (NB. last I checked, clang currently uses -Og as an alias for -O1, so it may behave less satisfactorily than under GCC). It's also not said enough but: you don't need a special debug build to be able to debug. It's less user-friendly to debug a fully-optimised release build but it's totally possible. You just need to retain the DWARF debug info (instead of throwing it away). This is really important to know if you're debugging on a customer system or analysing a bug that's only in release builds.
- saagarjha 2y ago
  Having debugged a lot of optimized code I would strongly recommend against it unless you are in a context where performance of your build is paramount (games?) or you cannot reproduce the bug when compiled without optimizations. Compilers really do a terrible job at preserving useful debug info when you turn them on. It’s a massive pain to have everything be marked as “optimized out” and reassemble the things you want from other variables or by using a disassembler to manually track which register the value is hiding in.
  - SleepyMyroslav 2y ago
    It's not only games. Anything sizeable that needs to run to repro will crumble under 20-100x times slowdown. Multithreaded behaviors will be just different. All those wonderful templated abstractions do not come for free in -O0. Ranges are especially egregious example. Debug build is truly dead outside of unit testing (imho). Realistic scenario that gamedev uses: deoptimize translation units you are interested in finding or reproducing bugs.
    - senkora 2y ago
      > Realistic scenario that gamedev uses: deoptimize translation units you are interested in finding or reproducing bugs. Yep, looks like that’s this bullet point: https://dhashe.com/category/blog.html#partition-your-tus-into-debuggable-and-fast https://dhashe.com/category/blog.html#partition-your-tus-int...
    - flohofwoe 2y ago
      > 20-100x This would be really unusual though right? For reference, in plain C code I see a 2x slowdown, in Zig a 4x slowdown (which I still need to investigate why exactly that's the case), and in C++ (even with heavy stdlib usage and on MSVC) at most 10x - which is the absolute worst case I've seen yet. My C++ info is a bit outdated though, have things gotten much worse in "modern" C++? Or in other words: if you see a slowdown of 100x in debug mode, I would be really concerned about why the performance is so heavily dependent on the optimizer doing it's thing and would start investigating what's the reason for such a massive slowdown.
      - Cu3PO42 2y ago
        At work, I have a large C++ codebase in which I can observe slowdowns of one order of magnitude or more with -O0 compared to -O3. And for many bugs it still takes 10 minutes of runtime to hit them, so not using optimizations just isn't tenable. As far as I can tell, this is worse than for the average C++ codebase. I have some ideas for why the optimizer is able to achieve such large improvements with this particular code and some of that could surely be done with better source, but it's a huge code base and a refactoring on that scale just isn't going to happen.
- o11c 2y ago
  > you don't need a special debug build to be able to debug Note that this is highly dependent on choice of compiler. Clang is utter crap for debugging even at -O1, but I've encountered basically no trouble ever using GCC at -O2 (you do have to learn a little about how the binary changes but that's easy enough to pick up). I really would not recommend -O3; historically it introduced bugs and regardless it makes the build process much slower, and the performance gain is fairly negligible (I can't say how much it destroys debuggability due to lack of experience). I can't speak for MSVC personally but it's a bad sign that its culture strongly promotes separate debug builds. That said, sanitizers are a place where a special debug build does help. Valgrind can do many of the things that sanitizers can but is around 10× slower which is a real pain if you can't isolate what you're targeting, so recompiling for sanitizers is a good idea. (Other brief notes) I have never actually encountered a case where the lack of frame pointers actually caused problems. As far as I'm concerned, any tool that breaks without them is a broken tool. (Theoretically they can speed up large traceback contexts if you're doing extensive profiling; good API design probably helps for the sanitizers case here) Rather than assembly int3, Unix-portable `SIGTRAP` is very useful for breakpoints; debuggers handle it specially. You can ignore it for non-debugged runs but get breakpoints when you are debugging without changing the binary or options! Alternatively you could leave it unignored if you have tooling that dumps core or something nicely for you.
  - omoikane 2y ago
    Debugging experience aside, I found that "-O3" is generally worth it if you also set "-march=native". For example, here are some run times for computing SHA256, you can see that there is slightly more to be gained going from -O2 to -O3 with -march=native: -O2: 10.22 -O3: 9.82 -O2 -march=native: 9.86 -O3 -march=native: 9.43 This is basically SHA256 over ~8GB of data, averaged over 5 runs. The numbers are rather crude here since I measured them just now, but I remember it was more significant when I first did it last month for https://news.ycombinator.com/item?id=40687942 https://news.ycombinator.com/item?id=40687942
    - josephg 2y ago
      Yeah -march=native is amazing. I use it when compiling & benchmarking rust code. But - to anyone reading this later - please don’t do this blindly. You probably never want to distribute binaries with this flag set. It enables all the features available on the host CPU. So your build will change depending on the physical cpu you have installed. If you have a modern amd cpu, it may enable avx512 extensions and make your binary unusable on many Intel CPUs.
- dhashe 2y ago
  (author) > I believe GDB's built-in record / replay debugging tends to be further behind Yep, I hit this issue on gdb’s builtin stuff. I added a footnote linking here and saying that this is probably unnecessary for rr and Undo. Thanks!
hurpdurpdurp 2y ago
Great article with lots of good advice, but it makes me wonder what the consensus is on using ASAN in production. Once upon a time it was widely said that ASAN should not be used for production code. The authors advocated against it and from a general-purpose security perspective it gives attackers a very large writable memory region at a fixed offset to play with. But over time I see more and more ASAN code in production on the theory that ASAN may make a system easier to exploit, but a memory corruption will make it easier to exploit. And so it's better to have knowledge of the issue. Also, I've personally found the glibc malloc tunables very useful for debugging.
- ryandrake 2y ago
  To me, leaving a debug tool on in production because it happens to mask a bug is like the old (mal)practice of turning off optimizations in Release builds because of hard-to-debug crashes. Better to just fix the crashes.
  - zbentley 2y ago
    I only have a problem with one word in that statement: "just".
  - debatem1 2y ago
    ASAN doesn't mask bugs-- quite the opposite, it turns silent memory corruptions into crashes.
- self_awareness 2y ago
  I think that using ASAN in production is a terrible idea, from the reasons you've provided. Also a memory corruption might not be there, but ASAN is always there, so we're switching a potential open attack vector for an guaranteed open attack vector. But generally, using ASAN in production is not what ASAN is for. If someone needs "memory safety" that ASAN provides, and doesn't care about slower runtime, then why did they use C++ in the first place? Just use Java. I understand this is not an option for old codebases though. Also, using ASAN in production is like using a library for which the author states it's only meant for debugging and they doesn't really care about introducing any attack vectors in future versions. Even if it's not "exploitable" now, it might be in the future. Why would anyone want to use such library and take the responsibility that nothing bad will happen on the customer machine?
jeffbee 2y ago
Anyone have tips on getting good stack traces in opt builds? I am really struggling with it at the moment. LLVM sanitizers all generate brilliant stack traces by forking llvm-symbolizer and feeding it the goods. But during runtime crashes on optimized binaries I don't seem to get good stack traces. One of the problems is that some library backtrace functions do not print the base address of the DSO mapping, which means they are printing a meaningless PC that can't be used to find file and line later.
- saagarjha 2y ago
  Is calling dladdr on the addresses not enough for you?
  - jeffbee 2y ago
    It's not async signal safe, so I did not even try that. I think there's a huge amount of complexity both inherent to the problem and caused by fifty years of accumulated bad habits, which is indicated by the thousands of lines of code in compiler-rt dedicated to handling this issue. I'd like to call their library functions but they are all in private-looking namespaces. I also tried to use the Abseil failure signal handler but it often fails to unwind and even when it does unwind has a habit of just printing unknown for the symbol name or file, and never prints the DSO base addresses.
- bogwog 2y ago
  Have you looked into using a library like Breakpad (https://chromium.googlesource.com/breakpad/breakpad/ https://chromium.googlesource.com/breakpad/breakpad/)? It's probably too much work to integrate for local debugging only though.
- mark_undoio 2y ago
  If you're on *NIX have you tried just invoking gstack or similar as an external process? https://linux.die.net/man/1/gstack https://linux.die.net/man/1/gstack Or, indeed, getting a core dump and applying GDB to it. GDB seems generally pretty good at reconstructing stacks at arbitrary points in application runtime. We've also used a combination of libunwind and https://linux.die.net/man/1/addr2line https://linux.die.net/man/1/addr2line to produce good crash dumps when GDB is not necessarily available.
ggambetta 2y ago
About a million years ago (OK, more like 20) I was making casual videogames in C++ and I wanted a cross-platform (Linux, Mac, Windows) way to get a stack trace whenever a game crashed. What I ended up doing was adding a macro to the first line of every function, let's call it STACKTRACE, which was something like #define STACKTRACE GLOBAL_STACK_FILE[GLOBAL_STACK_IDX] = __FILE__; GLOBAL_STACK_LINE[GLOBAL_STACK_IDX++] = __LINE__; StackTraceCleaner stc; StackTraceCleaner was a class that didn't do anything but execute GLOBAL_STACK_IDX-- in its destructor. So at any point in time I could inspect GLOBAL_STACK_FILE and GLOBAL_STACK_LINE and have a complete stack trace of the game. Obviously this only worked because these games weren't performance-critical and because they were essentially single-threaded, but it did the job at the time. We're talking about a time when Visual Studio 6's support for templates was half-broken, and the STL wasn't exactly S, to the point that I had to roll out my own string, smart pointers, containers, etc -- made twice as hard because of the aforementioned broken template support in VS6 :( I do miss these simpler, more innocent times, though.
- VikingCoder 2y ago
  Back in VS6 days, I did a similar thing, but I generated a GUID for the first line of every function. STACKTRACE("e0957136-fed3-414d-80b9-8bbf84f3fa03"); With the GUID, I could see where functions moved as they were refactored. I would write out the GUID to a thread-local file handle, along with a time-stamp, and an "enter" or "exit" when an RAII object left the stack. Then I could retrospectively debug after running my program. I could see the callstack, and step through in time. I would walk my source and record GUID-to-filename/linenumber in a map. Then I could dump out a Visual Studio output that had the file name and line number, and execution time... allowing me to step forward and back through the execution. Stone knives and bearskins.
  - trentnelson 2y ago
    That's neat. The modern equivalent to that these days, on Windows, is to leverage ETW and Windows Performance Analyzer. Potentially with a custom plugin that can visualize your specific perf data as a first-class WPA citizen (i.e. indistinguishable from any other perf data being analyzed, which means you can group/query/filter etc. just like anything else). I wrote a plugin for a past employer to visualize our internal product event hierarchy performance as if it were a normal C/C++ call stack, it was pretty cool. ETW and WPA are phenomenal tools. I miss them both dearly when on Linux.
forrestthewoods 2y ago
Great post. I’m surprised it requires so much effort. On Windows you pretty just need to make a debug build and… that’s it! A nice trick with MSVC is you can turn off optimizations for TU or any block of code with: #pragma optimize( "", off ) Leaps and bounds easier than hacking the build the system.
- FpUser 2y ago
  >"On Windows you pretty just need to make a debug build and… that’s it!" I got pretty much the same on Linux when using CLion IDE from JetBrains.
- aseipp 2y ago
  I mean, to be fair, if you're fiddling with/invoking cl.exe manually then you'd need to know a lot of the general equivalents and nits listed here. MSVC's debug build will do a lot for you out of the box though which is great. That said you often have to support/know 400 random build tools when using C++ to enable things like this, so it's often useful knowledge anyway.
- o11c 2y ago
  Assuming spatulas are absent, I've never had a problem with the build system on Linux. Nonetheless, for GCC the equivalent is: #pragma GCC optimize ("O0") (see also target, push_options, and pop_options) This is also available as a per-function attribute, using both gnu and standard syntaxes: __attribute__((optimize("O0"))) [[gnu::optimize("O0")]]
  - bialpio 2y ago
    For completeness, in Clang `[[clang::optnone]]` as a per-function attribute also works fine. I'm using it for debugging quite frequently lately.
- self_awareness 2y ago
  > I’m surprised it requires so much effort. On Windows you pretty just need to make a debug build and... that’s it! Well, except the Debug build in MSVC doesn't do half of the things from this list. Also, the list tells you how to use the compilation driver directly, so when comparing stuff, you would need to use the "cl.exe" compiler. For a "default" debugging experience it's enough to use CMake's Debug build type. It even has a built-in "release with debug info" build type.
Galanwe 2y ago
What irks me the most is that in 2024, I still can't reliably embed source code in dwarf5 to get meaningful source-contextualized stacktraces and have to ship source code separately and override the source mapping.
- forrestthewoods 2y ago
  Strong agree. At least on Windows you can setup Symbol Server + Source Indexing to achieve the same result. Once upon a time I wrote a small tool that can embed full source code into PDBs. I doubt anyone has ever used it though. For proprietary software it's not uncommon to leak PDBs on accident at some point. It could be disastrous to also leak full source code! https://www.forrestthewoods.com/blog/embedding-source-code-in-pdbs-with-rust/ https://www.forrestthewoods.com/blog/embedding-source-code-i... It's relatively easy to add source indexing to PDBs. I've successfully done that for a non-standard Monorepo. Works great.
  - mark_undoio 2y ago
    There's `debuginfod` on Linux: https://developers.redhat.com/blog/2019/10/14/introducing-debuginfod-the-elfutils-debuginfo-server https://developers.redhat.com/blog/2019/10/14/introducing-de... It builds a lot on quite a simple conceptual base, benefiting from native support in gcc / clang (for embedding unique build IDs) and in GDB (for contacting the server). It can serve up both source and symbol information. I would like to see this adopted more - e.g. build infrastructure automatically populating a debuginfod server so debugging is seamless.
    - trentnelson 2y ago
      Interesting... I've been lamenting the absence of .pdbs on Linux. It sounds like this would allow dissasociating symbol info from the build artifact itself? (There's no other out-of-the-box solution to this right? i.e. having symbol info live somewhere else other than the .so/exe, that can be loaded on demand when debugging? Like .pdbs basically.)
Const-me 2y ago
Tangentially related, a few tips about offline debugging on Windows: http://const.me/articles/windbg/windbg-intro.pdf http://const.me/articles/windbg/windbg-intro.pdf Not a silver bullet but still, being able to collect and analyze user-mode crash dumps is sometimes the best way to investigate and fix bugs.
weinzierl 2y ago
Good points, but for me number one would be to avoid runtime polymorphism like the plague. If your call graph has more roots than your neighbors garden and the whole thing is a forest and not a tree you will have a hard time understanding, analyzing and ultimately debugging.
- zbentley 2y ago
  What do you mean by "runtime polymorphism" here? Vtable dispatches? Something else? I guess I'm confused by the mention of call graph roots; in my mind those are just ... entry points. Edges in the call graph might be a PITA to follow in a debugger because of indirections like vtables/optimizer inlining/etc., but isn't that separate? Or is my terminology model wrong?
- pjc50 2y ago
  People respond to the awful complexity of C++ in many ways, each trying to cut out huge sections of the language, but I think this is the first time I've seen someone argue against using virtual dispatch. Somewhere out there a Smalltalk programmer is laughing.
- gpderetta 2y ago
  In the general case, I agree that runtime dispatch makes a code base harder to understand. But specifically for runtime debugging, you can get the actual stack trace, so runtime polymorphism is not much an issue. In fact a debugger can be a convenient way to understand how complex class hierarchies end up interacting.
binary132 2y ago
I particularly liked the point that not every TU needs to be compiled in debug mode. I am working on a build system and now I’m thinking of setting aside some time to make sure debug and optimization is an object level option. In general, I think the usefulness of a well specified ABI over object code is vastly underappreciated!
- trentnelson 2y ago
  I like the idea of hacking the crap out of `compile_commands.json` and subverting it for your evil machinations outside of the normal build process. Such a hideously pragmatic tip.
  - binary132 2y ago
    Duh, just alias clang to a script that replaces its debug option and invokes clang, but only for the files you want
renox 2y ago
> Avoid stepping into irrelevant code Thanks a lot!! I don't know how many times I've stepped into the C++ standard library and it gets really annoying..
deleted 2y ago
[deleted]
amluto 2y ago
if (breakpoint_1 && (x_id == 153827)) { __asm("int $3"); } No, don’t do it quite like that. Do: __asm(“int3\n\tnop”); int3 is a “trap”, so gdb sees IP set to the instruction after int3: it’s literally the correct address plus one. gdb’s core architecture code is, to be polite, not very enlightened, and gdb does not understand that int3 works this way. So gdb may generate an incorrect backtrace, and I’ve even caught gdb completely failing to generate a trace at all in some cases. By adding the nop, IP + 1 is still inside the inline asm statement, which is definitely in the same basic block and even on the same line, and gdb is much happier.
- deleted 2y ago
  [deleted]
- cordenr 2y ago
  Doesn't gdb allow breakpoints to be made to be conditional? Add a breakpoint somewhere in the code, say added as breakpoint #2. Then; condition 2 (x_id == 153827) Or is there some other reason to not do this?
  - flohofwoe 2y ago
    Only reason I can think of is that conditional breakpoints in the debugger can be much slower than compiling that same condition right into the debuggee.
    - pjmlp 2y ago
      If the CPU doesn't support conditional hardware breakpoints, some do.
  - pjmlp 2y ago
    The only reason is that many still don't learn how to use debuggers, people write blog posts about language featuritis, rewrite X in Y, and then keep using debuggers as if stuck in the 1960's.
  - mgaunard 2y ago
breatheoften 2y ago
I recently added some pretty printers for a type we have a lot of in our codebase (the c++ Eigen library). Unlike the article we are using lldb rather than gdb ... and while I appreciate thats its possible _at all_ to script the debugger to do some pretty printing -- I found it quite a bit more frought to implement than initially expected ... To take the Eigen example ... Eigen is a 'header only' library and offers Templated vector and matrix types. The types are template over (optionally), data type, number of rows, number of columns, and matrix row order (row major or column major). All that information is not actually even available at runtime -- just the type name (with the instantiated values for template arguments) ... I ended up having to super hackily parse information out of the template type name in order to be able to pretty print the matrix appropriately in lldb ... Problems of this nature abound when debugging c++ ... Very often with a header only library, there isn't even a symbol for methods you might want to call -- so you want to eg, call the size() method on some object within the debugger to see how big it is, you'll often be out of look due to an undefined symbol reference since the 0-overahead compilation models ensures the symbol doesn't even have to be created in the binary ... Would be nice if there was some kind of way around that -- I guess I need to try the workaround mentioned in the article of explicitly instantiating template classes for common classes in 'debug' mode ... My fuzzy mental model derived from previous experience somehow doesn't think that will actually help the issue tho -- but I'd be happy to be wrong!
- Const-me 2y ago
  > Would be nice if there was some kind of way around that I’m using Windows and compile with Visual Studio. That debugger visualization file https://github.com/cdcseacave/Visual-Studio-Visualizers/blob/master/Eigen.natvis https://github.com/cdcseacave/Visual-Studio-Visualizers/blob... makes Eigen vectors and matrices show up nicely in debugger.
- vintagedave 2y ago
  A colleague was doing this for std::map<> for libc++ in LLDB recently, and he found the same thing: he had to hackily parse the template name for the types. He told me that usually for formatters, PDB has a type ID you can use to look up the type, but templates don't.
dataflow 2y ago
In case the author is here: I see --gdb3 in a lot of places on the post where I think they meant --ggdb3.
- dhashe 2y ago
  (author) Fixed, thanks!
teleforce 2y ago
>Enable frame-pointers for all functions >Compile with frame-pointers. It's good to see that enabling frame pointers are included in the recommendations for debugging purposes. The discussions on the relevance and the usefulness of frame pointers earlier this year on HN [1]: [1] The return of the frame pointers: https://news.ycombinator.com/item?id=39731824 https://news.ycombinator.com/item?id=39731824
account42 2y ago
> Enable "debug mode" or "debug hardening" within your stdlib > If using libc++: > Add this define to your CXXFLAGS: -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_DEBUG ... until the bikeshedding bastards change the define yet again in the next release.
rramadass 2y ago
One more technique is to instrument the build process for gprof/gcov to generate the runtime call graph. Ignoring the timings; With these call trees in hand for categories of inputs, i have found it easier to figure out code paths in a new codebase.

218 points by synergy20 2y ago

Good to see this discussed - debuggability is not talked about enough but, done right, it could be a superpower. Setting the build for an old x64 machine (https://dhashe.com/how-to-build-highly-debuggable-c-binaries.html#set-the-build-architecture-to-base-x86_64 https://dhashe.com/how-to-build-highly-debuggable-c-binaries...) for reversible / time travel debuggers seems unnecessarily restrictive to me. I'd expect a modern time travel debug tool (e.g. either rr or Undo - disclaimer, which I work on) to cope fine with most modern instructions (I believe GDB's built-in record / replay debugging tends to be further behind the curve on new CPU instructions - but if you're doing anything at scale it's not the right choice anyhow). Regarding compilation (https://dhashe.com/how-to-build-highly-debuggable-c-binaries.html#partition-your-tus-into-debuggable-and-fast https://dhashe.com/how-to-build-highly-debuggable-c-binaries...) - we generally advise customers to use -Og rather than -O0. As the article states, this will still optimise out some code but should be a good trade-off without being too slow. (NB. last I checked, clang currently uses -Og as an alias for -O1, so it may behave less satisfactorily than under GCC). It's also not said enough but: you don't need a special debug build to be able to debug. It's less user-friendly to debug a fully-optimised release build but it's totally possible. You just need to retain the DWARF debug info (instead of throwing it away). This is really important to know if you're debugging on a customer system or analysing a bug that's only in release builds.

saagarjha 2y ago
Having debugged a lot of optimized code I would strongly recommend against it unless you are in a context where performance of your build is paramount (games?) or you cannot reproduce the bug when compiled without optimizations. Compilers really do a terrible job at preserving useful debug info when you turn them on. It’s a massive pain to have everything be marked as “optimized out” and reassemble the things you want from other variables or by using a disassembler to manually track which register the value is hiding in.
- SleepyMyroslav 2y ago
  It's not only games. Anything sizeable that needs to run to repro will crumble under 20-100x times slowdown. Multithreaded behaviors will be just different. All those wonderful templated abstractions do not come for free in -O0. Ranges are especially egregious example. Debug build is truly dead outside of unit testing (imho). Realistic scenario that gamedev uses: deoptimize translation units you are interested in finding or reproducing bugs.
  - senkora 2y ago
    > Realistic scenario that gamedev uses: deoptimize translation units you are interested in finding or reproducing bugs. Yep, looks like that’s this bullet point: https://dhashe.com/category/blog.html#partition-your-tus-into-debuggable-and-fast https://dhashe.com/category/blog.html#partition-your-tus-int...
  - flohofwoe 2y ago
    > 20-100x This would be really unusual though right? For reference, in plain C code I see a 2x slowdown, in Zig a 4x slowdown (which I still need to investigate why exactly that's the case), and in C++ (even with heavy stdlib usage and on MSVC) at most 10x - which is the absolute worst case I've seen yet. My C++ info is a bit outdated though, have things gotten much worse in "modern" C++? Or in other words: if you see a slowdown of 100x in debug mode, I would be really concerned about why the performance is so heavily dependent on the optimizer doing it's thing and would start investigating what's the reason for such a massive slowdown.
    - Cu3PO42 2y ago
      At work, I have a large C++ codebase in which I can observe slowdowns of one order of magnitude or more with -O0 compared to -O3. And for many bugs it still takes 10 minutes of runtime to hit them, so not using optimizations just isn't tenable. As far as I can tell, this is worse than for the average C++ codebase. I have some ideas for why the optimizer is able to achieve such large improvements with this particular code and some of that could surely be done with better source, but it's a huge code base and a refactoring on that scale just isn't going to happen.
o11c 2y ago
> you don't need a special debug build to be able to debug Note that this is highly dependent on choice of compiler. Clang is utter crap for debugging even at -O1, but I've encountered basically no trouble ever using GCC at -O2 (you do have to learn a little about how the binary changes but that's easy enough to pick up). I really would not recommend -O3; historically it introduced bugs and regardless it makes the build process much slower, and the performance gain is fairly negligible (I can't say how much it destroys debuggability due to lack of experience). I can't speak for MSVC personally but it's a bad sign that its culture strongly promotes separate debug builds. That said, sanitizers are a place where a special debug build does help. Valgrind can do many of the things that sanitizers can but is around 10× slower which is a real pain if you can't isolate what you're targeting, so recompiling for sanitizers is a good idea. (Other brief notes) I have never actually encountered a case where the lack of frame pointers actually caused problems. As far as I'm concerned, any tool that breaks without them is a broken tool. (Theoretically they can speed up large traceback contexts if you're doing extensive profiling; good API design probably helps for the sanitizers case here) Rather than assembly int3, Unix-portable `SIGTRAP` is very useful for breakpoints; debuggers handle it specially. You can ignore it for non-debugged runs but get breakpoints when you are debugging without changing the binary or options! Alternatively you could leave it unignored if you have tooling that dumps core or something nicely for you.
- omoikane 2y ago
  Debugging experience aside, I found that "-O3" is generally worth it if you also set "-march=native". For example, here are some run times for computing SHA256, you can see that there is slightly more to be gained going from -O2 to -O3 with -march=native: -O2: 10.22 -O3: 9.82 -O2 -march=native: 9.86 -O3 -march=native: 9.43 This is basically SHA256 over ~8GB of data, averaged over 5 runs. The numbers are rather crude here since I measured them just now, but I remember it was more significant when I first did it last month for https://news.ycombinator.com/item?id=40687942 https://news.ycombinator.com/item?id=40687942
  - josephg 2y ago
    Yeah -march=native is amazing. I use it when compiling & benchmarking rust code. But - to anyone reading this later - please don’t do this blindly. You probably never want to distribute binaries with this flag set. It enables all the features available on the host CPU. So your build will change depending on the physical cpu you have installed. If you have a modern amd cpu, it may enable avx512 extensions and make your binary unusable on many Intel CPUs.
dhashe 2y ago
(author) > I believe GDB's built-in record / replay debugging tends to be further behind Yep, I hit this issue on gdb’s builtin stuff. I added a footnote linking here and saying that this is probably unnecessary for rr and Undo. Thanks!

hurpdurpdurp 2y ago

Great article with lots of good advice, but it makes me wonder what the consensus is on using ASAN in production. Once upon a time it was widely said that ASAN should not be used for production code. The authors advocated against it and from a general-purpose security perspective it gives attackers a very large writable memory region at a fixed offset to play with. But over time I see more and more ASAN code in production on the theory that ASAN may make a system easier to exploit, but a memory corruption will make it easier to exploit. And so it's better to have knowledge of the issue. Also, I've personally found the glibc malloc tunables very useful for debugging.

ryandrake 2y ago
To me, leaving a debug tool on in production because it happens to mask a bug is like the old (mal)practice of turning off optimizations in Release builds because of hard-to-debug crashes. Better to just fix the crashes.
- zbentley 2y ago
  I only have a problem with one word in that statement: "just".
- debatem1 2y ago
  ASAN doesn't mask bugs-- quite the opposite, it turns silent memory corruptions into crashes.
self_awareness 2y ago
I think that using ASAN in production is a terrible idea, from the reasons you've provided. Also a memory corruption might not be there, but ASAN is always there, so we're switching a potential open attack vector for an guaranteed open attack vector. But generally, using ASAN in production is not what ASAN is for. If someone needs "memory safety" that ASAN provides, and doesn't care about slower runtime, then why did they use C++ in the first place? Just use Java. I understand this is not an option for old codebases though. Also, using ASAN in production is like using a library for which the author states it's only meant for debugging and they doesn't really care about introducing any attack vectors in future versions. Even if it's not "exploitable" now, it might be in the future. Why would anyone want to use such library and take the responsibility that nothing bad will happen on the customer machine?

jeffbee 2y ago

Anyone have tips on getting good stack traces in opt builds? I am really struggling with it at the moment. LLVM sanitizers all generate brilliant stack traces by forking llvm-symbolizer and feeding it the goods. But during runtime crashes on optimized binaries I don't seem to get good stack traces. One of the problems is that some library backtrace functions do not print the base address of the DSO mapping, which means they are printing a meaningless PC that can't be used to find file and line later.

saagarjha 2y ago
Is calling dladdr on the addresses not enough for you?
- jeffbee 2y ago
  It's not async signal safe, so I did not even try that. I think there's a huge amount of complexity both inherent to the problem and caused by fifty years of accumulated bad habits, which is indicated by the thousands of lines of code in compiler-rt dedicated to handling this issue. I'd like to call their library functions but they are all in private-looking namespaces. I also tried to use the Abseil failure signal handler but it often fails to unwind and even when it does unwind has a habit of just printing unknown for the symbol name or file, and never prints the DSO base addresses.
bogwog 2y ago
Have you looked into using a library like Breakpad (https://chromium.googlesource.com/breakpad/breakpad/ https://chromium.googlesource.com/breakpad/breakpad/)? It's probably too much work to integrate for local debugging only though.
mark_undoio 2y ago
If you're on *NIX have you tried just invoking gstack or similar as an external process? https://linux.die.net/man/1/gstack https://linux.die.net/man/1/gstack Or, indeed, getting a core dump and applying GDB to it. GDB seems generally pretty good at reconstructing stacks at arbitrary points in application runtime. We've also used a combination of libunwind and https://linux.die.net/man/1/addr2line https://linux.die.net/man/1/addr2line to produce good crash dumps when GDB is not necessarily available.

ggambetta 2y ago

About a million years ago (OK, more like 20) I was making casual videogames in C++ and I wanted a cross-platform (Linux, Mac, Windows) way to get a stack trace whenever a game crashed. What I ended up doing was adding a macro to the first line of every function, let's call it STACKTRACE, which was something like #define STACKTRACE GLOBAL_STACK_FILE[GLOBAL_STACK_IDX] = __FILE__; GLOBAL_STACK_LINE[GLOBAL_STACK_IDX++] = __LINE__; StackTraceCleaner stc; StackTraceCleaner was a class that didn't do anything but execute GLOBAL_STACK_IDX-- in its destructor. So at any point in time I could inspect GLOBAL_STACK_FILE and GLOBAL_STACK_LINE and have a complete stack trace of the game. Obviously this only worked because these games weren't performance-critical and because they were essentially single-threaded, but it did the job at the time. We're talking about a time when Visual Studio 6's support for templates was half-broken, and the STL wasn't exactly S, to the point that I had to roll out my own string, smart pointers, containers, etc -- made twice as hard because of the aforementioned broken template support in VS6 :( I do miss these simpler, more innocent times, though.

VikingCoder 2y ago
Back in VS6 days, I did a similar thing, but I generated a GUID for the first line of every function. STACKTRACE("e0957136-fed3-414d-80b9-8bbf84f3fa03"); With the GUID, I could see where functions moved as they were refactored. I would write out the GUID to a thread-local file handle, along with a time-stamp, and an "enter" or "exit" when an RAII object left the stack. Then I could retrospectively debug after running my program. I could see the callstack, and step through in time. I would walk my source and record GUID-to-filename/linenumber in a map. Then I could dump out a Visual Studio output that had the file name and line number, and execution time... allowing me to step forward and back through the execution. Stone knives and bearskins.
- trentnelson 2y ago
  That's neat. The modern equivalent to that these days, on Windows, is to leverage ETW and Windows Performance Analyzer. Potentially with a custom plugin that can visualize your specific perf data as a first-class WPA citizen (i.e. indistinguishable from any other perf data being analyzed, which means you can group/query/filter etc. just like anything else). I wrote a plugin for a past employer to visualize our internal product event hierarchy performance as if it were a normal C/C++ call stack, it was pretty cool. ETW and WPA are phenomenal tools. I miss them both dearly when on Linux.

forrestthewoods 2y ago

Great post. I’m surprised it requires so much effort. On Windows you pretty just need to make a debug build and… that’s it! A nice trick with MSVC is you can turn off optimizations for TU or any block of code with: #pragma optimize( "", off ) Leaps and bounds easier than hacking the build the system.

FpUser 2y ago
>"On Windows you pretty just need to make a debug build and… that’s it!" I got pretty much the same on Linux when using CLion IDE from JetBrains.
aseipp 2y ago
I mean, to be fair, if you're fiddling with/invoking cl.exe manually then you'd need to know a lot of the general equivalents and nits listed here. MSVC's debug build will do a lot for you out of the box though which is great. That said you often have to support/know 400 random build tools when using C++ to enable things like this, so it's often useful knowledge anyway.
o11c 2y ago
Assuming spatulas are absent, I've never had a problem with the build system on Linux. Nonetheless, for GCC the equivalent is: #pragma GCC optimize ("O0") (see also target, push_options, and pop_options) This is also available as a per-function attribute, using both gnu and standard syntaxes: __attribute__((optimize("O0"))) [[gnu::optimize("O0")]]
- bialpio 2y ago
  For completeness, in Clang `[[clang::optnone]]` as a per-function attribute also works fine. I'm using it for debugging quite frequently lately.
self_awareness 2y ago
> I’m surprised it requires so much effort. On Windows you pretty just need to make a debug build and... that’s it! Well, except the Debug build in MSVC doesn't do half of the things from this list. Also, the list tells you how to use the compilation driver directly, so when comparing stuff, you would need to use the "cl.exe" compiler. For a "default" debugging experience it's enough to use CMake's Debug build type. It even has a built-in "release with debug info" build type.

Galanwe 2y ago

What irks me the most is that in 2024, I still can't reliably embed source code in dwarf5 to get meaningful source-contextualized stacktraces and have to ship source code separately and override the source mapping.

forrestthewoods 2y ago
Strong agree. At least on Windows you can setup Symbol Server + Source Indexing to achieve the same result. Once upon a time I wrote a small tool that can embed full source code into PDBs. I doubt anyone has ever used it though. For proprietary software it's not uncommon to leak PDBs on accident at some point. It could be disastrous to also leak full source code! https://www.forrestthewoods.com/blog/embedding-source-code-in-pdbs-with-rust/ https://www.forrestthewoods.com/blog/embedding-source-code-i... It's relatively easy to add source indexing to PDBs. I've successfully done that for a non-standard Monorepo. Works great.
- mark_undoio 2y ago
  There's `debuginfod` on Linux: https://developers.redhat.com/blog/2019/10/14/introducing-debuginfod-the-elfutils-debuginfo-server https://developers.redhat.com/blog/2019/10/14/introducing-de... It builds a lot on quite a simple conceptual base, benefiting from native support in gcc / clang (for embedding unique build IDs) and in GDB (for contacting the server). It can serve up both source and symbol information. I would like to see this adopted more - e.g. build infrastructure automatically populating a debuginfod server so debugging is seamless.
  - trentnelson 2y ago
    Interesting... I've been lamenting the absence of .pdbs on Linux. It sounds like this would allow dissasociating symbol info from the build artifact itself? (There's no other out-of-the-box solution to this right? i.e. having symbol info live somewhere else other than the .so/exe, that can be loaded on demand when debugging? Like .pdbs basically.)

Const-me 2y ago

Tangentially related, a few tips about offline debugging on Windows: http://const.me/articles/windbg/windbg-intro.pdf http://const.me/articles/windbg/windbg-intro.pdf Not a silver bullet but still, being able to collect and analyze user-mode crash dumps is sometimes the best way to investigate and fix bugs.

weinzierl 2y ago

Good points, but for me number one would be to avoid runtime polymorphism like the plague. If your call graph has more roots than your neighbors garden and the whole thing is a forest and not a tree you will have a hard time understanding, analyzing and ultimately debugging.

zbentley 2y ago
What do you mean by "runtime polymorphism" here? Vtable dispatches? Something else? I guess I'm confused by the mention of call graph roots; in my mind those are just ... entry points. Edges in the call graph might be a PITA to follow in a debugger because of indirections like vtables/optimizer inlining/etc., but isn't that separate? Or is my terminology model wrong?
pjc50 2y ago
People respond to the awful complexity of C++ in many ways, each trying to cut out huge sections of the language, but I think this is the first time I've seen someone argue against using virtual dispatch. Somewhere out there a Smalltalk programmer is laughing.
gpderetta 2y ago
In the general case, I agree that runtime dispatch makes a code base harder to understand. But specifically for runtime debugging, you can get the actual stack trace, so runtime polymorphism is not much an issue. In fact a debugger can be a convenient way to understand how complex class hierarchies end up interacting.

binary132 2y ago

I particularly liked the point that not every TU needs to be compiled in debug mode. I am working on a build system and now I’m thinking of setting aside some time to make sure debug and optimization is an object level option. In general, I think the usefulness of a well specified ABI over object code is vastly underappreciated!

trentnelson 2y ago
I like the idea of hacking the crap out of `compile_commands.json` and subverting it for your evil machinations outside of the normal build process. Such a hideously pragmatic tip.
- binary132 2y ago
  Duh, just alias clang to a script that replaces its debug option and invokes clang, but only for the files you want

renox 2y ago

> Avoid stepping into irrelevant code Thanks a lot!! I don't know how many times I've stepped into the C++ standard library and it gets really annoying..

deleted 2y ago

[deleted]

amluto 2y ago

if (breakpoint_1 && (x_id == 153827)) { __asm("int $3"); } No, don’t do it quite like that. Do: __asm(“int3\n\tnop”); int3 is a “trap”, so gdb sees IP set to the instruction after int3: it’s literally the correct address plus one. gdb’s core architecture code is, to be polite, not very enlightened, and gdb does not understand that int3 works this way. So gdb may generate an incorrect backtrace, and I’ve even caught gdb completely failing to generate a trace at all in some cases. By adding the nop, IP + 1 is still inside the inline asm statement, which is definitely in the same basic block and even on the same line, and gdb is much happier.

deleted 2y ago
[deleted]
cordenr 2y ago
Doesn't gdb allow breakpoints to be made to be conditional? Add a breakpoint somewhere in the code, say added as breakpoint #2. Then; condition 2 (x_id == 153827) Or is there some other reason to not do this?
- flohofwoe 2y ago
  Only reason I can think of is that conditional breakpoints in the debugger can be much slower than compiling that same condition right into the debuggee.
  - pjmlp 2y ago
    If the CPU doesn't support conditional hardware breakpoints, some do.
- pjmlp 2y ago
  The only reason is that many still don't learn how to use debuggers, people write blog posts about language featuritis, rewrite X in Y, and then keep using debuggers as if stuck in the 1960's.
- mgaunard 2y ago

breatheoften 2y ago

I recently added some pretty printers for a type we have a lot of in our codebase (the c++ Eigen library). Unlike the article we are using lldb rather than gdb ... and while I appreciate thats its possible _at all_ to script the debugger to do some pretty printing -- I found it quite a bit more frought to implement than initially expected ... To take the Eigen example ... Eigen is a 'header only' library and offers Templated vector and matrix types. The types are template over (optionally), data type, number of rows, number of columns, and matrix row order (row major or column major). All that information is not actually even available at runtime -- just the type name (with the instantiated values for template arguments) ... I ended up having to super hackily parse information out of the template type name in order to be able to pretty print the matrix appropriately in lldb ... Problems of this nature abound when debugging c++ ... Very often with a header only library, there isn't even a symbol for methods you might want to call -- so you want to eg, call the size() method on some object within the debugger to see how big it is, you'll often be out of look due to an undefined symbol reference since the 0-overahead compilation models ensures the symbol doesn't even have to be created in the binary ... Would be nice if there was some kind of way around that -- I guess I need to try the workaround mentioned in the article of explicitly instantiating template classes for common classes in 'debug' mode ... My fuzzy mental model derived from previous experience somehow doesn't think that will actually help the issue tho -- but I'd be happy to be wrong!

Const-me 2y ago
> Would be nice if there was some kind of way around that I’m using Windows and compile with Visual Studio. That debugger visualization file https://github.com/cdcseacave/Visual-Studio-Visualizers/blob/master/Eigen.natvis https://github.com/cdcseacave/Visual-Studio-Visualizers/blob... makes Eigen vectors and matrices show up nicely in debugger.
vintagedave 2y ago
A colleague was doing this for std::map<> for libc++ in LLDB recently, and he found the same thing: he had to hackily parse the template name for the types. He told me that usually for formatters, PDB has a type ID you can use to look up the type, but templates don't.

dataflow 2y ago

In case the author is here: I see --gdb3 in a lot of places on the post where I think they meant --ggdb3.

dhashe 2y ago
(author) Fixed, thanks!

teleforce 2y ago

>Enable frame-pointers for all functions >Compile with frame-pointers. It's good to see that enabling frame pointers are included in the recommendations for debugging purposes. The discussions on the relevance and the usefulness of frame pointers earlier this year on HN [1]: [1] The return of the frame pointers: https://news.ycombinator.com/item?id=39731824 https://news.ycombinator.com/item?id=39731824

account42 2y ago

> Enable "debug mode" or "debug hardening" within your stdlib > If using libc++: > Add this define to your CXXFLAGS: -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_DEBUG ... until the bikeshedding bastards change the define yet again in the next release.

rramadass 2y ago

One more technique is to instrument the build process for gprof/gcov to generate the runtime call graph. Ignoring the timings; With these call trees in hand for categories of inputs, i have found it easier to figure out code paths in a new codebase.