5 ms·

Logging all C++ destructors, poor mans run-time tracing

90 points by jandeboevrie 2y ago

omnicognate 2y ago
Don't know why the link to the GCC runtime instrumentation docs points to the internet archive. The direct link is https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.h...
meindnoch 2y ago
And what was the bug in the end?
neverartful 2y ago
I did something similar once but my implementation didn't rely on any compiler features. I made tracing macros for constructors, destructors, and regular c++ methods. If the tracing was turned on in the macros, the information given to the macro (class name, method name, etc.) would be passed to the tracing manager. The tracing manager would serialize to a string and send it through a TCP socket. I also wrote a GUI tracing monitor that would listen on a socket for tracing messages and then display the trace messages received (including counts by class and method). The tracing monitor had filters to tweak. It was a nice tool to have and was very instrumental in finding memory leaks and obscure crashes. This was back in the late 1990s or early 2000s.
jprete 2y ago
Address/MemorySanitizer are also meant for this kind of problem. https://github.com/google/sanitizers/wiki/AddressSanitizer https://github.com/google/sanitizers/wiki/AddressSanitizer https://github.com/google/sanitizers/wiki/MemorySanitizer https://github.com/google/sanitizers/wiki/MemorySanitizer Also valgrind, but I'm more familiar with the first two.
- richardwhiuk 2y ago
  The author explicitly name checks valgrind. I think plain gdb would have been sufficient if it's exiting with a segfault or terminating...
loeg 2y ago
But what was the shutdown bug you were trying to identify? Was this destructor logging actually useful? The article teases the problem and provides detailed instructions for reproducing the logging, but doesn't actually describe solving the problem.
grahamj 2y ago
[flagged]
rqtwteye 2y ago
I did this a long time ago with macros. It helped me to find a ton of leaks in a huge video codec codebase. I still don't understand the hate for the C preprocessor. It enables doing this like this without any overhead. Set a flag and you get constructor/destructor logging and whatever else you want. Don't set it and you get the regular behavior. Zero overhead.
- jonathrg 2y ago
  The hate might have to do with it being such a primitive and blunt tool; doing anything moderately complex becomes extremely complicated and fragile.
  - tialaramex 2y ago
    Yeah, this very primitive tool easily creates the programming equivalent of the "iwizard problem". [You replace straight forwardly "mage" with "wizard" and oops, now your images are "iwizards" and your "magenta" is "wizardnta"]
  - deleted 2y ago
    [deleted]
  - immibis 2y ago
    By contrast something like AspectJ for C++ (if such a thing existed) could express this requirement cleanly.
  - akira2501 2y ago
    Complex arrangements _are_ fragile. There is no magic environment that can fix this for you. If you feel you've seen one, then you've focused on the parts of it that were important to you, while ignoring the parts of it everyone else actually needs.
- synergy20 2y ago
  do you have a write-up how you did it? I'm interested, thanks.
meisel 2y ago
I’d say address sanitizer is a better starting point, and likely to show memory issues faster than this
- tempodox 2y ago
  Yep, ASAN to find use-after-free, and valgrind memcheck to find forgotten-to-free.
  - manwe150 2y ago
    ASAN also checks for memory leaks like valgrind, the main difference with the tools is whether you can recompile all of libraries to get the compiler support for detection or whether binary instrumentation is better (https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer https://github.com/google/sanitizers/wiki/AddressSanitizerLe...)
    - tempodox 2y ago
      Thanks, I didn't know the `ASAN_OPTIONS` part for macOS.
MontagFTB 2y ago
I consider Tracy the state of the art for profiling C++ applications. It’s straightforward to integrate, toggle, gather data, analyze, and respond. It’s also open source, but rivals any product you’d have to pay for: https://github.com/wolfpld/tracy https://github.com/wolfpld/tracy
- deleted 2y ago
  [deleted]
- Veserv 2y ago
  Looks fine, but it does not look like there is a automatic full function entry/exit trace, just sampling. The real benefit is when you do not even need to insert manual instrumentation points, you just hit run and you get a full system trace. How well does the visualizer handle multi-TB traces? Usually pretty uncommon, but a 10-100 GB is not that hard to produce when doing full tracing.
  - jms55 2y ago
    Of note is that tracy is aimed at games, where sampling is often too expensive and not fine-grained enough. Hence the manual instrumenting. For the Bevy game engine, we automatically insert tracy spans for each ECS system. In practice, users can just compile with the tracy feature enabled, and get a rough but very usable overview of which part of their game is taking a long time on the CPU.
    - Veserv 2y ago
      I was talking about automatic instrumentation of every single function call by default. No manual instrumentation needed because everything is already instrumented. To be fair, you do still want some manual instrumentation to correlate higher level things, but full trace everywhere answers most questions. You also want to be able to manually suppress calls for small functions since that can be performance relevant or distorting, but the point is “default on, manual off” over “default off, manual on”.
      - 2y ago
rerdavies 2y ago
Just spent three days of debugging hell getting my app to shut down gracefully, so that it gracefully turns off all the things that it asynchronously turned on without performing use-after deletes). I can sympathise with that.
tempodox 2y ago
Too bad, it doesn't work like this on macOS.
kccqzy 2y ago
One of my favorite hacky workarounds is to simply call _exit(0) for an immediate exit without running destructors. Most of the time, the destructors are just freeing memory that will be reclaimed by the OS anyways so they are not worth running if you know the program is exiting. And even if the destructor does more than just freeing memory, maybe the work it's doing isn't needed if you know the process is ending soon: maybe it's joining threads or releasing mutexes or deleting timers. You will find that in a typical C++ codebase, the destructors that do useful things (say, flushing useful buffers and closing files etc) are much fewer.
- ignoramous 2y ago
  Sub-optimal if the exiting process must relinquish resources external to the current process or the host.
  - gpderetta 2y ago
    Optimal of you are writing crash-only, i.e. reliable, software.
    - TeMPOraL 2y ago
      Suboptimal if you're relying on RAII not for "regular" resources, but more abstract ones like "correctly bookended piece of persisted data". Then again, I guess it's optimal if you're writing fault-resistant software.
  - akira2501 2y ago
    And you guarantee you can release those resources with out error or exception all the time? If this is not the case then a destructor is the wrong place to be doing this work anyways. If an external service requires a client experiencing an error to manually release resources than that external service has significant design or protocol issues.
    - ignoramous 2y ago

90 points by jandeboevrie 2y ago

Don't know why the link to the GCC runtime instrumentation docs points to the internet archive. The direct link is https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.h...

meindnoch 2y ago

And what was the bug in the end?

neverartful 2y ago

I did something similar once but my implementation didn't rely on any compiler features. I made tracing macros for constructors, destructors, and regular c++ methods. If the tracing was turned on in the macros, the information given to the macro (class name, method name, etc.) would be passed to the tracing manager. The tracing manager would serialize to a string and send it through a TCP socket. I also wrote a GUI tracing monitor that would listen on a socket for tracing messages and then display the trace messages received (including counts by class and method). The tracing monitor had filters to tweak. It was a nice tool to have and was very instrumental in finding memory leaks and obscure crashes. This was back in the late 1990s or early 2000s.

jprete 2y ago

Address/MemorySanitizer are also meant for this kind of problem. https://github.com/google/sanitizers/wiki/AddressSanitizer https://github.com/google/sanitizers/wiki/AddressSanitizer https://github.com/google/sanitizers/wiki/MemorySanitizer https://github.com/google/sanitizers/wiki/MemorySanitizer Also valgrind, but I'm more familiar with the first two.

richardwhiuk 2y ago
The author explicitly name checks valgrind. I think plain gdb would have been sufficient if it's exiting with a segfault or terminating...

loeg 2y ago

But what was the shutdown bug you were trying to identify? Was this destructor logging actually useful? The article teases the problem and provides detailed instructions for reproducing the logging, but doesn't actually describe solving the problem.

grahamj 2y ago

[flagged]

rqtwteye 2y ago

I did this a long time ago with macros. It helped me to find a ton of leaks in a huge video codec codebase. I still don't understand the hate for the C preprocessor. It enables doing this like this without any overhead. Set a flag and you get constructor/destructor logging and whatever else you want. Don't set it and you get the regular behavior. Zero overhead.

jonathrg 2y ago
The hate might have to do with it being such a primitive and blunt tool; doing anything moderately complex becomes extremely complicated and fragile.
- tialaramex 2y ago
  Yeah, this very primitive tool easily creates the programming equivalent of the "iwizard problem". [You replace straight forwardly "mage" with "wizard" and oops, now your images are "iwizards" and your "magenta" is "wizardnta"]
- deleted 2y ago
  [deleted]
- immibis 2y ago
  By contrast something like AspectJ for C++ (if such a thing existed) could express this requirement cleanly.
- akira2501 2y ago
  Complex arrangements _are_ fragile. There is no magic environment that can fix this for you. If you feel you've seen one, then you've focused on the parts of it that were important to you, while ignoring the parts of it everyone else actually needs.
synergy20 2y ago
do you have a write-up how you did it? I'm interested, thanks.

meisel 2y ago

I’d say address sanitizer is a better starting point, and likely to show memory issues faster than this

tempodox 2y ago
Yep, ASAN to find use-after-free, and valgrind memcheck to find forgotten-to-free.
- manwe150 2y ago
  ASAN also checks for memory leaks like valgrind, the main difference with the tools is whether you can recompile all of libraries to get the compiler support for detection or whether binary instrumentation is better (https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer https://github.com/google/sanitizers/wiki/AddressSanitizerLe...)
  - tempodox 2y ago
    Thanks, I didn't know the `ASAN_OPTIONS` part for macOS.

MontagFTB 2y ago

I consider Tracy the state of the art for profiling C++ applications. It’s straightforward to integrate, toggle, gather data, analyze, and respond. It’s also open source, but rivals any product you’d have to pay for: https://github.com/wolfpld/tracy https://github.com/wolfpld/tracy

deleted 2y ago
[deleted]
Veserv 2y ago
Looks fine, but it does not look like there is a automatic full function entry/exit trace, just sampling. The real benefit is when you do not even need to insert manual instrumentation points, you just hit run and you get a full system trace. How well does the visualizer handle multi-TB traces? Usually pretty uncommon, but a 10-100 GB is not that hard to produce when doing full tracing.
- jms55 2y ago
  Of note is that tracy is aimed at games, where sampling is often too expensive and not fine-grained enough. Hence the manual instrumenting. For the Bevy game engine, we automatically insert tracy spans for each ECS system. In practice, users can just compile with the tracy feature enabled, and get a rough but very usable overview of which part of their game is taking a long time on the CPU.
  - Veserv 2y ago
    I was talking about automatic instrumentation of every single function call by default. No manual instrumentation needed because everything is already instrumented. To be fair, you do still want some manual instrumentation to correlate higher level things, but full trace everywhere answers most questions. You also want to be able to manually suppress calls for small functions since that can be performance relevant or distorting, but the point is “default on, manual off” over “default off, manual on”.
    - 2y ago

rerdavies 2y ago

Just spent three days of debugging hell getting my app to shut down gracefully, so that it gracefully turns off all the things that it asynchronously turned on without performing use-after deletes). I can sympathise with that.

tempodox 2y ago

Too bad, it doesn't work like this on macOS.

kccqzy 2y ago

One of my favorite hacky workarounds is to simply call _exit(0) for an immediate exit without running destructors. Most of the time, the destructors are just freeing memory that will be reclaimed by the OS anyways so they are not worth running if you know the program is exiting. And even if the destructor does more than just freeing memory, maybe the work it's doing isn't needed if you know the process is ending soon: maybe it's joining threads or releasing mutexes or deleting timers. You will find that in a typical C++ codebase, the destructors that do useful things (say, flushing useful buffers and closing files etc) are much fewer.

ignoramous 2y ago
Sub-optimal if the exiting process must relinquish resources external to the current process or the host.
- gpderetta 2y ago
  Optimal of you are writing crash-only, i.e. reliable, software.
  - TeMPOraL 2y ago
    Suboptimal if you're relying on RAII not for "regular" resources, but more abstract ones like "correctly bookended piece of persisted data". Then again, I guess it's optimal if you're writing fault-resistant software.
- akira2501 2y ago
  And you guarantee you can release those resources with out error or exception all the time? If this is not the case then a destructor is the wrong place to be doing this work anyways. If an external service requires a client experiencing an error to manually release resources than that external service has significant design or protocol issues.
  - ignoramous 2y ago