4 ms·

There's no mention here of inspecting core files, something that very much allows you to inspect the state of a program when the bug occurred, but after the fac

There's no mention here of inspecting core files, something that very much allows you to inspect the state of a program when the bug occurred, but after the fact. Is this because inspecting core files also sucks, since it's a snapshot of a point in time, and it's no use if your stack is corrupted? Or is it because this is a lost art? In my embedded work I avoided using debuggers because setting up remote debugging can be a PITA. I figured out a long time ago that building remote debug setup into my workflow will save oodles of time in the end. Still, I find myself using plain old logging more often than not - the bug may not be in C or C++, it may be on the JavaScript run in the interpreter embedded in that C++ application.

gmueckl 7y ago
Core dumps are great, especially for the occasional odd bug that occurs only in long running processes. However, the ridiculous systemd default of attempting and failing to capture the kernel core dump into the binary journal completely ruins this. Only small, uninteresting core dumps are ever captured; large ones are silently discarded because the journal is not allowed to grow that big.
- sfink 7y ago
  Yeah, when I'm crashing and want to keep cores, I'll temporarily modify /proc/sys/kernel/core_pattern to prevent systemd from eating them. I sometimes want them suppressed, sometimes want them written out, and never want them to go to systemd.
  - Pahr3yah 7y ago
    One issue is that that sysctl is not a namespaced one.
clarry 7y ago
> Is this because inspecting core files also sucks, since it's a snapshot of a point in time Yes. Something is wrong, but you can't tell how you got into that state.
- roca 7y ago
  Right. If a core file is all you've got, then you do your best (and there are some nifty researchy tools that help with that). But if you could have a full recording instead of the core file, then of course you'd want the recording, because that dramatically increases the probability you'll be able to debug the bug (to near 100%, if the bug is in the code you've recorded).
Ace17 7y ago
"The only valid use of a debugger is to analyze deadlocks in post Mortem core dumps. Interactive debugging is the greatest time waste. Write automated tests for your debug hypotheses instead." (Peter Sommerlad)

gmueckl 7y ago

Core dumps are great, especially for the occasional odd bug that occurs only in long running processes. However, the ridiculous systemd default of attempting and failing to capture the kernel core dump into the binary journal completely ruins this. Only small, uninteresting core dumps are ever captured; large ones are silently discarded because the journal is not allowed to grow that big.

sfink 7y ago
Yeah, when I'm crashing and want to keep cores, I'll temporarily modify /proc/sys/kernel/core_pattern to prevent systemd from eating them. I sometimes want them suppressed, sometimes want them written out, and never want them to go to systemd.
- Pahr3yah 7y ago
  One issue is that that sysctl is not a namespaced one.

clarry 7y ago

> Is this because inspecting core files also sucks, since it's a snapshot of a point in time Yes. Something is wrong, but you can't tell how you got into that state.

roca 7y ago
Right. If a core file is all you've got, then you do your best (and there are some nifty researchy tools that help with that). But if you could have a full recording instead of the core file, then of course you'd want the recording, because that dramatically increases the probability you'll be able to debug the bug (to near 100%, if the bug is in the code you've recorded).

Ace17 7y ago

"The only valid use of a debugger is to analyze deadlocks in post Mortem core dumps. Interactive debugging is the greatest time waste. Write automated tests for your debug hypotheses instead." (Peter Sommerlad)