3 ms·
Nobody should have to decode register values, look up source files versions, or reconstruct a stack trace by hand. Crashes, both in user- and kernel-space, shou
by quotemstr 5y ago
Nobody should have to decode register values, look up source files versions, or reconstruct a stack trace by hand. Crashes, both in user- and kernel-space, should produce neat, tidy, self-contained dumps that include not only the entire machine state, but globally unique build IDs for all binaries involved in the crash. And the debugger ought to be able to load one of these crash dumps and find the debug symbols and source files automatically.
Windows has been able to do this for decades. Why, in Unix-land, are we still reading text reports about crashes and puzzling over specific register values?
- bitcharmer 5y agoHow on earth are you going to map a particular spot in the code without access to the sources? I'd love you to explain how hunting kernel bugs is easier on windows.
- quotemstr 5y agoMicrosoft publishes symbols that you can use to decode kernel crashes. The availability of source code isn't my point though: the point is that parsing text debug dumps is stone age crap and we should move to a model where crashes produce self-describing crash dumps that users can load into debuggers that find all relevant metadata automatically. You shouldn't have to puzzle over the meaning of each register in the x86_64 to figure out what value a function parameter had at crash time
- drewg123 5y agoI don't know why this is downvoted. A crashdump would have made this 10x easier to debug. The answer is because "its linux" and crashdumps (except on a few distros) are not part of the culture. These days linux actually kexec's another kernel with reserved physical memory to handle doing the dumps. This was absurdly hard to setup the last time I needed to do linux kernel debugging (years ago). The linux kernel is generally compiled without frame pointers, and with other optimizations that are hostile to debugging. We have kernel and userspace crashdumps enabled on our fleet of FreeBSD CDN servers, and having a real crashdump and not just a stack has been super helpful.
- uvdn7 5y agoI agree with you but we should leave frame pointer out of this — as it almost has nothing to do with it? libunwind knows how to read DWARF anyway. It’s more an implementation detail.
- uvdn7 5y agoIs that the same as the decode_stacktrace.sh mentioned in the blog post or significantly more sophisticated? I am sure we can always automate more things :)
- quotemstr 5y agoA debugger lets you follow pointers and inspect the entire memory space of a program. It gives you a lot more than just a text dump of a stack.
- throwawaylinux 5y agoLinux has debuggers and crash dumps.
- stefan_ 5y agoDebugging tools in Linux are in a really sad state indeed, certainly for collecting intel on production machines when crashes happen. Ironically the kernel is almost advanced here, because at least you can rely on the stacktrace working. In userland, there is ancient stuff like libunwind - last I tried it on ARM, it was too dumb to follow the instruction pointer on a NULL function call. Not to mention there is some sort of deadlock when you use an external crashdump handler - the crashed process hangs in the kernel trying to feed your handler the dump and that interferes with using ptrace to find out what happened. The only reliable way is to put the crash handling hooks directly into libc, like Android does it - but that is not a thing at all with musl/glibc.