6 ms·
Inspecting coredumps like it's 2021
- jandrese 5y agoWas anybody else disappointed that it's still just gdb for the debugger? I've thought for a long time that something better must be out there, especially a debugger that better handles multiple threads in an application, but every time I need to debug something I'm back in the 80s.
- selfhoster11 5y agoReally, my kingdom for a debugger that is as accessible as the one from IntelliJ.
- astange 5y agoThe CLion front end to gdb/lldb is very good. Not perfect, but that's mainly do the oddities of interacting with the underlying debugger. https://www.jetbrains.com/help/clion/debugging-code.html https://www.jetbrains.com/help/clion/debugging-code.html
- bsdubernerd 5y agoWhat about lldb? Has anybody had extensive experience with both to say anything about the actual debugging experience? Does it offer anything more than just "infrastructure"?
- tazjin 5y agoSame basic concept, though it had some nicer representations of non-C language things OOB when I tried it.
- AshamedCaptain 5y agoWhat would be better? How to improve the handling of multiple threads? Every time I have tried a different debugger, I have always ended up coming running back to GDB. My only gripes with it are basically gripes with DWARF.
- jjav 5y agoI still use dbx from Sun Studio. The now-oracle connection taints it, but it was from good old Sun. It was always better than gdb for threaded debugging. Haven't checked if it is still available but I have it installed from long ago.
- xxpor 5y agoI don't get what it's missing other than a GUI. I can view stacktraces, view memory in various formats, create new commands on the fly or in .gdbinit... what are people missing?
- deleted 5y ago[deleted]
- tom_ 5y agoWell I miss a GUI.
- db48x 5y agoThere is something better out there. https://pernos.co/ https://pernos.co/
- GuB-42 5y agoPernosco is certainly 2021... Debugging as a service... Did anyone try it? It looks like it a web frontend for rr, but the idea seems alien to me, especially the pricing part. A subscription for a number of submissions per month?! If there is anything you can't really plan, it is bugs, one month, you may need only one or two, the next it will be 50 in a day.
- db48x 5y agoI’ve tried it, and it’s amazing. It’s more than merely a front–end for rr, because with rr you are still inspecting a single moment in time in your program. You can rewind to past moments any time you want, but that fundamental limitation remains. With Pernosco you are really querying a database that contains all the information about all the states your program was ever in. It’s a superpower that you can purchase. I think the price is generally worth it, if you have at least a couple bugs to use it on every month. If you have more than 5 per month but don’t want to go up a tier, I believe you can pay “a la carte” just for the extras. If you have enough people, and you can afford the hardware to run it, I really recommend the on–premises installation.
- richardwhiuk 5y agoDoes Nix really not have debug symbols available for most packages? That's astonishing....
- outworlder 5y agoNix is the package manager. NixOS is the OS. It's very common to strip debug symbols from packages. For Ubuntu, it's even a different repository! I fail to see why that's astonishing.
- whateveracct 5y agoPeople can be astonished by anything if they didn't ask the right questions before spouting answers :)
- comex 5y agoSplitting debug symbols into separate packages is expected. Splitting them into a separate repository is a bit surprising but no big deal. The problem with NixOS is that it doesn’t ship debug symbols at all. Nix does at least make it easy to rebuild a package with debug symbols enabled, but: - Some packages take a very long time to rebuild. - IIUC, the standard builds are done without debug info enabled at all, rather than building with it enabled and then deleting it. Usually, whether debug info is enabled shouldn’t affect any other aspects of code generation, so rebuilding with debug info should produce a symbol-rich binary that’s compatible with existing core dumps for the original binary. But this is not always the case. - If the package you want debug info for is a library, Nix will want to rebuild not just the library itself but everything that (transitively) depends on it. This is really a more general problem with Nix: it has zero notion of ABI compatibility. But always building with debug info would work around the problem.
- kingforaday 5y agoI think standard practice is to strip the binary before release.
- er4hn 5y agoOne feature of Nix is to be able to rebuild something the same way on different systems. The article suggests that adding the debug symbols is a simple flag in the build expression. Based on that, you shouldn't need the debug symbols in advance, though ymmv, when trying to debug an issue in container A on dev machine B.
- hyperman1 5y agoI see a lot of sudo in there, for debugging a program presumably not running as root. Is it possible with this tooling to debug core dumps as a normal user?
- necrotic_comp 5y agoI agree, that seems like a very bad idea.
- jandrese 5y agoSystemd's core dump utility stuffs the files in /var/lib/systemd/coredump with 640 permissions and ownership of root:root regardless of who owned the process. It might be paranoia on someone's part, but I agree that it causes a lot of unnecessary privilege escalation for people who want to actually fix the problem.
- pengaru 5y ago> Systemd's core dump utility stuffs the files in /var/lib/systemd/coredump with 640 permissions and ownership of root:root regardless of who owned the process. This isn't true for +ACL builds of systemd, look at the output of `getfacl /var/lib/systemd/coredump/$foo` and you'll see.
- jcelerier 5y ago.. but you can just run coredumpctl gdb or coredumpctl dump <the dump> > /tmo/YOLO.dump
- rwmj 5y agoThis isn't true, at least on Fedora. When my program crashes I run "coredumpctl gdb" without sudo and it works fine - I use this all the time. (It could be that Fedora has done some special configuration and that it's not like this out of the box for upstream systemd)
- e12e 5y agoI'm not at my laptop right now, but from https://www.freedesktop.org/software/systemd/man/coredumpctl.html https://www.freedesktop.org/software/systemd/man/coredumpctl... It looks like coredumpctl might be suid? Or that the service is, and coredumpctl gets access through the service - so there's a certain elevation going on either way?
- seiferteric 5y agoThis brings up a question I have had, is it not possible to put the debug info in a separate file that gdb could load instead of in the binary itself? If you could do this, you could then easily have xyz-dbg packages that only contained these files, or even an distro repository of these files that the debugger could download as needed.
- gtirloni 5y agoLike this? https://access.redhat.com/solutions/9907 https://access.redhat.com/solutions/9907
- 10000truths 5y agoYes. You can separate the debug info from an object file with objcopy: objcopy --only-keep-debug foo.o foo.o.debug objcopy --strip-debug --add-gnu-debuglink=foo.o.debug foo.o And then link foo.o as normal. GDB should automatically be able to pick up the debug file by looking at the debug link. For more info, here's a link: https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Fil... You can also use gcc's -gsplit-dwarf command line option when compiling to produce a .dwo file containing your debug information: https://gcc.gnu.org/wiki/DebugFission https://gcc.gnu.org/wiki/DebugFission
- ceztko 5y agoI wanted to ask you if the "-gsplit-dwarf" is a 100% complete alternative to strip the symbols as done previously. I ask you because some modern projects[1] still prefer to compile with symbols and strip afterward. [1] https://github.com/dotnet/runtime/blob/main/eng/native/functions.cmake https://github.com/dotnet/runtime/blob/main/eng/native/funct...
- stefan_ 5y agoIf you are in control of your build, ideally you want to embed build IDs and use a symbol server. Here is an older article from Bruce Dawson on this (unfortunately way underdeveloped) area of Linux: https://randomascii.wordpress.com/2013/02/20/symbols-on-linux-part-three-linux-versus-windows/ https://randomascii.wordpress.com/2013/02/20/symbols-on-linu...
- deleted 5y ago[deleted]
- donio 5y agoSo I guess it's still using gdb like it's 1999.
- Joker_vD 5y agoOne day we may get a vim or even emacs mode for gdb, who knows? Until then, I will continue to "enjoy" interacting with the data of the program under debugger in the ed-style, like a True Programmer™. The stories have it that back in the days some people could read actual core dumps with the tips of their fingers à la Braille, but surely that's just an urban myth.
- AshamedCaptain 5y agoGDB has a GUI (TUI). Several, in fact...
- Arnavion 5y agoIndeed, and coredumpctl can be told to pass `--tui` to the gdb invocation with `-A --tui` Also you can use another debugger of your choice with the `--debugger` flag.
- Joker_vD 5y agoI've tried gdb --tui once, and it managed to mangle my xterm's terminal geometry into something non-Euclidean, so thanks but no thanks.
- billfruit 5y agoEmacs already has the GUD front end mode for GDB.
- bla3 5y agovim has `:help terminal-debug`.
- deleted 5y ago[deleted]
- 5y ago
- waynesonfire 5y agoi love the attempt here but isn't 2021 all about containers? does k8s even generate coredumps? lol. what a joke.
- cardy31 5y agoK8s can generate coredumps if you configure it to, and they can actually still be useful in debugging certain classes of problems! However, at the multi-thousand person company I work at, there is like one guy concerned with core dumps.
- jamesfinlayson 5y agoLast time I tried to get coredumps working in a container (maybe four years ago) I got nowhere (it wasn't for a C/C++ application configuring everything to generate a core dump and then store it somewhere retrievable was too much).
- soraminazuki 5y agoCoredumps are a crucial source of information when investigating crashes and containerization doesn't invalidate any of that. Sure application restarts could minimize the impact of issues, but investigating them still is and should be standard practice. Furthermore, just because something isn't easy to do with k8s doesn't mean that it should be laughed at or scorned upon. I guess it's things like this that's causing articles to pop up that are somewhat skeptical of k8s.
- vlovich123 5y agoVisual studios support for minidumps is top notch. You can load it, there’s a symbol server if you’re dealing with a binary not built locally, and then you can fire up the parallel stacks to visually see what all the threads are doing. Coredumps and command line analysis with gdb by comparison leave much to be desired.
- tdiff 5y agoIt would be more correct to compare gdb to windbg. Clion has some support for core dumps, yet it is lagging behind msvs, ofc, but is slowly catching up.
- Torkel 5y agoYeah, my thoughts too… Inspecting coredumps with command line gdb, feels like title should be ”like it’s 2001”. Because that’s when I learned gdb :P
- soraminazuki 5y agoLinux was created in 1991, and its use is growing more than ever. When something was created doesn't really say anything about modern day usage. GDB is still in wide use today and shows no sign of becoming history. If anything, its easy scripting features continues to prompt new tools based on GDB to pop up.
- tdiff 5y agoI probably missed what exactly in this article is "like it's 2021"? Does a reference to Prometheus or systemd makes something more "modern"?
- osullivj 5y agoVisual Studio 2022 has support for debugging Linux binaries running under WSL2: https://devblogs.microsoft.com/cppblog/build-and-debug-c-with-wsl-2-distributions-and-visual-studio-2022/ https://devblogs.microsoft.com/cppblog/build-and-debug-c-wit...