10 ms·
Debugging with GDB
- ho_schi 5y agoTook me some time to get familiar with it but GDB is well documented and fulfills its duty. It looks ancient but it is powerful. Do not underestimate the power of the print command! And the advanced stuff like revers-debugging, multi-threading (all-stop mode - which is the default - and the non-stop mode), binary manipulation (yep, it is possible), and remote debugging. Nowadays accompanied by stuff like debuginfod. What still need to try is the TUI modes because I assume it will improve my personal experience once I find a way of using it. I've ended up writing all stuff useful for me onto paper, later a markdown and I can recommend everyone doing similar. GDB's learning curve is not like VIM (which is either a wall or a rocket - both looking similar on paper) but it is going upwards. Today's highlight is "Why do I've to set up all my breakpoints again? You need to save them like an IDE": https://sourceware.org/gdb/onlinedocs/gdb/Save-Breakpoints.html https://sourceware.org/gdb/onlinedocs/gdb/Save-Breakpoints.h...
- dotancohen 5y ago> I've ended up writing all stuff useful for me onto paper, later a markdown Would you like to share? :)
- pif 5y ago> It looks ancient but it is powerful It is powerful because it is ancient.
- nyanpasu64 5y ago> Do not underestimate the power of the print command! Still waiting for it to print the contents of a QList or QString, or pull actual data in cases where I see <optimized out>.
- gnud 5y agoAs I understand it, you can plug in the qt pretty printers from https://invent.kde.org/kdevelop/kdevelop/-/tree/master/plugins/gdb/printers https://invent.kde.org/kdevelop/kdevelop/-/tree/master/plugi...
- nyanpasu64 5y agoHow do I use these pretty printers outside of KDevelop? When I source that file from my global .gdbinit, I get the classic Python relative import error: ModuleNotFoundError: No module named 'qt' Changing KDevelop's gdbinit to `from .qt import register_qt_printers` results in: ImportError: attempted relative import with no known parent package The problem is that sourcing KDevelop's gdbinit doesn't let Python import packages relative to that file. Is there an alternative way to do this?
- gnud 5y agoI've never actually used these, I just knew they existed. I found them via DuckDuckGo, in the github repository of qBittorrent. They have a step-by-step guide for using them, without mentioning kdevelop: https://github.com/qbittorrent/qBittorrent/wiki/Setup-GDB-with-Qt-pretty-printers https://github.com/qbittorrent/qBittorrent/wiki/Setup-GDB-wi...
- mark_undoio 5y agoThe best solution I'm aware of for <optimized out> is to choose your preferred flavour of time travel debug and then step backwards until the variable comes back into scope. This will do the trick, so long as the compiler hasn't completely elided a value. It feels like there's scope for the DWARF debug info to make optimised variables debuggable normally but that's probably a pretty hairy problem.
- mark_undoio 5y agoOn a similar note - add `set history save on` to your `~/.gdbinit` and retain command history between runs of GDB (just like with your shell). https://sourceware.org/gdb/onlinedocs/gdb/Command-History.html https://sourceware.org/gdb/onlinedocs/gdb/Command-History.ht...
- kmarc 5y agoto my surprise, vim has built-in support (:help termdebug) for gdb. It's not as chiselled as some GUI debuggers but still usable and makes it (visually) easier to debug in terminal. Accidently I also moved my mouse cursor over a variable while stopped at a breakpoint, in a terminal, over ssh, and there is a hovering balloon popping up with the current state of that variable. Mind blown.
- dijit 5y agoI still struggle with GDB but my excuse is that I seldom use it. When I was studying reverse engineering though, I came across a really cool kit (which I've yet to find an alternative for lldb, which would be nice given: rust) I'd recommend checking it out, if for no other reason than it makes a lot of things really obvious (like watching what value lives in which register). https://github.com/hugsy/gef https://github.com/hugsy/gef LLDB's closest alternative to this is called Venom, but it's not the same at all. https://github.com/ovh/venom https://github.com/ovh/venom
- synergy20 5y agoI was using gdb-dashboard: https://github.com/cyrus-and/gdb-dashboard https://github.com/cyrus-and/gdb-dashboard seems kind of like gef.
- doix 5y agoIn a past life I used Voltron[0] with lldb. Depending on your use case, it might be enough? Watching what value lives in what register works at least (or at least worked last time I used it). It's designed around having things in separate terminals, so you'll need tmux/screen/tiling window manager to get a similar view to gef. [0] https://github.com/snare/voltron https://github.com/snare/voltron
- buserror 5y agoOne absolutely essential companion to gdb is 'rr' [0], it's a 'post mortem' debugger, but unlike a core file, you can replay the action, in reverse if necessary. You can reverse-cont, reverse-step create watchpoints, run forward again etc etc. It's absolutely amazing! [0]: https://rr-project.org/ https://rr-project.org/
- mark_undoio 5y agoThe best thing about time travel debugging is that you effectively get an instruction-perfect reproducer for the problem you're looking at. Even if the problem only occurs one time in a thousand, if you have recorded it then you can review it in arbitrary detail as many times as you need. You can even send the recording to another developer for their feedback. Probably the second best thing about time travel is watchpoints[0] + reverse-continue[1] - combining these two features, you can answer "why is that value there?" incredibly quickly. Reverse execution + watchpoints is so powerful it can speed up the diagnosis of memory corruptions, race conditions, algorithmic bugs, etc from days/weeks to hours. [0] https://sourceware.org/gdb/onlinedocs/gdb/Set-Watchpoints.html#Set-Watchpoints https://sourceware.org/gdb/onlinedocs/gdb/Set-Watchpoints.ht... [1] https://sourceware.org/gdb/onlinedocs/gdb/Reverse-Execution.html#Reverse-Execution https://sourceware.org/gdb/onlinedocs/gdb/Reverse-Execution.... - GDB has an implementation of reverse debugging but larger projects like `rr` (https://rr-project.org/ https://rr-project.org/) or `udb` (https://undo.io/solutions/products/udb/ https://undo.io/solutions/products/udb/ - which, disclaimer, I work on) exist to do this with higher performance on complex, real-world programs. (edit: format links better)
- westurner 5y agoTime travel debugging https://en.wikipedia.org/wiki/Time_travel_debugging https://en.wikipedia.org/wiki/Time_travel_debugging : > Interactive debuggers include the ability to modify code and step forward based on updated information.[4] Reverse debugging tools allow users to step backwards in time through the steps that resulted in reaching a particular point in the program. Time traveling debuggers provide these features and also allow users to interact with the program, changing the history if desired, and watch how the program responds.[5] https://github.com/rr-debugger/rr https://github.com/rr-debugger/rr : > System requirements: Linux kernel ≥ 3.11 is required (for PTRACE_SETSIGMASK). > rr currently requires either: > An Intel CPU with Nehalem (2010) or later microarchitecture. [OR] Certain AMD Zen or later processors Is there an rr-like reverse time-travel debugging tool for ARM64/aarch64? Are GUIs like Voltron and Ghidra helpful for gdb and/or rr-like traces?
- bee_rider 5y agoI generally stick pretty close to the classic "old but still good" tools, but GDB is something I really could not get a handle on (probably because it isn't a daily-use tool for me).
- 0xbadcafebee 5y agoEven if the rest of GDB remains a mystery to you, you can learn the two simplest and most useful functions: 1) how to create a backtrace, and 2) how to create a core dump. That way you can always send a bug report with lots of detail to somebody else for analysis.
- jmclnx 5y ago> 2) how to create a core dump This is a PITA under Linux, I figured this out but nor happy I had to look. I wish Linux was like OpenBSD, where you get core files by default. This give me yet another reason to switch to OpenBSD, I only wish some other issues I am having with OpenBSD could get solved.
- mark_undoio 5y agoWhat did you figure out as a solution? I've previously configured `/proc/sys/kernel/core_pattern` to generate core dumps by default (and not pipe them to a core dump collection program, which may be useful for error reporting but is rarely what I want). There's also `gcore` for doing on-demand core dumps. I wish it were possible to configure core dump behaviour per-process (or something similar).
- jmclnx 5y agoThis is what I did: ulimit -S -c unlimited sudo sysctl -w kernel.core_pattern=core took me a while to figure this out, HTH
- mark_undoio 5y ago
- jokoon 5y agoHow difficult is it to setup an editor like sublime text with gdb? I'm curious if it can be an alternative to visual studio, meaning a window showing the stack, locals, watch, etc?
- jraph 5y agoI don't know if it is possible in Sublime text (I never tried it), but there are several GUI frontend to GDB, including Nemiver and KDbg, as well as some IDEs allowing to use GDB to debug, like KDevelop and Eclipse. I'd expect Emacs to have some GDB integration as well.
- mark_undoio 5y agoEmacs's GDB integration is probably my favourite (aside from using GDB directly): https://www.gnu.org/software/emacs/manual/html_node/emacs/GDB-Graphical-Interface.html https://www.gnu.org/software/emacs/manual/html_node/emacs/GD... It gives you an IDE-like interface but supports easy access to the command line (as you'd expect, given Emacs's focus on text-based UX). Unlike (some of) the full IDEs it doesn't require you to be fully committed to a particular way of doing things (build systems, projects, etc). I understand that VScode and CLion are also fairly lightweight IDEs and have GDB integration too, though ISTR VScode doesn't have watchpoint support yet. You can always type directly into the console, though.
- GuB-42 5y agoIt is possible, and it kinds of work but I didn't manage to get productive with it. It is far from Visual Studio or even QtCreator. I went back to the command line. Emacs has GDB integration too, but it has been a long time since I used Emacs. Again, I don't remember being very successful with it and went back to the command line. Edit: By Visual Studio, I mean the real one, not VS Code. VS Code is on the same level as Sublime Text, maybe a little bit better integrated. Among the different environments I have used, I consider Visual Studio to have the best C/C++ debugger, by far.
- lanstin 5y agoyeah, c with emacs and gdb works like a dream. Truly integrated deveopment experience. Scary langiage but great tooling. and valgrind works with them also, so your violating line pops up in the editor when it does a UMR or other memory error.
- Decabytes 5y agoI learned a lot of debugging using GDB from Greg Laws CPPCon talk https://www.youtube.com/watch?v=-n9Fkq1e6sg https://www.youtube.com/watch?v=-n9Fkq1e6sg. With that being said, one of the things I've heard is that debugging on Linux is a poorer experience than it is on Windows. I believe that RAD Game Tools tried to create a debugger for Linux that was better, but the project failed for some reason. I hope with the increased interest in Linux, GDB or another debugger can be improved enough to address the issues people have historically had with it
- Decabytes 5y agoI learned a lot of debugging using GDB from Greg Laws CPPCon talk https://www.youtube.com/watch?v=-n9Fkq1e6sg https://www.youtube.com/watch?v=-n9Fkq1e6sg. With that being said, one of the things I've heard is that debugging on Linux is a poorer experience than it is on Windows. I believe that RAD Game Tools tried to create a debugger for Linux that was better, but the project failed for some reason. I hope with the increased interest in Linux, GDB or another debugger can be improved enough to address the issues people have historically had with it
- edgyquant 5y agoInteresting, the last time GDB hit the front page the consensus seemed to be it was far ahead of all the competition. Having never seriously used a debugger it seems like it’s an opinion to me.
- Decabytes 5y agoI found the link http://www.radgametools.com/debug.htm http://www.radgametools.com/debug.htm. Though it looks like the project has been abandoned https://github.com/ValveSoftware/vogl https://github.com/ValveSoftware/vogl Taken from their FAQ on that page about "Why make a new source-level debugger?" > To put it bluntly, debugging on Linux is just really bad. We believe it is the biggest roadblock to great software for that platform. Don't get us wrong - debugging on Windows isn't great either. But the standard Visual Studio debugger is still much better than anything on Linux. > So we want to make Linux debugging as good as Windows debugging. And then we want to make both better - much better. There are so many better debugging tools that we can imagine having, and yet even today's best debuggers provide barely any functionality that wasn't in debuggers twenty years ago. > Also, we think there's a lot of value in having one great debugger that you can use on every platform. RAD develops software for roughly 18 platforms now. A big part of the friction of a new platform is dealing with different toolsets. > Compilers, linkers and runtimes are all hard to deal with, certainly, but you typically finish setting those up in a few days or weeks. The debuggers, on the other hand, are tools you have to use every day for as long as you develop on that platform! I guess it's just really hard to do this stuff
- billfruit 5y agoOn Emacs the GUD major mode, makes using GDB from the GUI quite convenient. Does VIM has anything similar?
- kmarc 5y agoYes. https://news.ycombinator.com/item?id=30760084 https://news.ycombinator.com/item?id=30760084 There is also vimspector, which is more advanced, a full blown debugging solution, using LSP and its lesser known cousin, the debugger adapter protocol. However, its quite hard to set up and success isn't guaranteed.
- adaszko 5y agoThis article goes into a little more detail which may be useful: https://www.dannyadam.com/blog/2019/05/debugging-in-vim/ https://www.dannyadam.com/blog/2019/05/debugging-in-vim/
- israrkhan 5y agoif you are using neovim, nvim-dap has support for debug adapters, which includes gdb in addition to several other languages (python, nodejs, java etc). Here [1] are available debug adapters. It is noteable that nvim-dap just provides support for debug adapters, if you are looking for nice UI, there are other plugins such as nvim-dap-ui [2]. In vim land, there is vimspector. I have not used vimspector myself, but from what I hear, it is slower than nvim-dap. [1] https://microsoft.github.io/debug-adapter-protocol/implementors/adapters/ https://microsoft.github.io/debug-adapter-protocol/implement... [2] https://github.com/rcarriga/nvim-dap-ui https://github.com/rcarriga/nvim-dap-ui
- Alekhine 5y agoAre there any easier to use C debuggers than GDB? All I really need to do is watch my variables change with each line.
- amir734jj 5y agoYou can use jetbrain clion an attach it to a binary. Or use even vscode.
- mariusor 5y agoIf you're on Windows you can use RemedyBG: https://remedybg.itch.io/remedybg https://remedybg.itch.io/remedybg
- j1elo 5y agoA nice UI on top of GDB might be all you need. I've used this with some success in the past: https://www.gdbgui.com/ https://www.gdbgui.com/
- zoomablemind 5y agoQtCreator has a very capable debug view. No need to have a Qt project, simply start a binary and step-in into it, or attach to a running process. Most of IDEs have a similar capability to be able to use debug view without setting up a project.
- lsllc 5y agoTry GDB Dashboard, it makes gdb much easier to use: https://github.com/cyrus-and/gdb-dashboard https://github.com/cyrus-and/gdb-dashboard There's also Voltron which works with both gdb and lldb (amongst others): https://github.com/snare/voltron https://github.com/snare/voltron
- harry8 5y agoEclipse-cdt is a reasonable option for this.
- pjmlp 5y agoVisual Studio, C++ Builder and XCode. But if you really only need to watch variables any GUI frontend to gdb like QtCreator or DDD will do.
- zebnyc 5y agoUnrelated but does anyone have tips on debugging async code in Nodejs. My work has a gigantic Nodejs monorepo and frankly I am struggling so pointers would be appreciated. There is dependency injection & layers of async / await abstraction. Not fun, when I model my code / unit test against a working sample and some method 25+ layers down does not work as expected as it does not have expected input when the stack frame does not even give me the caller. Is there a better way than peppering the codebase with logging statements?
- apnsngr 5y agoYou can use the debugger from Chrome dev tools to debug Node.js. Just run node with the `—-inspect-brk` flag, open `about://inspect` in Chrome, and select your app under remote targets. This has saved me tons of debugging time. See also: https://nodejs.org/en/docs/guides/debugging-getting-started/ https://nodejs.org/en/docs/guides/debugging-getting-started/
- havkom 5y agoHave not used gdb for a while but I think a good gdb tip is that you can evaluate functions with the print p command: > p myfunction()
- mark_undoio 5y agoGetting more advanced, you can generally do this anywhere GDB will accept a value. Which means you can do things like: > break my_file.c:123 if my_function() == 0x5 i.e. please stop at `my_file.c` line 123 but when `my_function()` is returning the value 0x5. GDB will handle calling the function and checking its value every time you get there.
- habibur 5y agoFor my 98% of use cases these 3 gdb commands suffices : bt, f, p.
- synergy20 5y agotwo more to add: b,l
- mark_undoio 5y agoDisclaimer: I work for undo.io (see my username) on time travel debugging. We use GDB as the front end but the engine underneath is proprietary. The GDB Documentation is really nice (as GNU project docs tend to be) and I've made lots of use of them. The fact that they've also documented things like the serial comms protocol is a level further still. Some related stuff we (Undo) have produced about GDB and might be interesting to the audience here... Articles about less-known parts of GDB functionality: https://undo.io/resources/gdb-watchpoint/ https://undo.io/resources/gdb-watchpoint/ Greg Law's quick talk on little-known features of GDB: https://undo.io/resources/cppcon-2015-greg-law-give-me-15-minutes-ill-change/ https://undo.io/resources/cppcon-2015-greg-law-give-me-15-mi... Greg's bigger talk on GDB: https://www.youtube.com/watch?v=-n9Fkq1e6sg https://www.youtube.com/watch?v=-n9Fkq1e6sg
- drmeister 5y agoThe GDB JIT interface implementation is seriously flawed. I love GDB - but this causes me a LOT of grief when debugging clasp (Common Lisp implemented using llvm as the backend https://github.com/clasp-developers/clasp.git https://github.com/clasp-developers/clasp.git). Every time a JITted object file is added using the API, the entire symbol table is sorted. If you have 10,000+ JITted object files as we do - it takes hours to days to weeks to register them all. We use the Undo time traveling debugger that builds on top of GDB. It's awesome but we are crippled because of the JIT API implementation. I'd love to see this get fixed - if anyone knows who to talk with about it - drop me a line.
- knome 5y agoHave you peeked into it? What's the implementation like in there? Anything you could replace with an incremental merge instead of a full resort each time?
- wiz21c 5y agoNot entirely related, but the other day I wanted to add 6502 CPU (old stuff) support to GDB. The code of GDB is a nightmare to understand. I don't say it's good or bad, I really can't judge. But adding a CPU is scary, at best... Even if there is a supportive community and documentation. Are there other debuggers out there that are easier to extend ?
- mark_undoio 5y agoWell, there's LLDB (https://lldb.llvm.org/ https://lldb.llvm.org/) - I've heard it's got some nifty architectural features (e.g. having access to the Clang framework for handling C/C++ expressions). I've done some minimal poking about in the code; I found its object-orientation a bit hard to grok (just for me personally) but it seemed to be quite uniformly applied so it might well be easier to work with.
- jcadam 5y agoEarly in my career I took a job at Boeing working on spacecraft simulation tools written in Ada. When I asked I was told by the other folks on the team that they had no ability to debug the system (i.e., they did debugging via print statements). For background, our dev environments consisted of Windows PCs which we used to ssh (using putty, exceed, etc) into a SPARC/Solaris machine to do all of our actual work. Our tools consisted of vi (not vim), clearcase, and whatever other command line tools were available (good luck asking for anything new to be installed). At some point, I guess somebody installed GNU tools on the solaris machine because I managed to find gdb while rooting around... which I'd never used before - so I set about learning. It proved to be a huge help. I tried to show it to some of the other engineers, but few seemed interested.
- gonza83 5y ago