28 ms·

The State of Linux Debuggers

237 points by pyb 6y ago

2bluesc 6y ago
I'm a big fan of cgdb; https://cgdb.github.io/ https://cgdb.github.io/ Works great for native and cross compiled apps
- rightbyte 6y ago
  That looks like magic. GDB should really add such functionality. It allready has dependency on curses. Edit: Oh 'gdb -tui' ... I should read the docs more ofent.
  - cipherboy 6y ago
    There's also shortcuts to enter and leave tui mode: https://sourceware.org/gdb/current/onlinedocs/gdb/TUI-Keys.html https://sourceware.org/gdb/current/onlinedocs/gdb/TUI-Keys.h... (I never start with gdb -tui but usually end up there for a little bit). As the author pointed out, this hijacks arrow keys, so you have to use readline editing: https://sourceware.org/gdb/current/onlinedocs/gdb/Readline-Bare-Essentials.html#Readline-Bare-Essentials https://sourceware.org/gdb/current/onlinedocs/gdb/Readline-B... https://sourceware.org/gdb/current/onlinedocs/gdb/Readline-Movement-Commands.html#Readline-Movement-Commands https://sourceware.org/gdb/current/onlinedocs/gdb/Readline-M... Fortunately, there's a lot of stuff that uses readline so it is all very portable.
- mkhnews 6y ago
  I agree, cgdb is great. I've added some small things I want to my fork of cgdb for navigating windows etc.
kaszanka 6y ago
Anyone have recommendations of graphical "reverse engineering debuggers"/binary-only debuggers (not sure about the name of this genre) like OllyDbg or x64dbg for Linux?
- praptak 6y ago
  Not sure if I understood what genre you have in mind. Does Ghidra fit the bill?
  - kaszanka 6y ago
    I meant a debugger that doesn't require or expect you to have the source code of the application you're debugging and just shows you disassembly, lets you use instruction execution/memory access breakpoints, patch instructions, etc. Ghidra doesn't have a debugger yet, but it's a planned feature.
  - enedil 6y ago
    Ghidra doesn't have a debugger (which afaik is a WIP).
- mvuksano 6y ago
  Try radare2. There is also GUI fronted called cutter (https://cutter.re/ https://cutter.re/).
- nekitamo 6y ago
  Evan’s debugger gets you most of the way there: https://github.com/eteran/edb-debugger https://github.com/eteran/edb-debugger You have to build from source but it’s easy to do. I’m thinking of devoting a summer to writing an Ollydbg-like UI wrapper for LLDB, because I miss Olly so bad when I leave Windows.
ris 6y ago
I'm currently using pwndbg, which is really just a few python enhancements to the gdb cli, mainly focused on developing exploits, hence the name. It's good enough for me, for now. Something I've realized about gdb in general is it gets a lot better once you figure out the python api and use it to ease your repetitive tasks.
billfruit 6y ago
Not exactly the same as other debuggers, but the "rr" debugger could be given a look.
- alexhutcheson 6y ago
  Highly recommend using rr if you're in an environment that supports it. Once you've learned to use it, it feels like a debugging superpower. You can use most of the gdb commands you are familiar with, and most of the gdb interfaces listed here will still work. The docs are here: https://rr-project.org/ https://rr-project.org/
  - db48x 6y ago
    Have you tried Pernosco yet? It's a meta-superpower; I use it almost exclusively now. https://pernos.co/ https://pernos.co/
    - alexhutcheson 6y ago
      No, that looks really slick, though.
- cipherboy 6y ago
  rr is nice when it works, but I've only had it work on a single system of mine (recent Intel). gdb is portable and works most anywhere. If you're looking at other architectures, you'll almost certainly have a gdb port but probably won't have a rr port. It looks like some changes have been made recently: - https://github.com/rr-debugger/rr/issues/1373 https://github.com/rr-debugger/rr/issues/1373 - https://github.com/rr-debugger/rr/issues/2034 https://github.com/rr-debugger/rr/issues/2034
  - khuey 6y ago
    Really modern AMD processors will work in rr. AArch64 support is pretty experimental still. But any Core or Xenon Intel processor you bought in the last 8 or 9 years should work. Also if you have trouble please file issues!
nemetroid 6y ago
I wonder what results the author would have had if they'd tried to debug a binary written in C or C++.
- chubot 6y ago
  Yeah I think that's really the problem. Also, they're using Nix. I haven't had problems getting source locations to work with gdb console, Eclipse, or CLion on Ubuntu 16.04, debugging C++ binaries I built with a shell script.
alexhutcheson 6y ago
On Emacs, I'd recommend using M-x gdb, rather than running gdb through GUD (which is M-x gud-gdb). Emacs has a much tighter integration with gdb than with other debuggers. The features and interface are documented here: https://www.gnu.org/software/emacs/manual/html_node/emacs/GDB-Graphical-Interface.html#GDB-Graphical-Interface https://www.gnu.org/software/emacs/manual/html_node/emacs/GD...
- evmar 6y ago
  After 10 years as a vim user, I switched to emacs in part for the gdb integration. (It turns out you can run emacs with vim keybindings and get the best of both worlds -- vi keybindings but a real programming language for scripting...)
  - xyproto 6y ago
    You also get the sluggishness of Emacs, though. Slow startup and syntax coloring, compared to ViM and NeoViM.
    - AsyncAwait 6y ago
      Daemon mode is your friend.
    - donaldihunter 6y ago
      Worth trying Emacs 28. The speedup is impressive. https://news.ycombinator.com/item?id=24117853 https://news.ycombinator.com/item?id=24117853
    - segfaultbuserr 6y ago
      > Slow startup Simply run Emacs in server mode - it's an operating system, so don't shut it down too often ;-)
- pksadiq 6y ago
  And it's much better when used with gdb-many-windows and speedbar
wazari972 6y ago
I'm personally a big fan of vanilla GDB, at least for plain C debugging. Disclaimer: I spent >5years on a PhD work searching how to extend GDB for parallel and embedded programming. I can't easily see the surrounding code I have Emacs on the side window, so I use GDB to see the immediately surrounding code, and Emacs to see the rest of the function. I do exactly the same for Python debbuging with PDB I have to manually request information rather than just glancing at the display I just print the variables I want to see, there are not so many of them I'm actually interested in while debugging I have to remember syntax rather than just clicking on things ...and in this case, it took me a few tries to correctly deref this pointer to an fixed-size array those are the tradeoff between flexibility and ease. You can access any memory locations, and cast it in many different ways, examine the assembly to precisely understand the next steps, examine the bits to precisely understand the storage (only endianness issues remained tricky to follow)
- FpUser 6y ago
  >"You can access any memory locations, and cast it in many different ways, examine the assembly to precisely understand the next steps" This is not the exclusive domain of GDB. I have all those features in various Windows based IDEs all working in friendly manner.
- jamii 6y ago
  > those are the tradeoff between flexibility and ease It isn't a required tradeoff. Most of the gdb frontends I tried display the local variables at all time, but also allow writing arbitrary gdb expressions if I want to. Having both is clearly better.
pushedx 6y ago
I think you would love pwndbg https://github.com/pwndbg/pwndbg https://github.com/pwndbg/pwndbg
- nhuck 6y ago
  I'll second pwndbg, but also recommend gef. Gef is a little more lightweight and optimized. It doesn't include the reverse engineering tools that pwndbg has, but it doesn't sound like you're using those anyway. For general debugging it should work the exact same way as pwndbg. https://github.com/hugsy/gef https://github.com/hugsy/gef
taviso 6y ago
I use vanilla gdb, the secret to the arrow-key problem the author had is they needed to switch window focus. The keybinding is Ctrl-X O, but if you don't want to remember that - and who could blame you - you can use the focus command, e.g. `focus cmd` will get you back to the command window, or `focus src` will get you back to the source window. The display isn't really buggy, it's just the debugee output messed up the terminal. You can redraw it with ^L, or disable the tui while it's running. You can toggle it with Ctrl-X A, or just `tui enable` or `tui disable`. I quite like the tui, the windows are configurable (use `show tui` and `show style`), and you can display more than just source, there's also registers and disassembly mode, e.g. `tui reg general`.
- tehnicaorg 6y ago
  Tavis, it would be great to see a screencast from you using gdb.
  - taviso 6y ago
    I don't think I have any of gdb, but I was just talking about a hobby project using frida! Frida is a tool that lets you inject v8 into any process, then exposes a javascript debugging api, I find it incredibly useful. https://frida.re/docs/javascript-api/ https://frida.re/docs/javascript-api/ https://twitter.com/taviso/status/1336693403510140929 https://twitter.com/taviso/status/1336693403510140929
    - haberman 6y ago
      Wow, that is very interesting. The docs say it is injecting QuickJS though, not v8? > Frida’s core is written in C and injects QuickJS into the target processes, where your JS gets executed with full access to memory, hooking functions and even calling native functions inside the process. https://frida.re/docs/home/ https://frida.re/docs/home/
      - bretthoerner 6y ago
        Might be a case of outdated documentation/website: https://frida.re/news/2019/05/15/frida-12-5-released/ https://frida.re/news/2019/05/15/frida-12-5-released/
db48x 6y ago
The rendering problems in curses programs could easily be the fault of his terminal emulator or even an incorrect value in his TERM variable.
- tehnicaorg 6y ago
  Is there an easy way to check if everything is set up properly and also to get tips on what needs to be changed?
  - db48x 6y ago
    Not really, but the most common mistake is to set TERM in your shell startup files (.bashrc, for example). If you do that, then you could end up setting it to a value not supported by your terminal emulator. If you don't set it, then your terminal emulator will set it to an appropriate value for you.
- xioxox 6y ago
  I thought one needed Ctrl+L to refresh the screen when it gets corrupted by program output in "gdb -tui".
  - db48x 6y ago
    That's absolutely true, I'd forgotten about that. It's hard to tell from the screenshot if that's what happened, but it's quite likely.
jiva 6y ago
I use gdb dashboard https://github.com/cyrus-and/gdb-dashboard https://github.com/cyrus-and/gdb-dashboard It makes for a very pleasant debugging experience https://i.imgur.com/iw21sbU.jpg https://i.imgur.com/iw21sbU.jpg
- ship_it 6y ago
  Off topic but can you share your .tmux.conf? Thats hot af.
  - jiva 6y ago
    Sure: https://gist.github.com/jiva/954a22bafcafec2d594b5c3454a10b03 https://gist.github.com/jiva/954a22bafcafec2d594b5c3454a10b0...
    - ship_it 6y ago
      Thanks a lot @jiva <3
- jamii 6y ago
  This is the best option I've tried so far. Thanks :)
ktpsns 6y ago
The article should be named "the state of C/C++ debuggers and gdb in particular on Linux". Because there are debuggers for virtually any software ecosystem (for instance Pdb for Python) which frequently work the same cross platform. Considering gdb/Linux: I love the Kdevelop GUI front-end for gdb. It allows introspection of variables in source code at mouse hovering. Of course it also can open coredumps. The Fortran gdb is also great for debugging arrays. If you don't mind OSS, check out https://totalview.io/ https://totalview.io/ -- it is an instance of a domain specific debugger for C/C++ in HPC (massively parallel computing).
- JoachimSchipper 6y ago
  Eh, more particularly "the state of Zig debuggers and gdb in particular on NixOS Linux" - the author isn't wrong to want things to work, of course, but the author isn't using the most boring environment either.
  - jamii 6y ago
    This is true but note that: * gdb works on zig code, so it isn't unreasonable to expect gdb frontends to also work * being on nixos might explain crashes or failing to start, but it's hard to imagine how it could be responsible for various gdb frontends not being able to set breakpoints, run to the correct line or render a pointer to an array of bytes, especially when those same features work fine in gdb * it also doesn't explain why the memory view in most frontends was buggy or didn't work at all * I also tried lldb, whose gui I couldn't operate at all, and code-lldb, whose breakage might actually be nix-related since I had to patch a binary (but the error message was complaining about invalid utf8...) I'm also happy to try non-gdb/lldb debuggers if you have any suggestions.
    - JoachimSchipper 6y ago
      Unfortunately, I don't really have a better suggestion. :-( I know Solaris has its own debugger(s?), and rr looks really useful, but I haven't worked on a platform that could run either.
pjmlp 6y ago
I just stick with ddd, QtCreator or KDevelop. https://www.gnu.org/software/ddd/ https://www.gnu.org/software/ddd/ https://doc.qt.io/qtcreator/creator-debugging.html https://doc.qt.io/qtcreator/creator-debugging.html https://docs.kde.org/trunk5/en/extragear-kdevelop/kdevelop/debugging-programs-in-kdevelop.html https://docs.kde.org/trunk5/en/extragear-kdevelop/kdevelop/d... Back in the old XEmacs days, I used the gdb integration, no idea how far Emacs has improved on it.
hpcjoe 6y ago
I've used ddd for debugging statically compiled code, C C++ and Fortran. It is quirky. Its weirder with MPI, as you get lots of debug windows. For dynamic languages, Perl Python and others, I use komodo from ActiveState. Been a customer of theirs since 2007 or so. Excellent tool, though with a few quirks as well. Julia has/had a native debugger, and an atom plugin, but seems to be moving towards vscode. Vscode is many things, and is highly opinionated. I've gotten the debugger to "work", albeit not being helpful for the Julia I was working on. I tried the C/Fortran plugins for it and was disappointed. I wasn't aware that totalview went OSS. I thought it was closed.
sriram_sun 6y ago
I keep going back to Vim + commandline gdb. However, QtCreator + CMake on a large project worked well enough to be productive.
thewebcount 6y ago
What's even more depressing to me is that this is the experience that most students get when they first learn debugging. It's no wonder that so many professional developers don't bother using the debugger when they've had such an awful experience using it.
- deleted 6y ago
  [deleted]
Asooka 6y ago
Yep, it's horrible. I use Vim 8's Termdebug. It just splits Vim in 3 windows - gdb console on top, program output in the middle, source code on the bottom. It does absolutely nothing fancy - you still type commands in the gdb window - but for me just having all 3 windows together is enough and it's super stable. In general any program that tries to do a fancy interface on Linux will be an absolutely miserable experience, there seems to be zero intersection between people who can write solid software and people who can design UI/UX in OSS and for some reason the two camps can't talk together. C'est la vie, I've just given up and use the console. This is not really a criticism, just an observation. I suspect it stems from the fact that both software design and interface design are very complicated disciplines, but most OSS projects are done by a single person scratching an itch, so you either have programmers designing interface, or designers programming software, and both don't work out well. I suspect if we had some GUI libraries that were as easy as dumping text to console, the programmers would be able to churn out passable interfaces without having to do design. Quite possibly Tcl/Tk is that library, based on how rough, but usable gitk and git gui are.
modeless 6y ago
It can't be a coincidence that literally every Linux C++ debugger majorly sucks. I think the blame has to lie ultimately with GDB itself and GDB/MI, because it's the only thing they all have in common (except for lldb I guess).
- makuto 6y ago
  I think that the major problem is that gdb was never designed/suitable for graphical debugging. I use Visual Studio on C projects at work, and found it very interesting to find that Windbg actually has similar terrible usability as gdb. Gdb and Windbg were both command line interfaces originally, whereas Visual Studio has it's own debugger backend which was visual-first (afaik). Really, VS is exceptional even in Windows land. I've tried RemedyDB, an attempted rewrite of debugging on windows, and found it useless. It had similar bugs to this article - no symbols, failure to attach, failed to bind breakpoints at all, failed to display variables. I think it would be a huge project to replace GDB with a graphical-oriented debugger, but it would be worthwhile. If I were in charge, I'd make it a C lib that didn't come with a frontend. That way, the user can stay in their familiar editor and link the lib. (most programs should be able to use C libraries).
  - pjmlp 6y ago
    Only if you use classical WinDgb instead of the new one. https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugging-using-windbg-preview https://docs.microsoft.com/en-us/windows-hardware/drivers/de... Mac OS also is another example of having good graphical debuggers, even in the MPW days.
    - makuto 6y ago
      I disagree. I was using WinDbg downloaded from the Microsoft Store (so it had the WPF or whatever UI), and it was unusable. (Note: I was using it on a very large project - 4mLOC with hundreds of first- and third-party libs; It's likely I could get some use out of it on a trivial program, but it just didn't get the "real" program; and no, I wasn't simply missing the PDBs).
vfclists 6y ago
How about this reddit comment? https://www.reddit.com/r/emacs/comments/6p872z/make_emacs_pay_what_you_want/dko3pae/ https://www.reddit.com/r/emacs/comments/6p872z/make_emacs_pa... Here it is for those who don't like reddit ======================== I think this approach by the programming profession towards tools that are critical to their occupation is somewhat ass-backwards. Their idea of professionals depending on critical tools which are created by other professionals in their spare time is wrong. It is time software engineers created guilds that fund the development of tools they need and use. It may be that software development is a new profession which is failing to get properly organized on account of the involvement of big corporations. But that is a throwback to an era when computers were expensive resources only big companies (with lashings of DARPA and NSF funding) could afford and thus were able to influence the tools and the outlook. Those days are long gone in an era where a powerful computer can be had for less than $200, and there is no need for professionals to allow corporate agendas to undermine the proper development of software tools, as it has been since the internet came around, and it is time software developers learned to see themselves as professionals and stop seeing their jobs as gigs as though they are part time semi-employed musicians or something. I mean it is 2017 and GCC has been available since the 1990s, but where is the graphical C++ debugger rivaling Visual Studio that we would have by now if developers belonged to a well organized profession which saw the need to fund the development of their software tools through membership dues just like any of the professional bodies out there? By that I mean the software developers themselves, wholly independent of any corporations out there, ie no Linux Foundation type f&ckery where BigCo come in and shape the direction to their agenda. The software could be Free GNU License software but the hard part would be done by paid professionals funded by guild membership fees.
- phendrenad2 6y ago
  It's an interesting idea, but like it or not, companies DO set the standards for corporate software development, and they unfortunately don't understand the value of wasted time. A company would rather tell their devs to use free tools like GCC than pay $x,xxx per head to give them something world-class like Visual Studio.
dahart 6y ago
> [Re: gdb -tui] Unfortunately, it's pretty buggy I feel like this article gave up way too fast on gdb's tui mode, especially given that gdb works and tui didn't crash like half the debuggers in the list, other than "the command tui disable crashes gdb." which I can't reproduce. The complaints about rendering and arrow keys are very easy to overlook with just a few tui mode hotkeys. ctrl-x-a toggles tui, ctrl-x-o toggles arrow-key focus between code and the gdb command line, and ctrl-l repaints whenever the rendering gets borked. Those three things cover all the concerns listed here. The single most important thing about gdb, however, is the .gdbinit file. Being able to script your debug session is hands down the best thing about gdb. Bugs that require several steps to repro can be completely automated in gdb. You have conditionals and looping available in the debug script, this is very powerful. I suspect a bunch of the debuggers listed don't really support such a workflow. Since I work in a CUDA environment often, it's also nice that cuda-gdb mirrors gdb, I can have scripting & tui mode for both.
- DoofusOfDeath 6y ago
  > I feel like this article gave up way too fast on gdb's tui mode, especially given that gdb works and tui didn't crash like half the debuggers in the list, other than "the command tui disable crashes gdb." which I can't reproduce. I've had the same conundrum for a various categories of development tools on Linux: For a particular category (e.g., remote debugging, Vim C++ autocomplete system, etc.), an initial search uncovers 3-10 software combinations that people have documented as working. And when I work through the list, trying to find one that works for me, I often hit problem after problem. It's hard to know if (a) that particular recipe is no longer workable, (b) there's something about my particular setup that will forever damn that approach, or (c) I just need to keep working through the problems until I finally get things working. So it's hard to know how long to bang away at a particular recipe to determine if it's (a), (b), or (c). The unfortunate result is I often run out of time for the investigation, and fall back to using Vim + gdb + ctags for developing on remote servers. Mostly I'm just surprised that Visual Studio in 1999 gave me a better debugging experience than I've been able to cobble together on Linux in 2020.
  - ensiferum 6y ago
jandrese 6y ago
I've always found bitter irony in that the buggiest applications are the debuggers. Just like the author of this article I've been annoyed at GDB's teletype oriented interface and wanted something better, only to find a lot of applications that are even more clumsy to use and crash more than the application I'm trying to fix. This goes back a long way too. xxgdb used to be reasonably stable but these days it seems to crash quite frequently. ddd seemed so awesome when I first discovered it, but it too failed spectacularly on the various occasions when I tried actually using it. I occasionally go back to see if they've fixed the bugs but thus far have not had any luck.
zwieback 6y ago
This is a great resource, hadn't heard of most of them. I still think Eclipse is fantastic although not something for a quickie session, for that I'd probably use command line or the simple curses GUI. I have to say, though, much as I enjoy working on and with Linux full-blown Visual Studio Enterprise is really in a league of its own but not really complaining that a similar debugger for C++ under Linux doesn't exist yet.
georgewsinger 6y ago
I recently discovered a Linux debugger & tool which allowed me to solve problems 10x faster than gdb: pernos.co (which is layered over Mozilla's rr time-tracking debugger). Pernosco's tool is described pretty well on their website (which is full of minimal fluff and marketing -- another good sign). But basically it allows you to view a program inside and out, forwards /and/ backwards, with zero replay lag. Everything from stack traces to variable displays (at any point in time in your code execution) is extremely easy to view and understand. The best part is the lightning fast search functionality (again: zero lag). On top of this: extraordinary customer service if anything breaks (in my experience, they fix bugs within 24 hours and are highly communicative). I'm not a paid Pernosco shill. I'm just so blown away by this product that I wanted to word-of-mouth spread it around because it's saved so much time for me.
- tom_ 6y ago
  Relevant recent(ish) HN discussion: https://news.ycombinator.com/item?id=25042827 https://news.ycombinator.com/item?id=25042827
- bogwog 6y ago
  That looks cool, but $20/month for 5 submissions is absurd. That's practically a product demo you have to pay for. Seems like the type of tool you'd only use on a really tough and rare bug, but I am not running into those consistently enough to justify a monthly payment. GDB's CLI might be ugly/annoying, but it's still usable and perfectly adequate. Plus it's a cloud service I need to trust with my code? This is a very tough sell. I might pay an annual fee for a local/self-hosted version though, kind of like a Jetbrains subscription.
  - jandrese 6y ago
    5 submissions a month means I can't just play around with it to figure out how it works. I'll have to learn its interface while also fighting some particularly difficult bug. Maybe it's a great product, I'll never know.
    - 6y ago
shocks 6y ago
I quite like gdb dashboard. https://github.com/cyrus-and/gdb-dashboard https://github.com/cyrus-and/gdb-dashboard
benibela 6y ago
It is bad with C/C++ Unfortunately, I use Pascal and there it is even worse GDB tries to parse Pascal syntax in print, but many expressions do not work. It is almost a mix of Pascal and C. The integration in the Lazarus IDE makes it even worse, when you cannot enter all expressions. And there are so many bugs. You get GDB crashing, Lazarus crashing, or GDB freezing the entire X server if it pauses the program in a bad moment. Then you need to restart the system or kill GDB from outside the X server.
- revanx_ 6y ago
  On linux it works fine, on windows it sucks but there are some workarounds to make it usable : https://wiki.freepascal.org/Debugger_Setup https://wiki.freepascal.org/Debugger_Setup
- tromey 6y ago
  No-one maintains the Pascal support in GDB. In fact, I think the test suite doesn't even run any more (at least for me, FPC seems totally broken). This is an area where a volunteer could make useful progress pretty easily.
jnurmine 6y ago
Have you tried https://github.com/cyrus-and/gdb-dashboard https://github.com/cyrus-and/gdb-dashboard ? Edit: just saw that several people mentioned this already. Good, it is a nice tool!
29athrowaway 6y ago
Evan's debugger (edb) is pretty good too. The Linux equivalent of OllyDbg.
snakedoctor 6y ago
What about https://en.wikipedia.org/wiki/Strace https://en.wikipedia.org/wiki/Strace or Geohot's timeless debugger Qira https://github.com/geohot/qira https://github.com/geohot/qira
- saagarjha 6y ago
  strace isn’t really a debugger per se.
trishume 6y ago
One I haven't seen anyone mention yet but also haven't tried myself is https://codeclap.handmade.network/ https://codeclap.handmade.network/ It's not free, but it looks like a high-quality cross-platform gdb and lldb frontend with an imgui UI.
- jamii 6y ago
  I'll add it to my list for a followup post.
zubspace 6y ago
I guess, writing debuggers is hard and if you need to support multiple environments or console debugging it's probably wise, as a programming language community, to simply make your users adopt print-statement- or gdb-debugging. But as a long-time Visual Studio developer (C# and C++) I am constantly amazed how bad that status quo is for other languages. I'm probably spoiled by all its features: * Conditional Breakpoints. * Move instruction pointer. * Jump between threads, freeze and thaw them. * Showing variable Values when hovering over them. * Inspecting (large) Lists and Dictionaries easily. * Changing variable values or execute methods while breaking. * Changing code and continue debugging. * Historical debugging (moving backward) I believe that someone who never experience debugging like this cannot possibly imagine how much you could improve debugging in other environments.
- taviso 6y ago
  I think those are all basic features that all major debuggers support, certainly gdb can do those including moving backwards (gdb calls it reverse debugging). - Conditional breakpoints, break main.c:26 if a > 6 - Move instruction pointer, jump main.c:84 - Jump between threads, thread 123 - etc, etc. I've barely used Visual Studio, but I'm very familiar with windbg which uses the same debugger engine with a different UI.
the_mitsuhiko 6y ago
The biggest issue with debugging on Linux is still the awful state of debug symbols. Linux famously lacks the equivalent of Microsoft’s symbol server and that changes almost everything to the worse.
- glandium 6y ago
  That's not true anymore, but it's not widespread yet. https://developers.redhat.com/blog/2019/10/14/introducing-debuginfod-the-elfutils-debuginfo-server/ https://developers.redhat.com/blog/2019/10/14/introducing-de...
- db48x 6y ago
  Agreed. Fedora/Centos/RHEL have debug packages though, so the symbols are available for most software that you've got on your machine. GDB prints out a message reminding you to install them, but doesn't actually do it for you. We also have PolicyKit, so in principle GDB could just ask for those packages to be installed. We should close the gap, and while we're at it we should make PolicyKit not ask for the root password when installing debug packages. Then it could be a seamless process, at least for software installed from the official packages.
- forrestthewoods 6y ago
  Symbol server is critical. I wish the whole debugging environment was even simpler though. Binaries. Debug Symbols. Source Code. I wish those three things could be trivially merged and split. For internal builds I want to deploy a single binary that contains symbols and source code. This way issues can be trivially debugged by anyone on any machine. For public builds I wish to split out symbols and source which can be hosted on a symbol server.
drosan 6y ago
> The text boxes for eg the memory window ... didn't consistently accept keypresses at all Old Motif/X apps follow "focus on mouse hover" principle, so you can modify text fields only when there is a mouse cursor on them.
- jamii 6y ago
  I can't believe this comment ended up dead - that was the correct answer. I had more success with ddd this time around.
voltagex_ 6y ago
There's also the paid VisualGDB and you could probably make Visual Studio work for debugging a remote application via SSH.
bmn__ 6y ago
The state of Jamie Blogger > The gui doesn't seem to have a way to open an executable. Activate the "Load Executable" button in the toolbar. > Using "Open Source Code" produces a file dialog that doesn't seem to have a way to paste paths. Paste in the location bar or in the file name field, or access the context menu of the location bar and activate the "Paste" menu entry. > Passing the test exe at the command line (kdbg ./test) open up an editor at start.S but still doesn't seem to have a way to run the exe. Activate the "Run" button in the toolbar. ---- Verdict: too fucking rushed that he doesn't use common sense and tries the obvious things.
- jamii 6y ago
  > Activate the "Load Executable" button in the toolbar. There is not a "Load Executable" button in my toolbar. https://imgur.com/a/qOTZgy3 https://imgur.com/a/qOTZgy3 I also mentioned that in the article. Maybe you read it in a rush. > Paste in the location bar or in the file name field, or access the context menu of the location bar and activate the "Paste" menu entry. The location bar does not accept text. It's just a dropdown. Pressing C-l doesn't replace it either, like it does in the gtk file selector. https://imgur.com/a/wbuKniU https://imgur.com/a/wbuKniU > Activate the "Run" button in the toolbar. There is no "Run" button in my toolbar. https://imgur.com/a/rUHMRam https://imgur.com/a/rUHMRam
  - bmn__ 6y ago
    https://i.imgur.com/9Y0JXkO.png https://i.imgur.com/9Y0JXkO.png https://www.kdbg.org/kdbgtotal.png https://www.kdbg.org/kdbgtotal.png
forrestthewoods 6y ago
Visual Studio still has the best the debugger. It’s not perfect or even close. But it’s 50x better than cmdline gdb. RemedyBG is a cool one-man side-project debugger. It’s crazy impressive for what it is. However it has never worked for me on any professional project. There’s too many little one-off issues that are hard blockers. I wish it was open source. https://remedybg.handmade.network/ https://remedybg.handmade.network/
jart 6y ago
Try a new debugger tui that I just built a few months ago: Blinkenlights: https://justine.storage.googleapis.com/blinkenlights/index.html https://justine.storage.googleapis.com/blinkenlights/index.h...
- jamii 6y ago
  Looks interesting. I'll add it to my list for the followup post.
MayeulC 6y ago
> Eclipse does not play well with my window manager. For sway, _JAVA_AWT_WM_NONREPARENTING=1 solved most of my issues with java apps.
ajarmst 6y ago
"The State of Linux Debuggers". This is ... not that. Perhaps "One inexperienced user on an uncustomized system briefly trying several default-configuration front-ends (some of which are apparently malconfigured or missing dependencies) to GDB, without resorting to any documentation whatsoever, and gauging them solely on layout and visual appeal." Pro-tip. When working with a new debugging environment "I haven't found a way to display the backtrace, list local variables or view memory" would be where most would look at documentation or seek advice, not move on to the next tool. That the author didn't is more of a comment on the reviewer than the product.
- kdtop 6y ago
  I feel this review is overly harsh. I felt the author was showing that many of the packages were not intuitive and were often buggy. He was showing me what I could expect when I might try them. When one expects one's users to RTFM, they've already lost.
  - ajarmst 6y ago
    That's possibly fair---I am by nature sometimes overly brusque. But I will say that debugging on Linux is hardly a neophyte's realm. Even properly using the debugger in purposefully less obtuse interfaces like Visual Studio is non-trivial and generally requires the assistance of walkthroughs, manuals, and colleagues/instructors.
  - jjav 6y ago
    > When one expects one's users to RTFM, they've already lost. A debugger is a somewhat advanced tool, studying the documentation is absolutely required. It's a CNC machine, not a screwdriver. (Agreed that gdb could use better documentation.)
- howtofly 6y ago
  Can't agree more. As a longtime user of Eclipse CDT on Ubuntu Desktop， I've not seen the kind of bugs mentioned in the article for more than five years.
harry8 6y ago
Nothing about the proprietary options, green hills etc. I recall green hills had reverse debugging about 15 years ago, 5+ before gdb. I never used it in anger. Anyone got a link to anything that compares things that aren't just gdb frontends or under-developed workalikes?
- jamii 6y ago
  I've been avoiding proprietary options because it's usually a big time investment to get them working on nixos, but I'm open to suggestions.
ghoward 6y ago
I use `gdb-dashboard` along with others, but mine is heavily modified. I wrote about mine at https://gavinhoward.com/2020/12/my-development-environment-and-how-i-got-there/ https://gavinhoward.com/2020/12/my-development-environment-a... .

237 points by pyb 6y ago

2bluesc 6y ago

I'm a big fan of cgdb; https://cgdb.github.io/ https://cgdb.github.io/ Works great for native and cross compiled apps

rightbyte 6y ago
That looks like magic. GDB should really add such functionality. It allready has dependency on curses. Edit: Oh 'gdb -tui' ... I should read the docs more ofent.
- cipherboy 6y ago
  There's also shortcuts to enter and leave tui mode: https://sourceware.org/gdb/current/onlinedocs/gdb/TUI-Keys.html https://sourceware.org/gdb/current/onlinedocs/gdb/TUI-Keys.h... (I never start with gdb -tui but usually end up there for a little bit). As the author pointed out, this hijacks arrow keys, so you have to use readline editing: https://sourceware.org/gdb/current/onlinedocs/gdb/Readline-Bare-Essentials.html#Readline-Bare-Essentials https://sourceware.org/gdb/current/onlinedocs/gdb/Readline-B... https://sourceware.org/gdb/current/onlinedocs/gdb/Readline-Movement-Commands.html#Readline-Movement-Commands https://sourceware.org/gdb/current/onlinedocs/gdb/Readline-M... Fortunately, there's a lot of stuff that uses readline so it is all very portable.
mkhnews 6y ago
I agree, cgdb is great. I've added some small things I want to my fork of cgdb for navigating windows etc.

kaszanka 6y ago

Anyone have recommendations of graphical "reverse engineering debuggers"/binary-only debuggers (not sure about the name of this genre) like OllyDbg or x64dbg for Linux?

praptak 6y ago
Not sure if I understood what genre you have in mind. Does Ghidra fit the bill?
- kaszanka 6y ago
  I meant a debugger that doesn't require or expect you to have the source code of the application you're debugging and just shows you disassembly, lets you use instruction execution/memory access breakpoints, patch instructions, etc. Ghidra doesn't have a debugger yet, but it's a planned feature.
- enedil 6y ago
  Ghidra doesn't have a debugger (which afaik is a WIP).
mvuksano 6y ago
Try radare2. There is also GUI fronted called cutter (https://cutter.re/ https://cutter.re/).
nekitamo 6y ago
Evan’s debugger gets you most of the way there: https://github.com/eteran/edb-debugger https://github.com/eteran/edb-debugger You have to build from source but it’s easy to do. I’m thinking of devoting a summer to writing an Ollydbg-like UI wrapper for LLDB, because I miss Olly so bad when I leave Windows.

ris 6y ago

I'm currently using pwndbg, which is really just a few python enhancements to the gdb cli, mainly focused on developing exploits, hence the name. It's good enough for me, for now. Something I've realized about gdb in general is it gets a lot better once you figure out the python api and use it to ease your repetitive tasks.

billfruit 6y ago

Not exactly the same as other debuggers, but the "rr" debugger could be given a look.

alexhutcheson 6y ago
Highly recommend using rr if you're in an environment that supports it. Once you've learned to use it, it feels like a debugging superpower. You can use most of the gdb commands you are familiar with, and most of the gdb interfaces listed here will still work. The docs are here: https://rr-project.org/ https://rr-project.org/
- db48x 6y ago
  Have you tried Pernosco yet? It's a meta-superpower; I use it almost exclusively now. https://pernos.co/ https://pernos.co/
  - alexhutcheson 6y ago
    No, that looks really slick, though.
cipherboy 6y ago
rr is nice when it works, but I've only had it work on a single system of mine (recent Intel). gdb is portable and works most anywhere. If you're looking at other architectures, you'll almost certainly have a gdb port but probably won't have a rr port. It looks like some changes have been made recently: - https://github.com/rr-debugger/rr/issues/1373 https://github.com/rr-debugger/rr/issues/1373 - https://github.com/rr-debugger/rr/issues/2034 https://github.com/rr-debugger/rr/issues/2034
- khuey 6y ago
  Really modern AMD processors will work in rr. AArch64 support is pretty experimental still. But any Core or Xenon Intel processor you bought in the last 8 or 9 years should work. Also if you have trouble please file issues!

nemetroid 6y ago

I wonder what results the author would have had if they'd tried to debug a binary written in C or C++.

chubot 6y ago
Yeah I think that's really the problem. Also, they're using Nix. I haven't had problems getting source locations to work with gdb console, Eclipse, or CLion on Ubuntu 16.04, debugging C++ binaries I built with a shell script.

alexhutcheson 6y ago

On Emacs, I'd recommend using M-x gdb, rather than running gdb through GUD (which is M-x gud-gdb). Emacs has a much tighter integration with gdb than with other debuggers. The features and interface are documented here: https://www.gnu.org/software/emacs/manual/html_node/emacs/GDB-Graphical-Interface.html#GDB-Graphical-Interface https://www.gnu.org/software/emacs/manual/html_node/emacs/GD...

evmar 6y ago
After 10 years as a vim user, I switched to emacs in part for the gdb integration. (It turns out you can run emacs with vim keybindings and get the best of both worlds -- vi keybindings but a real programming language for scripting...)
- xyproto 6y ago
  You also get the sluggishness of Emacs, though. Slow startup and syntax coloring, compared to ViM and NeoViM.
  - AsyncAwait 6y ago
    Daemon mode is your friend.
  - donaldihunter 6y ago
    Worth trying Emacs 28. The speedup is impressive. https://news.ycombinator.com/item?id=24117853 https://news.ycombinator.com/item?id=24117853
  - segfaultbuserr 6y ago
    > Slow startup Simply run Emacs in server mode - it's an operating system, so don't shut it down too often ;-)
pksadiq 6y ago
And it's much better when used with gdb-many-windows and speedbar

wazari972 6y ago

I'm personally a big fan of vanilla GDB, at least for plain C debugging. Disclaimer: I spent >5years on a PhD work searching how to extend GDB for parallel and embedded programming. I can't easily see the surrounding code I have Emacs on the side window, so I use GDB to see the immediately surrounding code, and Emacs to see the rest of the function. I do exactly the same for Python debbuging with PDB I have to manually request information rather than just glancing at the display I just print the variables I want to see, there are not so many of them I'm actually interested in while debugging I have to remember syntax rather than just clicking on things ...and in this case, it took me a few tries to correctly deref this pointer to an fixed-size array those are the tradeoff between flexibility and ease. You can access any memory locations, and cast it in many different ways, examine the assembly to precisely understand the next steps, examine the bits to precisely understand the storage (only endianness issues remained tricky to follow)

FpUser 6y ago
>"You can access any memory locations, and cast it in many different ways, examine the assembly to precisely understand the next steps" This is not the exclusive domain of GDB. I have all those features in various Windows based IDEs all working in friendly manner.
jamii 6y ago
> those are the tradeoff between flexibility and ease It isn't a required tradeoff. Most of the gdb frontends I tried display the local variables at all time, but also allow writing arbitrary gdb expressions if I want to. Having both is clearly better.

pushedx 6y ago

I think you would love pwndbg https://github.com/pwndbg/pwndbg https://github.com/pwndbg/pwndbg

nhuck 6y ago
I'll second pwndbg, but also recommend gef. Gef is a little more lightweight and optimized. It doesn't include the reverse engineering tools that pwndbg has, but it doesn't sound like you're using those anyway. For general debugging it should work the exact same way as pwndbg. https://github.com/hugsy/gef https://github.com/hugsy/gef

taviso 6y ago

I use vanilla gdb, the secret to the arrow-key problem the author had is they needed to switch window focus. The keybinding is Ctrl-X O, but if you don't want to remember that - and who could blame you - you can use the focus command, e.g. `focus cmd` will get you back to the command window, or `focus src` will get you back to the source window. The display isn't really buggy, it's just the debugee output messed up the terminal. You can redraw it with ^L, or disable the tui while it's running. You can toggle it with Ctrl-X A, or just `tui enable` or `tui disable`. I quite like the tui, the windows are configurable (use `show tui` and `show style`), and you can display more than just source, there's also registers and disassembly mode, e.g. `tui reg general`.

tehnicaorg 6y ago
Tavis, it would be great to see a screencast from you using gdb.
- taviso 6y ago
  I don't think I have any of gdb, but I was just talking about a hobby project using frida! Frida is a tool that lets you inject v8 into any process, then exposes a javascript debugging api, I find it incredibly useful. https://frida.re/docs/javascript-api/ https://frida.re/docs/javascript-api/ https://twitter.com/taviso/status/1336693403510140929 https://twitter.com/taviso/status/1336693403510140929
  - haberman 6y ago
    Wow, that is very interesting. The docs say it is injecting QuickJS though, not v8? > Frida’s core is written in C and injects QuickJS into the target processes, where your JS gets executed with full access to memory, hooking functions and even calling native functions inside the process. https://frida.re/docs/home/ https://frida.re/docs/home/
    - bretthoerner 6y ago
      Might be a case of outdated documentation/website: https://frida.re/news/2019/05/15/frida-12-5-released/ https://frida.re/news/2019/05/15/frida-12-5-released/

db48x 6y ago

The rendering problems in curses programs could easily be the fault of his terminal emulator or even an incorrect value in his TERM variable.

tehnicaorg 6y ago
Is there an easy way to check if everything is set up properly and also to get tips on what needs to be changed?
- db48x 6y ago
  Not really, but the most common mistake is to set TERM in your shell startup files (.bashrc, for example). If you do that, then you could end up setting it to a value not supported by your terminal emulator. If you don't set it, then your terminal emulator will set it to an appropriate value for you.
xioxox 6y ago
I thought one needed Ctrl+L to refresh the screen when it gets corrupted by program output in "gdb -tui".
- db48x 6y ago
  That's absolutely true, I'd forgotten about that. It's hard to tell from the screenshot if that's what happened, but it's quite likely.

jiva 6y ago

I use gdb dashboard https://github.com/cyrus-and/gdb-dashboard https://github.com/cyrus-and/gdb-dashboard It makes for a very pleasant debugging experience https://i.imgur.com/iw21sbU.jpg https://i.imgur.com/iw21sbU.jpg

ship_it 6y ago
Off topic but can you share your .tmux.conf? Thats hot af.
- jiva 6y ago
  Sure: https://gist.github.com/jiva/954a22bafcafec2d594b5c3454a10b03 https://gist.github.com/jiva/954a22bafcafec2d594b5c3454a10b0...
  - ship_it 6y ago
    Thanks a lot @jiva <3
jamii 6y ago
This is the best option I've tried so far. Thanks :)

ktpsns 6y ago

The article should be named "the state of C/C++ debuggers and gdb in particular on Linux". Because there are debuggers for virtually any software ecosystem (for instance Pdb for Python) which frequently work the same cross platform. Considering gdb/Linux: I love the Kdevelop GUI front-end for gdb. It allows introspection of variables in source code at mouse hovering. Of course it also can open coredumps. The Fortran gdb is also great for debugging arrays. If you don't mind OSS, check out https://totalview.io/ https://totalview.io/ -- it is an instance of a domain specific debugger for C/C++ in HPC (massively parallel computing).

JoachimSchipper 6y ago
Eh, more particularly "the state of Zig debuggers and gdb in particular on NixOS Linux" - the author isn't wrong to want things to work, of course, but the author isn't using the most boring environment either.
- jamii 6y ago
  This is true but note that: * gdb works on zig code, so it isn't unreasonable to expect gdb frontends to also work * being on nixos might explain crashes or failing to start, but it's hard to imagine how it could be responsible for various gdb frontends not being able to set breakpoints, run to the correct line or render a pointer to an array of bytes, especially when those same features work fine in gdb * it also doesn't explain why the memory view in most frontends was buggy or didn't work at all * I also tried lldb, whose gui I couldn't operate at all, and code-lldb, whose breakage might actually be nix-related since I had to patch a binary (but the error message was complaining about invalid utf8...) I'm also happy to try non-gdb/lldb debuggers if you have any suggestions.
  - JoachimSchipper 6y ago
    Unfortunately, I don't really have a better suggestion. :-( I know Solaris has its own debugger(s?), and rr looks really useful, but I haven't worked on a platform that could run either.

pjmlp 6y ago

I just stick with ddd, QtCreator or KDevelop. https://www.gnu.org/software/ddd/ https://www.gnu.org/software/ddd/ https://doc.qt.io/qtcreator/creator-debugging.html https://doc.qt.io/qtcreator/creator-debugging.html https://docs.kde.org/trunk5/en/extragear-kdevelop/kdevelop/debugging-programs-in-kdevelop.html https://docs.kde.org/trunk5/en/extragear-kdevelop/kdevelop/d... Back in the old XEmacs days, I used the gdb integration, no idea how far Emacs has improved on it.

hpcjoe 6y ago

I've used ddd for debugging statically compiled code, C C++ and Fortran. It is quirky. Its weirder with MPI, as you get lots of debug windows. For dynamic languages, Perl Python and others, I use komodo from ActiveState. Been a customer of theirs since 2007 or so. Excellent tool, though with a few quirks as well. Julia has/had a native debugger, and an atom plugin, but seems to be moving towards vscode. Vscode is many things, and is highly opinionated. I've gotten the debugger to "work", albeit not being helpful for the Julia I was working on. I tried the C/Fortran plugins for it and was disappointed. I wasn't aware that totalview went OSS. I thought it was closed.

sriram_sun 6y ago

I keep going back to Vim + commandline gdb. However, QtCreator + CMake on a large project worked well enough to be productive.

thewebcount 6y ago

What's even more depressing to me is that this is the experience that most students get when they first learn debugging. It's no wonder that so many professional developers don't bother using the debugger when they've had such an awful experience using it.

deleted 6y ago
[deleted]

Asooka 6y ago

Yep, it's horrible. I use Vim 8's Termdebug. It just splits Vim in 3 windows - gdb console on top, program output in the middle, source code on the bottom. It does absolutely nothing fancy - you still type commands in the gdb window - but for me just having all 3 windows together is enough and it's super stable. In general any program that tries to do a fancy interface on Linux will be an absolutely miserable experience, there seems to be zero intersection between people who can write solid software and people who can design UI/UX in OSS and for some reason the two camps can't talk together. C'est la vie, I've just given up and use the console. This is not really a criticism, just an observation. I suspect it stems from the fact that both software design and interface design are very complicated disciplines, but most OSS projects are done by a single person scratching an itch, so you either have programmers designing interface, or designers programming software, and both don't work out well. I suspect if we had some GUI libraries that were as easy as dumping text to console, the programmers would be able to churn out passable interfaces without having to do design. Quite possibly Tcl/Tk is that library, based on how rough, but usable gitk and git gui are.

modeless 6y ago

It can't be a coincidence that literally every Linux C++ debugger majorly sucks. I think the blame has to lie ultimately with GDB itself and GDB/MI, because it's the only thing they all have in common (except for lldb I guess).

makuto 6y ago
I think that the major problem is that gdb was never designed/suitable for graphical debugging. I use Visual Studio on C projects at work, and found it very interesting to find that Windbg actually has similar terrible usability as gdb. Gdb and Windbg were both command line interfaces originally, whereas Visual Studio has it's own debugger backend which was visual-first (afaik). Really, VS is exceptional even in Windows land. I've tried RemedyDB, an attempted rewrite of debugging on windows, and found it useless. It had similar bugs to this article - no symbols, failure to attach, failed to bind breakpoints at all, failed to display variables. I think it would be a huge project to replace GDB with a graphical-oriented debugger, but it would be worthwhile. If I were in charge, I'd make it a C lib that didn't come with a frontend. That way, the user can stay in their familiar editor and link the lib. (most programs should be able to use C libraries).
- pjmlp 6y ago
  Only if you use classical WinDgb instead of the new one. https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugging-using-windbg-preview https://docs.microsoft.com/en-us/windows-hardware/drivers/de... Mac OS also is another example of having good graphical debuggers, even in the MPW days.
  - makuto 6y ago
    I disagree. I was using WinDbg downloaded from the Microsoft Store (so it had the WPF or whatever UI), and it was unusable. (Note: I was using it on a very large project - 4mLOC with hundreds of first- and third-party libs; It's likely I could get some use out of it on a trivial program, but it just didn't get the "real" program; and no, I wasn't simply missing the PDBs).

vfclists 6y ago

How about this reddit comment? https://www.reddit.com/r/emacs/comments/6p872z/make_emacs_pay_what_you_want/dko3pae/ https://www.reddit.com/r/emacs/comments/6p872z/make_emacs_pa... Here it is for those who don't like reddit ======================== I think this approach by the programming profession towards tools that are critical to their occupation is somewhat ass-backwards. Their idea of professionals depending on critical tools which are created by other professionals in their spare time is wrong. It is time software engineers created guilds that fund the development of tools they need and use. It may be that software development is a new profession which is failing to get properly organized on account of the involvement of big corporations. But that is a throwback to an era when computers were expensive resources only big companies (with lashings of DARPA and NSF funding) could afford and thus were able to influence the tools and the outlook. Those days are long gone in an era where a powerful computer can be had for less than $200, and there is no need for professionals to allow corporate agendas to undermine the proper development of software tools, as it has been since the internet came around, and it is time software developers learned to see themselves as professionals and stop seeing their jobs as gigs as though they are part time semi-employed musicians or something. I mean it is 2017 and GCC has been available since the 1990s, but where is the graphical C++ debugger rivaling Visual Studio that we would have by now if developers belonged to a well organized profession which saw the need to fund the development of their software tools through membership dues just like any of the professional bodies out there? By that I mean the software developers themselves, wholly independent of any corporations out there, ie no Linux Foundation type f&ckery where BigCo come in and shape the direction to their agenda. The software could be Free GNU License software but the hard part would be done by paid professionals funded by guild membership fees.

phendrenad2 6y ago
It's an interesting idea, but like it or not, companies DO set the standards for corporate software development, and they unfortunately don't understand the value of wasted time. A company would rather tell their devs to use free tools like GCC than pay $x,xxx per head to give them something world-class like Visual Studio.

dahart 6y ago

> [Re: gdb -tui] Unfortunately, it's pretty buggy I feel like this article gave up way too fast on gdb's tui mode, especially given that gdb works and tui didn't crash like half the debuggers in the list, other than "the command tui disable crashes gdb." which I can't reproduce. The complaints about rendering and arrow keys are very easy to overlook with just a few tui mode hotkeys. ctrl-x-a toggles tui, ctrl-x-o toggles arrow-key focus between code and the gdb command line, and ctrl-l repaints whenever the rendering gets borked. Those three things cover all the concerns listed here. The single most important thing about gdb, however, is the .gdbinit file. Being able to script your debug session is hands down the best thing about gdb. Bugs that require several steps to repro can be completely automated in gdb. You have conditionals and looping available in the debug script, this is very powerful. I suspect a bunch of the debuggers listed don't really support such a workflow. Since I work in a CUDA environment often, it's also nice that cuda-gdb mirrors gdb, I can have scripting & tui mode for both.

DoofusOfDeath 6y ago
> I feel like this article gave up way too fast on gdb's tui mode, especially given that gdb works and tui didn't crash like half the debuggers in the list, other than "the command tui disable crashes gdb." which I can't reproduce. I've had the same conundrum for a various categories of development tools on Linux: For a particular category (e.g., remote debugging, Vim C++ autocomplete system, etc.), an initial search uncovers 3-10 software combinations that people have documented as working. And when I work through the list, trying to find one that works for me, I often hit problem after problem. It's hard to know if (a) that particular recipe is no longer workable, (b) there's something about my particular setup that will forever damn that approach, or (c) I just need to keep working through the problems until I finally get things working. So it's hard to know how long to bang away at a particular recipe to determine if it's (a), (b), or (c). The unfortunate result is I often run out of time for the investigation, and fall back to using Vim + gdb + ctags for developing on remote servers. Mostly I'm just surprised that Visual Studio in 1999 gave me a better debugging experience than I've been able to cobble together on Linux in 2020.
- ensiferum 6y ago

jandrese 6y ago

I've always found bitter irony in that the buggiest applications are the debuggers. Just like the author of this article I've been annoyed at GDB's teletype oriented interface and wanted something better, only to find a lot of applications that are even more clumsy to use and crash more than the application I'm trying to fix. This goes back a long way too. xxgdb used to be reasonably stable but these days it seems to crash quite frequently. ddd seemed so awesome when I first discovered it, but it too failed spectacularly on the various occasions when I tried actually using it. I occasionally go back to see if they've fixed the bugs but thus far have not had any luck.

zwieback 6y ago

This is a great resource, hadn't heard of most of them. I still think Eclipse is fantastic although not something for a quickie session, for that I'd probably use command line or the simple curses GUI. I have to say, though, much as I enjoy working on and with Linux full-blown Visual Studio Enterprise is really in a league of its own but not really complaining that a similar debugger for C++ under Linux doesn't exist yet.

georgewsinger 6y ago

I recently discovered a Linux debugger & tool which allowed me to solve problems 10x faster than gdb: pernos.co (which is layered over Mozilla's rr time-tracking debugger). Pernosco's tool is described pretty well on their website (which is full of minimal fluff and marketing -- another good sign). But basically it allows you to view a program inside and out, forwards /and/ backwards, with zero replay lag. Everything from stack traces to variable displays (at any point in time in your code execution) is extremely easy to view and understand. The best part is the lightning fast search functionality (again: zero lag). On top of this: extraordinary customer service if anything breaks (in my experience, they fix bugs within 24 hours and are highly communicative). I'm not a paid Pernosco shill. I'm just so blown away by this product that I wanted to word-of-mouth spread it around because it's saved so much time for me.

tom_ 6y ago
Relevant recent(ish) HN discussion: https://news.ycombinator.com/item?id=25042827 https://news.ycombinator.com/item?id=25042827
bogwog 6y ago
That looks cool, but $20/month for 5 submissions is absurd. That's practically a product demo you have to pay for. Seems like the type of tool you'd only use on a really tough and rare bug, but I am not running into those consistently enough to justify a monthly payment. GDB's CLI might be ugly/annoying, but it's still usable and perfectly adequate. Plus it's a cloud service I need to trust with my code? This is a very tough sell. I might pay an annual fee for a local/self-hosted version though, kind of like a Jetbrains subscription.
- jandrese 6y ago
  5 submissions a month means I can't just play around with it to figure out how it works. I'll have to learn its interface while also fighting some particularly difficult bug. Maybe it's a great product, I'll never know.
  - 6y ago

shocks 6y ago

I quite like gdb dashboard. https://github.com/cyrus-and/gdb-dashboard https://github.com/cyrus-and/gdb-dashboard

benibela 6y ago

It is bad with C/C++ Unfortunately, I use Pascal and there it is even worse GDB tries to parse Pascal syntax in print, but many expressions do not work. It is almost a mix of Pascal and C. The integration in the Lazarus IDE makes it even worse, when you cannot enter all expressions. And there are so many bugs. You get GDB crashing, Lazarus crashing, or GDB freezing the entire X server if it pauses the program in a bad moment. Then you need to restart the system or kill GDB from outside the X server.

revanx_ 6y ago
On linux it works fine, on windows it sucks but there are some workarounds to make it usable : https://wiki.freepascal.org/Debugger_Setup https://wiki.freepascal.org/Debugger_Setup
tromey 6y ago
No-one maintains the Pascal support in GDB. In fact, I think the test suite doesn't even run any more (at least for me, FPC seems totally broken). This is an area where a volunteer could make useful progress pretty easily.

jnurmine 6y ago

Have you tried https://github.com/cyrus-and/gdb-dashboard https://github.com/cyrus-and/gdb-dashboard ? Edit: just saw that several people mentioned this already. Good, it is a nice tool!

29athrowaway 6y ago

Evan's debugger (edb) is pretty good too. The Linux equivalent of OllyDbg.

snakedoctor 6y ago

What about https://en.wikipedia.org/wiki/Strace https://en.wikipedia.org/wiki/Strace or Geohot's timeless debugger Qira https://github.com/geohot/qira https://github.com/geohot/qira

saagarjha 6y ago
strace isn’t really a debugger per se.

trishume 6y ago

One I haven't seen anyone mention yet but also haven't tried myself is https://codeclap.handmade.network/ https://codeclap.handmade.network/ It's not free, but it looks like a high-quality cross-platform gdb and lldb frontend with an imgui UI.

jamii 6y ago
I'll add it to my list for a followup post.

zubspace 6y ago

I guess, writing debuggers is hard and if you need to support multiple environments or console debugging it's probably wise, as a programming language community, to simply make your users adopt print-statement- or gdb-debugging. But as a long-time Visual Studio developer (C# and C++) I am constantly amazed how bad that status quo is for other languages. I'm probably spoiled by all its features: * Conditional Breakpoints. * Move instruction pointer. * Jump between threads, freeze and thaw them. * Showing variable Values when hovering over them. * Inspecting (large) Lists and Dictionaries easily. * Changing variable values or execute methods while breaking. * Changing code and continue debugging. * Historical debugging (moving backward) I believe that someone who never experience debugging like this cannot possibly imagine how much you could improve debugging in other environments.

taviso 6y ago
I think those are all basic features that all major debuggers support, certainly gdb can do those including moving backwards (gdb calls it reverse debugging). - Conditional breakpoints, break main.c:26 if a > 6 - Move instruction pointer, jump main.c:84 - Jump between threads, thread 123 - etc, etc. I've barely used Visual Studio, but I'm very familiar with windbg which uses the same debugger engine with a different UI.

the_mitsuhiko 6y ago

The biggest issue with debugging on Linux is still the awful state of debug symbols. Linux famously lacks the equivalent of Microsoft’s symbol server and that changes almost everything to the worse.

glandium 6y ago
That's not true anymore, but it's not widespread yet. https://developers.redhat.com/blog/2019/10/14/introducing-debuginfod-the-elfutils-debuginfo-server/ https://developers.redhat.com/blog/2019/10/14/introducing-de...
db48x 6y ago
Agreed. Fedora/Centos/RHEL have debug packages though, so the symbols are available for most software that you've got on your machine. GDB prints out a message reminding you to install them, but doesn't actually do it for you. We also have PolicyKit, so in principle GDB could just ask for those packages to be installed. We should close the gap, and while we're at it we should make PolicyKit not ask for the root password when installing debug packages. Then it could be a seamless process, at least for software installed from the official packages.
forrestthewoods 6y ago
Symbol server is critical. I wish the whole debugging environment was even simpler though. Binaries. Debug Symbols. Source Code. I wish those three things could be trivially merged and split. For internal builds I want to deploy a single binary that contains symbols and source code. This way issues can be trivially debugged by anyone on any machine. For public builds I wish to split out symbols and source which can be hosted on a symbol server.

drosan 6y ago

> The text boxes for eg the memory window ... didn't consistently accept keypresses at all Old Motif/X apps follow "focus on mouse hover" principle, so you can modify text fields only when there is a mouse cursor on them.

jamii 6y ago
I can't believe this comment ended up dead - that was the correct answer. I had more success with ddd this time around.

voltagex_ 6y ago

There's also the paid VisualGDB and you could probably make Visual Studio work for debugging a remote application via SSH.

bmn__ 6y ago

The state of Jamie Blogger > The gui doesn't seem to have a way to open an executable. Activate the "Load Executable" button in the toolbar. > Using "Open Source Code" produces a file dialog that doesn't seem to have a way to paste paths. Paste in the location bar or in the file name field, or access the context menu of the location bar and activate the "Paste" menu entry. > Passing the test exe at the command line (kdbg ./test) open up an editor at start.S but still doesn't seem to have a way to run the exe. Activate the "Run" button in the toolbar. ---- Verdict: too fucking rushed that he doesn't use common sense and tries the obvious things.

jamii 6y ago
> Activate the "Load Executable" button in the toolbar. There is not a "Load Executable" button in my toolbar. https://imgur.com/a/qOTZgy3 https://imgur.com/a/qOTZgy3 I also mentioned that in the article. Maybe you read it in a rush. > Paste in the location bar or in the file name field, or access the context menu of the location bar and activate the "Paste" menu entry. The location bar does not accept text. It's just a dropdown. Pressing C-l doesn't replace it either, like it does in the gtk file selector. https://imgur.com/a/wbuKniU https://imgur.com/a/wbuKniU > Activate the "Run" button in the toolbar. There is no "Run" button in my toolbar. https://imgur.com/a/rUHMRam https://imgur.com/a/rUHMRam
- bmn__ 6y ago
  https://i.imgur.com/9Y0JXkO.png https://i.imgur.com/9Y0JXkO.png https://www.kdbg.org/kdbgtotal.png https://www.kdbg.org/kdbgtotal.png

forrestthewoods 6y ago

Visual Studio still has the best the debugger. It’s not perfect or even close. But it’s 50x better than cmdline gdb. RemedyBG is a cool one-man side-project debugger. It’s crazy impressive for what it is. However it has never worked for me on any professional project. There’s too many little one-off issues that are hard blockers. I wish it was open source. https://remedybg.handmade.network/ https://remedybg.handmade.network/

jart 6y ago

Try a new debugger tui that I just built a few months ago: Blinkenlights: https://justine.storage.googleapis.com/blinkenlights/index.html https://justine.storage.googleapis.com/blinkenlights/index.h...

jamii 6y ago
Looks interesting. I'll add it to my list for the followup post.

MayeulC 6y ago

> Eclipse does not play well with my window manager. For sway, _JAVA_AWT_WM_NONREPARENTING=1 solved most of my issues with java apps.

ajarmst 6y ago

"The State of Linux Debuggers". This is ... not that. Perhaps "One inexperienced user on an uncustomized system briefly trying several default-configuration front-ends (some of which are apparently malconfigured or missing dependencies) to GDB, without resorting to any documentation whatsoever, and gauging them solely on layout and visual appeal." Pro-tip. When working with a new debugging environment "I haven't found a way to display the backtrace, list local variables or view memory" would be where most would look at documentation or seek advice, not move on to the next tool. That the author didn't is more of a comment on the reviewer than the product.

kdtop 6y ago
I feel this review is overly harsh. I felt the author was showing that many of the packages were not intuitive and were often buggy. He was showing me what I could expect when I might try them. When one expects one's users to RTFM, they've already lost.
- ajarmst 6y ago
  That's possibly fair---I am by nature sometimes overly brusque. But I will say that debugging on Linux is hardly a neophyte's realm. Even properly using the debugger in purposefully less obtuse interfaces like Visual Studio is non-trivial and generally requires the assistance of walkthroughs, manuals, and colleagues/instructors.
- jjav 6y ago
  > When one expects one's users to RTFM, they've already lost. A debugger is a somewhat advanced tool, studying the documentation is absolutely required. It's a CNC machine, not a screwdriver. (Agreed that gdb could use better documentation.)
howtofly 6y ago
Can't agree more. As a longtime user of Eclipse CDT on Ubuntu Desktop， I've not seen the kind of bugs mentioned in the article for more than five years.

harry8 6y ago

Nothing about the proprietary options, green hills etc. I recall green hills had reverse debugging about 15 years ago, 5+ before gdb. I never used it in anger. Anyone got a link to anything that compares things that aren't just gdb frontends or under-developed workalikes?

jamii 6y ago
I've been avoiding proprietary options because it's usually a big time investment to get them working on nixos, but I'm open to suggestions.

ghoward 6y ago

I use `gdb-dashboard` along with others, but mine is heavily modified. I wrote about mine at https://gavinhoward.com/2020/12/my-development-environment-and-how-i-got-there/ https://gavinhoward.com/2020/12/my-development-environment-a... .