3 ms·

> Normally, debuing the compiler is fairly straightforward: it is more or less a run of the mill executable. > In the bootstrap process, the entire thing becom

> Normally, debuing the compiler is fairly straightforward: it is more or less a run of the mill executable. > In the bootstrap process, the entire thing becomes way more complex. You see, rustc is not invoked directly. The bootstrap script calls a wrapper around the compiler. > Running that wrapped rustc is not easy to run either: it requires a whole lot of complex, environment flags to be set. > All that is to say: I don’t know how to debug the Rust compiler. I am 99.9 % sure there is an easy way to do this, documented somewhere I did not think to look. After I post this, somebody will tell me "oh, you just need to do X". > Still, at the time of writing, I did not know how to do this. > So, can we attach gdb to the running process? Nope, it crashes way to quickly for that. It's kind of funny how often this problem crops up and the variety of tricks I have in my back to deal with it. Sometimes I patch the script to invoke gdb --args [the original command] instead, but this is only really worthwhile if it's a simple shell script and also I can track where stdin/stdout are going. Otherwise I might patch the code to sleep a bit before actually running anything to give me a chance to attach GDB. On some platforms you can get notified of process execs and sometimes even intercept that (e.g. as an EDR solution) and sometimes I will use that to suspend the process before it gets a chance to launch. But I kind of wish there was a better way to do this in general…LLDB has a "wait for launch" flag but it just spins in a loop waiting for new processes and it can't catch anything that dies too early.

o11c 1y ago
Other ideas: * Run the whole tree of processes under `gdb` with `set detach-on-fork off`. * LD_PRELOAD a library that inserts the sleeps for you, maybe on startup or maybe on signal/exit. Ideally, we'd have some kind of infrastructure to name and identify particular processes recursively.
jcranmer 1y ago
I have a LD_PRELOAD library that hooks SIGSEGV into spawning gdb on the process using the best guess for the process's terminal (which currently isn't very smart because I haven't yet needed to debug processes that do a lot of stdio redirection).
- dezgeg 1y ago
  Do you have this publicly available? Sounds useful!
  - theMMaI 1y ago
    https://github.com/stass/libsegfault https://github.com/stass/libsegfault does something similar if not the same
CJefferson 1y ago
I agree, recently I was working with a large Java program and after spending about 90 minutes (far too long, I was getting obsessed), I just gave up trying to get it into a debugger. This is one area where rust disappoints me, there isn’t a “cargo debug” built in (there is an external program but it doesn’t work well), and when I just manually attach gdb most of the symbols are usually missing. I would seriously consider a language billed as “debugger-first”, just to see what the experience was like.
- dmitrygr 1y ago
  > I would seriously consider a language billed as “debugger-first”, just to see what the experience was like. Use C, tell gdb "-O0 -g -ggdb3"
  - CJefferson 1y ago
    You are probably fairly accurate there, but that would require me going back and writing more C, which I've try to avoid nowadays :) I still sometimes get caught out getting debug information all in the right place, particularly when someone is using ninja or some such, I've even ended up wrapping gcc just to strip optimisation options and add -g, rather than figure out how to fight some very complex build system mess.
- gavinray 1y ago
  For Java programs, you run the JAR with a flag telling it to launch a remote-debug server listening on some port (traditionally 5005): java -jar -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 If your app is a gradle app, the flag is "--debug-jvm" Then you connect to it with "jdb"
  - CJefferson 1y ago
    Thanks, I'll try that if I'm ever Java debugging again! I don't think I want to do it myself, but I'd love a "101 ways to get into a debugger" webpage.
touisteur 1y ago
Now wondering whether the author might be able to force a core dump. With recent snapshot abilities, on Modern Intel processors one can get a Processor Trace that can be helpful even without getting an actual interactive debugging session (haven't done one of those in a while as snapshots seem enough for my needs these days).
timhh 1y ago
I have a C library (I've also done a Python one in the past) that you load into the executable you want to debug. It activated based on an environment variable so normally I just permanently link it. When it is loaded it will automatically talk to VSCode and tell it to start a debugger and attach to it & it waits for the debugger to attach. End result is you just have to run your script with an environment variable set and it will automatically attach a nice GUI debugger to the process no matter how deeply buried in scripts and Makefiles it is. https://github.com/Timmmm/autodebug https://github.com/Timmmm/autodebug I currently use this for debugging C++ libraries that are dynamically loaded into Questa (a commercial SystemVerilog simulator) that is started by a Python script running in some custom build system. In the past I used it to debug Python code running in an interpreter launched by a C library loaded by Questa started by a Makefile started by a different Python interpreter that was launched by another Makefile. Yeah. It wasn't the only reason by a long shot but that company did not survive...
izacus 1y ago
I find it outright incredible how much software is built in a way that outright prevents debugging and observabiliy of what's going on in it (no hooks, no logging, no error messages, etc.). I have no idea how people fix bugs there outside vibing.
- mystified5016 1y ago
  The old fashioned way: elbow grease and lots of squinting and swearing at your computer
AndrewDucker 1y ago
This is where I really appreciate the .Net command System.Diagnostics.Debugger.Launch(); Which pops up a window asking you to select what debugger you want to use, and then opens your application in it.
mark_undoio 1y ago
Process recording by time travel debug seems like a good fit for this problem - then you can capture 100% of process execution and then go back and investigate further. We (Undo.io) came up with a technique for following a tree of processes and initiating process recording based on a glob of program name. It's the `--record-on` flag in https://docs.undo.io/UsingTheLiveRecorderTool.html https://docs.undo.io/UsingTheLiveRecorderTool.html. You can grab a free trial from our website. For open source, with rr (https://rr-project.org/ https://rr-project.org/) I think you'd just `rr record` the initial process and you'll end up capturing the whole process tree - then you can look at the one you're interested in. As others have said you could also do some smart things with GDB's follow-fork settings but I think process recording is ideal for capturing complicated situations like this as you can go and review what happened later on.

o11c 1y ago

Other ideas: * Run the whole tree of processes under `gdb` with `set detach-on-fork off`. * LD_PRELOAD a library that inserts the sleeps for you, maybe on startup or maybe on signal/exit. Ideally, we'd have some kind of infrastructure to name and identify particular processes recursively.

jcranmer 1y ago

I have a LD_PRELOAD library that hooks SIGSEGV into spawning gdb on the process using the best guess for the process's terminal (which currently isn't very smart because I haven't yet needed to debug processes that do a lot of stdio redirection).

dezgeg 1y ago
Do you have this publicly available? Sounds useful!
- theMMaI 1y ago
  https://github.com/stass/libsegfault https://github.com/stass/libsegfault does something similar if not the same

CJefferson 1y ago

I agree, recently I was working with a large Java program and after spending about 90 minutes (far too long, I was getting obsessed), I just gave up trying to get it into a debugger. This is one area where rust disappoints me, there isn’t a “cargo debug” built in (there is an external program but it doesn’t work well), and when I just manually attach gdb most of the symbols are usually missing. I would seriously consider a language billed as “debugger-first”, just to see what the experience was like.

dmitrygr 1y ago
> I would seriously consider a language billed as “debugger-first”, just to see what the experience was like. Use C, tell gdb "-O0 -g -ggdb3"
- CJefferson 1y ago
  You are probably fairly accurate there, but that would require me going back and writing more C, which I've try to avoid nowadays :) I still sometimes get caught out getting debug information all in the right place, particularly when someone is using ninja or some such, I've even ended up wrapping gcc just to strip optimisation options and add -g, rather than figure out how to fight some very complex build system mess.
gavinray 1y ago
For Java programs, you run the JAR with a flag telling it to launch a remote-debug server listening on some port (traditionally 5005): java -jar -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 If your app is a gradle app, the flag is "--debug-jvm" Then you connect to it with "jdb"
- CJefferson 1y ago
  Thanks, I'll try that if I'm ever Java debugging again! I don't think I want to do it myself, but I'd love a "101 ways to get into a debugger" webpage.

touisteur 1y ago

Now wondering whether the author might be able to force a core dump. With recent snapshot abilities, on Modern Intel processors one can get a Processor Trace that can be helpful even without getting an actual interactive debugging session (haven't done one of those in a while as snapshots seem enough for my needs these days).

timhh 1y ago

I have a C library (I've also done a Python one in the past) that you load into the executable you want to debug. It activated based on an environment variable so normally I just permanently link it. When it is loaded it will automatically talk to VSCode and tell it to start a debugger and attach to it & it waits for the debugger to attach. End result is you just have to run your script with an environment variable set and it will automatically attach a nice GUI debugger to the process no matter how deeply buried in scripts and Makefiles it is. https://github.com/Timmmm/autodebug https://github.com/Timmmm/autodebug I currently use this for debugging C++ libraries that are dynamically loaded into Questa (a commercial SystemVerilog simulator) that is started by a Python script running in some custom build system. In the past I used it to debug Python code running in an interpreter launched by a C library loaded by Questa started by a Makefile started by a different Python interpreter that was launched by another Makefile. Yeah. It wasn't the only reason by a long shot but that company did not survive...

izacus 1y ago

I find it outright incredible how much software is built in a way that outright prevents debugging and observabiliy of what's going on in it (no hooks, no logging, no error messages, etc.). I have no idea how people fix bugs there outside vibing.

mystified5016 1y ago
The old fashioned way: elbow grease and lots of squinting and swearing at your computer

AndrewDucker 1y ago

This is where I really appreciate the .Net command System.Diagnostics.Debugger.Launch(); Which pops up a window asking you to select what debugger you want to use, and then opens your application in it.

mark_undoio 1y ago

Process recording by time travel debug seems like a good fit for this problem - then you can capture 100% of process execution and then go back and investigate further. We (Undo.io) came up with a technique for following a tree of processes and initiating process recording based on a glob of program name. It's the `--record-on` flag in https://docs.undo.io/UsingTheLiveRecorderTool.html https://docs.undo.io/UsingTheLiveRecorderTool.html. You can grab a free trial from our website. For open source, with rr (https://rr-project.org/ https://rr-project.org/) I think you'd just `rr record` the initial process and you'll end up capturing the whole process tree - then you can look at the one you're interested in. As others have said you could also do some smart things with GDB's follow-fork settings but I think process recording is ideal for capturing complicated situations like this as you can go and review what happened later on.