3 ms·

This reminded me of a hail-mary use of debuggers in the late 1990s... Debugging a show-stopper synchronization flaw in the bootstrapping of a parallel job, I h

This reminded me of a hail-mary use of debuggers in the late 1990s... Debugging a show-stopper synchronization flaw in the bootstrapping of a parallel job, I had to tell the job system to launch each of 64 nodes wrapped in gdb wrapped in xterm with remote X display back to a laptop. It was something that had "worked in test" reliably, but that was always on a smaller number of nodes or simulating a larger number with time-sharing. It seemed to need real parallel hardware allocations to show up. So, I was able to launch the job, allow all nodes to run until distributed deadlock, then interrupt and show all thread stacks on one debugger after another until I found the odd process which was out of phase with the rest. As soon as I saw the stacks, I had no further need for the debuggers. Just knowing the "impossible" state configuration happened was the necessary clue. I have seen lots of satisfactory use of logs for diagnostics, but this is a case where I think the debuggers were almost essential. The debuggers gave a distributed state snapshot due to the deadlock-induced quiescence of the whole system. To have logs show the same snapshot would require a perfect arrangement of unbuffered logging so that the final state would be visible and not caught up in RAM of some or all of the stalled processes.

lamontcg 4y ago
Sounds like all you really needed something like a signal handler that would dump the stack traces of all the threads. I've used that a lot black-box debugging java processes in production. I also wrote such a signal handler for a ruby app.
- sitkack 4y ago
  One thing I have used is drop into the Python debugger (also ipdb) from within a signal handler. Then I can run a kill, and have it drop into the debugger. There are some of cute hacks to get a remote debugger which I haven't used in years. https://github.com/sassoftware/epdb https://github.com/sassoftware/epdb can start and connect to a remote debug instance https://github.com/gotcha/ipdb https://github.com/gotcha/ipdb
- saltcured 4y ago
  In a distributed parallel computing scenario, dumping stacks from 64 processes via a signal handler would have created 64 different files scattered in 64 different remote filesystems. Ironically, the purpose of our library was to bootstrap a distributed communication network and so could have easily supported gathering of remote diagnostic data _if it were not the part that deadlocked_... But worse, we were debugging our library linked into someone else's application. So, we would have needed their cooperation to add signal handlers. And, we naively thought our library had already been sufficiently tested and did not anticipate the need for last-minute debugging when our user moved their application to a different computing resource that was not available during prior months of preparation and prototypes. So, the ugly app-in-gdb-in-xterm with X over Internet addressed all that with adhoc instrumentation and communications that could be folded into the existing parallel, distributed job on short notice. This was also in the days of pthreads in C. The other valuable use of gdb I recall was for memory watchpoints to help track down unexpected changes to certain data. The other tool we got a lot of use out of in those days was Purify to help audit for use of uninitialized memory and for memory leaks.
  - geophile 4y ago
    > In a distributed parallel computing scenario, dumping stacks from 64 processes via a signal handler would have created 64 different files scattered in 64 different remote filesystems. I was working on a cluster and often needed to do the same thing on all the nodes, run some database query or os command, and gather and integrate the results. I wrote yet another Python stream-objects-instead-of-strings shell (https://geophile.com/osh https://geophile.com/osh), which included features for distribution: Run the same command on all nodes, bring back the results, merge streams, gather files, distribute files, etc. That was several years ago, and it wasn't a full shell. Since then, I've done an improved system that actually is a shell (https://marceltheshell.org https://marceltheshell.org, https://geophile.com/marcel https://geophile.com/marcel).

lamontcg 4y ago

Sounds like all you really needed something like a signal handler that would dump the stack traces of all the threads. I've used that a lot black-box debugging java processes in production. I also wrote such a signal handler for a ruby app.

sitkack 4y ago
One thing I have used is drop into the Python debugger (also ipdb) from within a signal handler. Then I can run a kill, and have it drop into the debugger. There are some of cute hacks to get a remote debugger which I haven't used in years. https://github.com/sassoftware/epdb https://github.com/sassoftware/epdb can start and connect to a remote debug instance https://github.com/gotcha/ipdb https://github.com/gotcha/ipdb
saltcured 4y ago
In a distributed parallel computing scenario, dumping stacks from 64 processes via a signal handler would have created 64 different files scattered in 64 different remote filesystems. Ironically, the purpose of our library was to bootstrap a distributed communication network and so could have easily supported gathering of remote diagnostic data _if it were not the part that deadlocked_... But worse, we were debugging our library linked into someone else's application. So, we would have needed their cooperation to add signal handlers. And, we naively thought our library had already been sufficiently tested and did not anticipate the need for last-minute debugging when our user moved their application to a different computing resource that was not available during prior months of preparation and prototypes. So, the ugly app-in-gdb-in-xterm with X over Internet addressed all that with adhoc instrumentation and communications that could be folded into the existing parallel, distributed job on short notice. This was also in the days of pthreads in C. The other valuable use of gdb I recall was for memory watchpoints to help track down unexpected changes to certain data. The other tool we got a lot of use out of in those days was Purify to help audit for use of uninitialized memory and for memory leaks.
- geophile 4y ago
  > In a distributed parallel computing scenario, dumping stacks from 64 processes via a signal handler would have created 64 different files scattered in 64 different remote filesystems. I was working on a cluster and often needed to do the same thing on all the nodes, run some database query or os command, and gather and integrate the results. I wrote yet another Python stream-objects-instead-of-strings shell (https://geophile.com/osh https://geophile.com/osh), which included features for distribution: Run the same command on all nodes, bring back the results, merge streams, gather files, distribute files, etc. That was several years ago, and it wasn't a full shell. Since then, I've done an improved system that actually is a shell (https://marceltheshell.org https://marceltheshell.org, https://geophile.com/marcel https://geophile.com/marcel).