3 ms·
Using a "proper" debugger on a supercomputer tends to be super tricky and the correct setup tends to depend on the machine. And even if you have a debugger atta
by petschge 5y ago
Using a "proper" debugger on a supercomputer tends to be super tricky and the correct setup tends to depend on the machine. And even if you have a debugger attached to all 100k MPI processes, where do you look? And if you try to debug a production run you certainly don't have enough RAM left over for a time travel debugger like rr. The "bad" print debugging still works at that scale. Sure, once you have understood the problem well enough that you can reproduce it on a single node within a couple of minutes a "proper" debugger can be very useful. But you need to get to that point first.
And no core dumps are not a "solves everything solution" either. A) because dumping the entire 1500 TB of memory to disk takes a lot of time and B) because bugs can trash the stack, leading to misleading stack traces.