To extend on inspecting stack traces: I found that I often wanted to know the the content of (relevant) (local) variables of a stack trace. Whenever I got a sta

To extend on inspecting stack traces: I found that I often wanted to know the the content of (relevant) (local) variables of a stack trace. Whenever I got a stacktrace, I would often restart it in a debugger, or add print statements on variables. Until I extended the stacktrace to just include this information. The tricky part is that printing all local variables is too much. And also global variables might be involved. Or attributes (e.g. self.x or so). So now I'm parsing all variables and attributes from the source code line from the traceback (in a very simple way) and print only those. This covers about 95% of all cases - i.e. in 95% of the case, the stack trace contains all information I need to understand and fix the problem. I published that here: https://pypi.org/project/better_exchook/ https://pypi.org/project/better_exchook/ https://github.com/albertz/py_better_exchook https://github.com/albertz/py_better_exchook I also often have a SIGUSR1 handler which will print the stacktrace of all threads. This is useful on long running processes, involving multi threading, where you might run into some strange hangs or deadlocks. In addition to that, if you might get crashes (segfault or so), something like faulthandler is useful. If you also want to see the C stack trace in addition in such cases, I load libSegFault.so, like here: https://github.com/rwth-i6/returnn/blob/5b8e34ec1fd725d0e20b5b422d213dbf17d9e069/Debug.py#L226 https://github.com/rwth-i6/returnn/blob/5b8e34ec1fd725d0e20b... Profiling is another topic. Something like Py-Spy (https://github.com/benfred/py-spy https://github.com/benfred/py-spy) can be very helpful. I also found a remote background ZMQ IPython/Jupyter kernel to be useful sometimes. I published that here: https://github.com/albertz/background-zmq-ipython https://github.com/albertz/background-zmq-ipython