... so it isn't like it is common practice to log in to a production database server and do apt-get/yum install gdb and start banging away. This may happen mor

... so it isn't like it is common practice to log in to a production database server and do apt-get/yum install gdb and start banging away. This may happen more often than you think. There is a technique popularized by some MySQL hackers at Facebook called "poor man's query profiling" that uses gdb to (among other things) dump the the stack traces of every MySQL thread. Some awk normalizes, aggregates and sorts the traces. I often do this when I encounter a badly flailing or completely wedged MySQL daemon. It's a good way to see inside MySQL, and if a bunch of threads are all blocked on a mutex or something it's pretty obvious.