Debugging mysterious recurring hangs in a storage appliance. The nodes were connected via Infiniband, so there were two parts to the hack. The first was to writ

Debugging mysterious recurring hangs in a storage appliance. The nodes were connected via Infiniband, so there were two parts to the hack. The first was to write a program which would find all of the kernel memory areas and register them with the Infiniband controller. The second was to slurp up all of the memory on a hung node via Infiniband from one of its peers, and convert that into dump format so we could use gdb on it. After collecting a couple of dozen such dumps and poring over them for a couple of days, I was finally able to debug one of our most intractable problems. That led to a second, though lesser, hack. The problem turned out to be code that was allocating way too much memory. (Userland programmers working in the kernel, ugh.) This would sometimes cause the stack pointer for one task to jump all the way over its task structure into the stack of the previous task, which would make that stack very confusing. That's why it took two days to figure out, even with dumps in hand. (Also, putting the stack right next to the task structure was a stupid decision on Linux's part.) To find the dozens of places where this was happening, I wrote a tool to disassemble all of our kernel code and look for the part of the function prolog that allocated stack space, thus creating a list of those that were allocating too much. Then I spent a week fixing them. It was a grueling process, under lots of pressure at a struggling startup, but ultimately the result was very rewarding.