I worked on software that encrypted entries with a custom C++ plugin to SQL Server. Nobody knew how it worked, it was implemented long ago and considered magic.

I worked on software that encrypted entries with a custom C++ plugin to SQL Server. Nobody knew how it worked, it was implemented long ago and considered magic. Some entries on some installations sometimes would become totally garbled, which was a seriously dangerous issue. This was a pure Java shop and I was almost the only engineer who even knew C++ since the reorganization that happened before I joined. Turns out the affected installations all used custom compiled DLLs with hard coded specific crypto keys made up by someone long ago. We didn't really know how to deploy new versions of this. But I found some version of the code at least, on some old file server, so I could sort of debug it. It took me a while to figure out but after working on it on and off for a few weeks I found a crucial race condition in the code path that looked up the crypto key. Turns out if you were unlucky after a SQL Server reboot, a few entries could get encrypted with a "default" key. So I was able to decrypt the garbled entries on affected customers at least. We decided to change the whole method of encryption since it was obviously hacked together by some cowboy coder a decade ago. But in the meantime, the easiest way to fix the issue for the affected customers was a method I taught to one of the tech support guys. I told him to RDP into the affected server, open the crypto DLL in Notepad++ and do a string substitution on a particular ASCII sequence that would change a certain character into a tilde. And then reload the DLL with a SQL Server command. Because through disassembling the binary I found that this would sufficiently increase a certain value and fix the bogus comparison that caused the race condition. The nightmare here was mostly because of custom DLLs running semi-unknown code on remote servers.