Yes, frequently. A couple of highlights: PostgreSQL allows C extensions. A function which used to work was now segfaulting occasionally so needed to be fixed.

Yes, frequently. A couple of highlights: PostgreSQL allows C extensions. A function which used to work was now segfaulting occasionally so needed to be fixed. Original author was gone, nobody really understood the PostgreSQL extension system (which involves a ton of macros.) No version control, no useful comments, no spec. I read the code (the core of the logic was straight-forward C, it was just the interface to PostgreSQL that was hard to understand) and wrote a new version in Python (another language for which PostgreSQL supports extension functions.) That version was too slow, by two orders of magnitude. Went back to the C implementation. Did a little light refactoring to separate the core logic from the PostgreSQL interface. Then wrote a test harness program in C to drive call only the inner function. Wrote unit tests until I reproduced the segfault. Fixed the C code and tested it with my test harness. The PostgreSQL wrapper just called the (now correct) inner function so it now worked too. Checked the fixed code and unit tests into version control. A PhD (no longer with the company) had fit some simple neural net models in R. He'd written his own code for this because, according to him, the standard packages in R didn't support a few of the bells and whistles he'd wanted like ReLU activation. Not only was the code in-house, but the models themselves were saved as serialized R objects. No specifications for any of this stuff. Apparently the company had been using this code to score medium size databases for several years. When we wanted to scale up to a much larger database (approximately 30 billion rows) the problems with the R implementation became apparent. Fairly slow, high memory usage, worse yet a slow memory leak on large jobs, and worst of all occasional silent crashes where it would simply stop and exit with a successful status of 0 and no error message. This time I took the approach of reading the code and de-serialized R objects. I re-wrote the implementation in Python using numpy arrays and wrote a small R program to read the serialized models in the .RData format and emit a cleaned up JSON object that could be easily read from Python. Luckily I didn't have to port any of the cross-validation/training/optimization stuff; just the prediction part. That meant 80% of the code could be safely ignored. However, the devil was in all the one-off special cases in the serialized R objects, many of which had behavior different from default. I would test by comparing the predictions by the two programs on small batches of a million. These predictions were floating point numbers between 0 and 1 so when both programs agreed to within 1e-5 for all million I knew it was correct. It took a week to track down all the special cases, though, and the special cases made it impossible to just use a standard neuralnet library like Keras. (We already had several Keras models in production and a set of tools to manage them; that would have been easy for us.) Proprietary code begets proprietary code I guess. At least the new implementation was much faster and didn't leak memory so could deal with the whole database in a single long running process. I pitched the idea of re-training all models from scratch in Keras using our more modern tools but management wanted 100% backwards compatibility and to preserve the value-add of the PhD.