I've had similar experience with scientific software. When I'm told that the existing software is "OK because it works", I ask "how do you know it works?" becau

I've had similar experience with scientific software. When I'm told that the existing software is "OK because it works", I ask "how do you know it works?" because typically there are no unit tests or tests of any sort of individual stages for that matter. I've found that scientists tend to assume "it works" when they like the results they see such as R^2 values high enough to publish. Recently I converted some scientific software that was using correlation^2 (calling it R^2) as a measure for model predictions, as opposed to something more appropriate like PRESS-derived R^2s (correlation is totally inappropriate for judging predictions because it's translation and scale independent on both observed and predicted sides). Nobody went looking for the problem because results seem good and reasonable. Converting to a proper prediction R^2, some of the results are now negative, meaning the models are doing worse than a simple constant-mean function. Yikes.