3 ms·
The more critical the infrastructure you work on, the less you will be willing to take the "slowly back away" route. Because as long as you don't understand eve
by anyfoo 5y ago
The more critical the infrastructure you work on, the less you will be willing to take the "slowly back away" route. Because as long as you don't understand every single detail of the failure, that feeling of "what horrible thing do I not understand here" won't go away, and you dread having a major "oh... crap, so that's what I did not understand" moment when it's inevitable.
This week, we had such a failure when testing a change. The root cause was found eventually, but what two extremely good engineers plus me still could not figure out was why that problem was only surfacing now, after years of having that buggy code in there. Very important code. I did not want to risk some obviously unknown property of it be our demise later on.
Fortunately, in this case it just turned out that the seemingly unrelated changes did, after all, hold the now failing thing differently than before. As it had been used, the problem was masked entirely, and the code did work well for all those years.