4 ms·

> Or even worse, saying "how the hell did that ever work?" The most unsettling feeling. I often have that feeling when doing the devops and infrastructure part

> Or even worse, saying "how the hell did that ever work?" The most unsettling feeling. I often have that feeling when doing the devops and infrastructure part of my job, e.g. encountering paradoxes in the dark dreary bowels of systemd. The idealist and perfectionist in you wants to keep digging deeper to arrive at a proper understanding. The realist and lazy SOB in you wants to slowly back away and pretend this never happened. This dialectic hopefully guides you toward a happy compromise, somewhere down the middle.

Karunamon 5y ago
Worse, you dig at it for too long and collapse the wave function, and it never works again despite you having made no changes.
- mensetmanusman 5y ago
  Because the code you are looking at isn’t what is compiled and deployed, doh!
anyfoo 5y ago
The more critical the infrastructure you work on, the less you will be willing to take the "slowly back away" route. Because as long as you don't understand every single detail of the failure, that feeling of "what horrible thing do I not understand here" won't go away, and you dread having a major "oh... crap, so that's what I did not understand" moment when it's inevitable. This week, we had such a failure when testing a change. The root cause was found eventually, but what two extremely good engineers plus me still could not figure out was why that problem was only surfacing now, after years of having that buggy code in there. Very important code. I did not want to risk some obviously unknown property of it be our demise later on. Fortunately, in this case it just turned out that the seemingly unrelated changes did, after all, hold the now failing thing differently than before. As it had been used, the problem was masked entirely, and the code did work well for all those years.

4 ms·

> Or even worse, saying "how the hell did that ever work?" The most unsettling feeling. I often have that feeling when doing the devops and infrastructure part

by kspacewalk2 5y ago

Karunamon 5y ago
Worse, you dig at it for too long and collapse the wave function, and it never works again despite you having made no changes.
- mensetmanusman 5y ago
  Because the code you are looking at isn’t what is compiled and deployed, doh!
anyfoo 5y ago
The more critical the infrastructure you work on, the less you will be willing to take the "slowly back away" route. Because as long as you don't understand every single detail of the failure, that feeling of "what horrible thing do I not understand here" won't go away, and you dread having a major "oh... crap, so that's what I did not understand" moment when it's inevitable. This week, we had such a failure when testing a change. The root cause was found eventually, but what two extremely good engineers plus me still could not figure out was why that problem was only surfacing now, after years of having that buggy code in there. Very important code. I did not want to risk some obviously unknown property of it be our demise later on. Fortunately, in this case it just turned out that the seemingly unrelated changes did, after all, hold the now failing thing differently than before. As it had been used, the problem was masked entirely, and the code did work well for all those years.