3 ms·
This is a pretty good article. Similarly to the "Heroism isn't…" section, I'd say: Breathe. I've been asked "how do you stay so calm when something is going wr
by deathanatos 2y ago
This is a pretty good article.
Similarly to the "Heroism isn't…" section, I'd say: Breathe. I've been asked "how do you stay so calm when something is going wrong?" and the honest truth is I'm scared! Or at least, I have that pit, in my stomach, going "oh no it's not working, will we figure this one out?" It's just not a useful thing. Tell that fear to take a backseat, and attempt to let the more logical side of you problem solve. And like TFA says, call for help if you need it; two minds are better than one.
At the management level, you can also do a sort corollary to basically everything in TFA too: "call for help": your engineers need to be able to call for help. That means retaining experience, so that the younger engineers can learn from the older ones, and hopefully not trial-by-fire their entire career, and have someone they can fall back on for help. Same goes for the experience devs, too: it means you need two experienced devs. I've been the only experienced person on the team, and it sucks, because I don't have the answer to everything. "It is your job to see that issues get addressed." — at the management layer, you need to make sure the incentives are focused on that, not something inane, like "mean time to resolution". Time to "the incident in PagerDuty is closed" is meaningless, and will be gamed to something like "we closed the incident because the immediate instance of the problem / symptom has been dealt with". You want the actual, underlying root cause debugged and fixed, and ideally, that eng should never see that entire class of problem again. But this means understanding the root cause, and understanding the system well enough to see the problem through to conclusion, which often means things like "ok, this needs to be fixed, *and I need to prioritize someone familiar with that portion of the system to fix it" — and all too often, that follow-through just doesn't happen. And when it doesn't, your eng pays for it, in the form of getting woken up. "Don't sacrifice your health" — are your eng sacrificing their health? Is your on-call experience too often? (At the lowest, I've been oncall 100% of the time. That was too often!)