3 ms·
Wait, is the idea that your SRE/ Ops person doing rollbacks, failovers, and restarts without understanding why... is a good thing? Runbooks are supposed to giv
by ang_cire 3y ago
Wait, is the idea that your SRE/ Ops person doing rollbacks, failovers, and restarts without understanding why... is a good thing?
Runbooks are supposed to give you a clear path during an incident so you don't run down rabbit holes, but you still shouldn't be running them without understanding what the impact of their steps actually is.
I thought the horror part was that some random person, who doesn't know the system, was doing all this stuff without just contacting the people who do know it.
If the runbook was sufficient to handle it, great! The proper team will have just as easy a time doing the steps, and not be mucking around in something they don't own and won't be the ones to fix when the runbook process itself breaks something because Error A is not Error J, but un-familiarized person can't tell the difference.
- theK 3y agoThe way I read it the horror was reading how that innocent guy was performing actions "they" didn't really understand without the setting itself being conclusive enough that "they" could actually trust those instructions blindly. Did I read it wrong?
- ang_cire 3y agoI think the ending is supposed to flip it into being that the person not having that runbook is the horror part.
- brainzap 3y agoit depends if the company designed it that way, in a bigger company I would have the policy that all deployments can be rolled back (except some bigger migrations)
- ang_cire 3y agoSure, but what if an infra change that is not part of that automated deployment also needs to happen? It's one thing to allow anyone to *call* for a rollback of a deployment. It's another thing entirely to allow anyone to, independently and without sign-offs, *perform* a rollback.