4 ms·

I have to agree. The environment that leads to a fast blog post may also lead to this quote from the post: > This was delayed as network engineers walked over

I have to agree. The environment that leads to a fast blog post may also lead to this quote from the post: > This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically. They are running as fast as they can and this extended the incident. There is a “slow is smooth, smooth is fast” lesson in here. I’d rather have a team that takes a day to put up the blog post, but doesn’t unnecessarily extend downtime because they are sprinting.

bombcar 4y ago
It can be easy to arm-chair it afterwards, but unless things can be done in parallel (and systems should be designed so this can be done, things like "we're not sure what's wrong, we're bringing up a new cluster on the last known good version even as we try to repair this one") you have to make a choice, and sometimes it won't be optimal.
jgrahamc 4y ago
There's normal operating procedure and sign offs and automation etc. etc. and then there's "we've lost contact with these data centers and normal procedures don't work we need to break glass and use the secondary channels". In that situation you are in an emergency without normal visibility.
bad416f1f5a2 4y ago
At a previous job, I worked with two guys who were excellent in a crisis. One of them used to run operations, the other was a crusty old programmer who’d been around for a while. I tried to learn as much as I could from both of them. At around the same time, I was watching HBO’s The Wire. One scene had a high-profile shooting with a frantic police response; people were running everywhere trying to help. The lead commander on the scene gave this instruction to his sergeant: “Slow this thing down to a crawl. Give these bastards no chance to fuck up in a meaningful way.” And then it hit me. That’s exactly how they ran the calls. I asked them about this, and they said absolutely – human nature, when things are broken badly, is biased towards action. You want to try to make a change, to reboot a system, to do something to hopefully make things better. But often if you were not careful, you risk losing information about the outage you are in. Best case scenario you luck into fixing the problem and don’t know how. Worst case? You’ve changed the state of an already broken system, and have done nothing but add more variables to unwind. So now, every time I’m on an outage call, I try to do what Wes and Tim and Major Rawls would all do: I take control, pump the brakes, and make sure that we are capturing enough information about the current state that we don’t confuse ourselves further.

bombcar 4y ago

It can be easy to arm-chair it afterwards, but unless things can be done in parallel (and systems should be designed so this can be done, things like "we're not sure what's wrong, we're bringing up a new cluster on the last known good version even as we try to repair this one") you have to make a choice, and sometimes it won't be optimal.

jgrahamc 4y ago

There's normal operating procedure and sign offs and automation etc. etc. and then there's "we've lost contact with these data centers and normal procedures don't work we need to break glass and use the secondary channels". In that situation you are in an emergency without normal visibility.

bad416f1f5a2 4y ago

At a previous job, I worked with two guys who were excellent in a crisis. One of them used to run operations, the other was a crusty old programmer who’d been around for a while. I tried to learn as much as I could from both of them. At around the same time, I was watching HBO’s The Wire. One scene had a high-profile shooting with a frantic police response; people were running everywhere trying to help. The lead commander on the scene gave this instruction to his sergeant: “Slow this thing down to a crawl. Give these bastards no chance to fuck up in a meaningful way.” And then it hit me. That’s exactly how they ran the calls. I asked them about this, and they said absolutely – human nature, when things are broken badly, is biased towards action. You want to try to make a change, to reboot a system, to do something to hopefully make things better. But often if you were not careful, you risk losing information about the outage you are in. Best case scenario you luck into fixing the problem and don’t know how. Worst case? You’ve changed the state of an already broken system, and have done nothing but add more variables to unwind. So now, every time I’m on an outage call, I try to do what Wes and Tim and Major Rawls would all do: I take control, pump the brakes, and make sure that we are capturing enough information about the current state that we don’t confuse ourselves further.