4 ms·

I agree, I think the transparency builds trust and I encourage it where I can. The counter thought I had when reading this case though, is it almost feels too f

I agree, I think the transparency builds trust and I encourage it where I can. The counter thought I had when reading this case though, is it almost feels too fast. What I mean by that is I hope there isn't an incentive to wrap up the internal investigation quickly and write the blog and send it, and go we're done. Doing incident response (both outage and security), the tactical fixes for a specific problem are usually pretty easy. We can fix a bug, or change this specific plan to avoid the problem. The search for conditions that allowed the incident to occur can be alot more time consuming, and most organizations I've worked for are happy to make a couple tactical changes and move on.

jgrahamc 4y ago
What I mean by that is I hope there isn't an incentive to wrap up the internal investigation quickly and write the blog and send it, and go we're done. There is not. From here there's an ongoing process with a formal post-mortem, all sorts of tickets tracking work to prevent further reoccurrence. This post is just the beginning internally.
cowsandmilk 4y ago
I have to agree. The environment that leads to a fast blog post may also lead to this quote from the post: > This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically. They are running as fast as they can and this extended the incident. There is a “slow is smooth, smooth is fast” lesson in here. I’d rather have a team that takes a day to put up the blog post, but doesn’t unnecessarily extend downtime because they are sprinting.
- bombcar 4y ago
  It can be easy to arm-chair it afterwards, but unless things can be done in parallel (and systems should be designed so this can be done, things like "we're not sure what's wrong, we're bringing up a new cluster on the last known good version even as we try to repair this one") you have to make a choice, and sometimes it won't be optimal.
- jgrahamc 4y ago
  There's normal operating procedure and sign offs and automation etc. etc. and then there's "we've lost contact with these data centers and normal procedures don't work we need to break glass and use the secondary channels". In that situation you are in an emergency without normal visibility.
- bad416f1f5a2 4y ago
  At a previous job, I worked with two guys who were excellent in a crisis. One of them used to run operations, the other was a crusty old programmer who’d been around for a while. I tried to learn as much as I could from both of them. At around the same time, I was watching HBO’s The Wire. One scene had a high-profile shooting with a frantic police response; people were running everywhere trying to help. The lead commander on the scene gave this instruction to his sergeant: “Slow this thing down to a crawl. Give these bastards no chance to fuck up in a meaningful way.” And then it hit me. That’s exactly how they ran the calls. I asked them about this, and they said absolutely – human nature, when things are broken badly, is biased towards action. You want to try to make a change, to reboot a system, to do something to hopefully make things better. But often if you were not careful, you risk losing information about the outage you are in. Best case scenario you luck into fixing the problem and don’t know how. Worst case? You’ve changed the state of an already broken system, and have done nothing but add more variables to unwind. So now, every time I’m on an outage call, I try to do what Wes and Tim and Major Rawls would all do: I take control, pump the brakes, and make sure that we are capturing enough information about the current state that we don’t confuse ourselves further.

jgrahamc 4y ago

What I mean by that is I hope there isn't an incentive to wrap up the internal investigation quickly and write the blog and send it, and go we're done. There is not. From here there's an ongoing process with a formal post-mortem, all sorts of tickets tracking work to prevent further reoccurrence. This post is just the beginning internally.

cowsandmilk 4y ago

I have to agree. The environment that leads to a fast blog post may also lead to this quote from the post: > This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically. They are running as fast as they can and this extended the incident. There is a “slow is smooth, smooth is fast” lesson in here. I’d rather have a team that takes a day to put up the blog post, but doesn’t unnecessarily extend downtime because they are sprinting.

bombcar 4y ago
It can be easy to arm-chair it afterwards, but unless things can be done in parallel (and systems should be designed so this can be done, things like "we're not sure what's wrong, we're bringing up a new cluster on the last known good version even as we try to repair this one") you have to make a choice, and sometimes it won't be optimal.
jgrahamc 4y ago
There's normal operating procedure and sign offs and automation etc. etc. and then there's "we've lost contact with these data centers and normal procedures don't work we need to break glass and use the secondary channels". In that situation you are in an emergency without normal visibility.
bad416f1f5a2 4y ago
At a previous job, I worked with two guys who were excellent in a crisis. One of them used to run operations, the other was a crusty old programmer who’d been around for a while. I tried to learn as much as I could from both of them. At around the same time, I was watching HBO’s The Wire. One scene had a high-profile shooting with a frantic police response; people were running everywhere trying to help. The lead commander on the scene gave this instruction to his sergeant: “Slow this thing down to a crawl. Give these bastards no chance to fuck up in a meaningful way.” And then it hit me. That’s exactly how they ran the calls. I asked them about this, and they said absolutely – human nature, when things are broken badly, is biased towards action. You want to try to make a change, to reboot a system, to do something to hopefully make things better. But often if you were not careful, you risk losing information about the outage you are in. Best case scenario you luck into fixing the problem and don’t know how. Worst case? You’ve changed the state of an already broken system, and have done nothing but add more variables to unwind. So now, every time I’m on an outage call, I try to do what Wes and Tim and Major Rawls would all do: I take control, pump the brakes, and make sure that we are capturing enough information about the current state that we don’t confuse ourselves further.