3 ms·

A war room / call is for coordination. If you need the person draining the bad region to know that "oh that other region is bad, so we can't drain too fast" or

by yuliyp 2y ago

A war room / call is for coordination. If you need the person draining the bad region to know that "oh that other region is bad, so we can't drain too fast" or "the metrics look better now". For truly understanding an incident? Hell no. That requires heads down *focus*. I've made a habit of taking a few hours later in the day after incidents to calmly go through and try to write a correct timeline. It's amazing how broken peoples' perceptions of what was actually happening at the time are (including my own!). Being able to go through it calmly, alone provided tons of insights. It's this type of analysis that leads to proper understanding and the right follow-ups rather than knee-jerk reactions.

DylanDmitri 2y ago
I've been in some good coordinating calls for widespread incidents. Many unique individuals (15+) talked in a ten minute period, sharing context on what their teams were seeing, what re-meditations had worked for them, etc..
master_crab 2y ago
This. People keep commenting about it being performative. That’s orthogonal to its purpose. Even the original blogpost points out the limitation of singular focused effort without acknowledging it. It took the author weeks to figure out the actual issue. If FB had been down that long, they’d be out of business.
thwarted 2y ago
It took me seven weeks (not full time, but from the initial incident to the final publishing) to do the research and write up for a recent event. This included in-person interviews, data correlation, reading code, and revision control spelunking across multiple repositories to understand the series of events and decisions that led to the event, some of them months earlier. Some people were advocating "get it out because we have to move on", which I pushed back on. Once published, the feedback was positive and some folks acknowledged that knee-jerk follow-up reactions would have made things worse. But to get to the point where the post-incident review is valuable someone has to put in the actual work and time to make it so. It should be a learning experience, not a checking a box; otherwise, we're just spinning our wheels without making any progress.

by yuliyp 2y ago

DylanDmitri 2y ago

I've been in some good coordinating calls for widespread incidents. Many unique individuals (15+) talked in a ten minute period, sharing context on what their teams were seeing, what re-meditations had worked for them, etc..

master_crab 2y ago

This. People keep commenting about it being performative. That’s orthogonal to its purpose. Even the original blogpost points out the limitation of singular focused effort without acknowledging it. It took the author weeks to figure out the actual issue. If FB had been down that long, they’d be out of business.

thwarted 2y ago

It took me seven weeks (not full time, but from the initial incident to the final publishing) to do the research and write up for a recent event. This included in-person interviews, data correlation, reading code, and revision control spelunking across multiple repositories to understand the series of events and decisions that led to the event, some of them months earlier. Some people were advocating "get it out because we have to move on", which I pushed back on. Once published, the feedback was positive and some folks acknowledged that knee-jerk follow-up reactions would have made things worse. But to get to the point where the post-incident review is valuable someone has to put in the actual work and time to make it so. It should be a learning experience, not a checking a box; otherwise, we're just spinning our wheels without making any progress.