3 ms·
> The chat history is a convenient record of what happened during the event and serves as the official record of the incident. We have a similar process for tr
by jessmartin 10y ago
> The chat history is a convenient record of what happened during the event and serves as the official record of the incident.
We have a similar process for triage, recovery, and review, mainly utilizing Slack for our chatops. We use Slack, GitHub and Trello as our three main teamwork tools.
But I have _not_ found chat history to a convenient record of what happened. How do you go back to _just_ show a specific event in history? Also, when there is serious downtime, the channel gets flooded with engineer chatter, and it's difficult to quickly comprehend the root issue, decipher what has been attempted, and know how to help.
In addition to our synchronous chatter in Slack during an event, we open a GitHub issue where we post periodic updates about specific things that have been attempted to fix the problem, graphs from server activity during the event, etc. We'll use checklists to keep track of things that still need to be investigated and what has been done. Sometimes, yes, you need to post in both the GitHub issue and the Slack channel, but it provides a much more digestible, high-level view of an outage.
Additionally, the GitHub issue serves as an ongoing record where you can post the content of the blameless post-mortem, reference follow-on work that was done to harden the infrastructure, etc.
Finally, since the GitHub issue is labelled as "Downtime", it's _actually_ convenient to scan through open and past issues and learn from past mistakes and see what can still be improved.