4 ms·

I'm a big fan of using chat as the real-time incident log even when we're all in the same room. It's trivial to generate a timeline of events for the postmortem

by kevan 10y ago

I'm a big fan of using chat as the real-time incident log even when we're all in the same room. It's trivial to generate a timeline of events for the postmortem and there's a lot of info you can extract from the log during the postmortem process.

Kiro 10y ago
Yes, I think that's what has added most value for us after introducing Slack. It's extremely useful.
- newman8r 10y ago
  I'm working on some communications software myself - so one of my hobbies is writing down as many UX issues as I can find with slack/hipchat/etc etc Have you noticed anything that you'd change? I actually like slack for the most part, but I still find some UX smells in it. And I actually use it for incident logging too.. never really thought about it but it's one of my favorite uses for it - and it helps the client on that job see the response times and get a general overview.
jessmartin 10y ago
> The chat history is a convenient record of what happened during the event and serves as the official record of the incident. We have a similar process for triage, recovery, and review, mainly utilizing Slack for our chatops. We use Slack, GitHub and Trello as our three main teamwork tools. But I have _not_ found chat history to a convenient record of what happened. How do you go back to _just_ show a specific event in history? Also, when there is serious downtime, the channel gets flooded with engineer chatter, and it's difficult to quickly comprehend the root issue, decipher what has been attempted, and know how to help. In addition to our synchronous chatter in Slack during an event, we open a GitHub issue where we post periodic updates about specific things that have been attempted to fix the problem, graphs from server activity during the event, etc. We'll use checklists to keep track of things that still need to be investigated and what has been done. Sometimes, yes, you need to post in both the GitHub issue and the Slack channel, but it provides a much more digestible, high-level view of an outage. Additionally, the GitHub issue serves as an ongoing record where you can post the content of the blameless post-mortem, reference follow-on work that was done to harden the infrastructure, etc. Finally, since the GitHub issue is labelled as "Downtime", it's _actually_ convenient to scan through open and past issues and learn from past mistakes and see what can still be improved.

by kevan 10y ago

Kiro 10y ago

Yes, I think that's what has added most value for us after introducing Slack. It's extremely useful.

newman8r 10y ago
I'm working on some communications software myself - so one of my hobbies is writing down as many UX issues as I can find with slack/hipchat/etc etc Have you noticed anything that you'd change? I actually like slack for the most part, but I still find some UX smells in it. And I actually use it for incident logging too.. never really thought about it but it's one of my favorite uses for it - and it helps the client on that job see the response times and get a general overview.

jessmartin 10y ago

> The chat history is a convenient record of what happened during the event and serves as the official record of the incident. We have a similar process for triage, recovery, and review, mainly utilizing Slack for our chatops. We use Slack, GitHub and Trello as our three main teamwork tools. But I have _not_ found chat history to a convenient record of what happened. How do you go back to _just_ show a specific event in history? Also, when there is serious downtime, the channel gets flooded with engineer chatter, and it's difficult to quickly comprehend the root issue, decipher what has been attempted, and know how to help. In addition to our synchronous chatter in Slack during an event, we open a GitHub issue where we post periodic updates about specific things that have been attempted to fix the problem, graphs from server activity during the event, etc. We'll use checklists to keep track of things that still need to be investigated and what has been done. Sometimes, yes, you need to post in both the GitHub issue and the Slack channel, but it provides a much more digestible, high-level view of an outage. Additionally, the GitHub issue serves as an ongoing record where you can post the content of the blameless post-mortem, reference follow-on work that was done to harden the infrastructure, etc. Finally, since the GitHub issue is labelled as "Downtime", it's _actually_ convenient to scan through open and past issues and learn from past mistakes and see what can still be improved.