7 ms·

Incident Response Documentation

246 points by blopeur 8y ago

xrayzerone 8y ago
Awesome. Will be reading these in short order. Can anyone recommend other good incident response resources (that are relevant in 2019)?
- brooksgarrett 8y ago
  Firefighter for over 10 years and an IT career spanning Red Team/Blue Team and now SRE, I can't recommend enough taking the FEMA Independent Study course in Incident Command. https://training.fema.gov/is/courseoverview.aspx?code=IS-100.c https://training.fema.gov/is/courseoverview.aspx?code=IS-100... Even if you don't adopt the system it will help you frame and understand how complex IR can be. Having a scalable system to a) grow with resources and b) grow with external interactions is crucial to have BEFORE you need it.
- twunde 8y ago
  Atlassian's handbook is pretty good: https://www.atlassian.com/software/jira/ops/handbook https://www.atlassian.com/software/jira/ops/handbook
- richadams 8y ago
  There are some good additional resources referenced in the docs here: https://response.pagerduty.com/resources/reading/ https://response.pagerduty.com/resources/reading/ Specifically, Google's SRE books are particularly useful (https://landing.google.com/sre/books/ https://landing.google.com/sre/books/) along with the book "Incident Management for Operations" (http://shop.oreilly.com/product/0636920036159.do http://shop.oreilly.com/product/0636920036159.do) and Etsy's Debriefing Facilitation Guide (http://extfiles.etsy.com/DebriefingFacilitationGuide.pdf http://extfiles.etsy.com/DebriefingFacilitationGuide.pdf). The book "Comparative Emergency Management" (https://training.fema.gov/hiedu/aemrc/booksdownload/compemmgmtbookproject/ https://training.fema.gov/hiedu/aemrc/booksdownload/compemmg...) is also quite interesting, as it compares the emergency management practices of about 30 different countries.
johnmarcus 8y ago
let me guess...it includes getting paged as a very important step. j/k, this is very well written. Good for sharing to those whom have not been on call at a good org before.
lifekaizen 8y ago
This is wonderful, especially the section on what didn't work ("Anti patterns," bottom of page). This one in particular feels like good advice to startup founders as their company grows: > Trying to take on multiple roles.# In past PagerDuty incidents, we've had instances where the Incident Commander has started to assume the Subject Matter Expert role and attempted to solve the problem themselves. This usually happens when the IC is an engineer in their day-to-day role. They are in an incident where the cause appears to be a system they know very well and have the requisite knowledge to fix. Wanting to solve the incident quickly, the IC will start to try and solve the problem. Sometimes you might get lucky and it will resolve the incident, but most of the time the immediately visible issue isn't necessarily the underlying cause of the incident. By the time that becomes apparent, you have an Incident Commander who is not paying attention to the other systems and is just focussed on the one problem in front of them. This effectively means there's no incident commander, as they would be busy trying to fix the problem. Inevitably, the problem turns out to be much bigger than anticipated and the response has become completely derailed.
- cco 8y ago
  An officer does not fire their weapon, they assess and respond to the situation.
uasm 8y ago
> "A guide to being on-call... we all have lives which might get in the way of on-call time" How dare those engineers live a life. They're on-call goddammit.
- lwf 8y ago
  Life gets in the way of scheduled work (on-call). The section you're quoting is about enabling on-call people to have lives without on-call constraints. It's encouraging people to be considerate and cover a colleague's shift if they want to go out and do something, and you're free.
  - uasm 8y ago
    > "Life gets in the way of scheduled work (on-call)." Scheduling 24h work days will inevitably cause "life to get in the way of work". On-calls commonly take a week or two at a time. Stop de-legitimizing work-life seperation.
    - lwf 8y ago
      You're reading way more into both the PagerDuty guide and my comments than was intended. Work-life balance is great. Personally, I don't feel like I need to respond to queries on nights or weekends, take plenty of vacation, and feel empowered to push back at anybody going out of those bounds. Then again, I also work at a company where your colleagues will yell at you if you show up on work chat when you're supposed to be on PTO :) That said, if you're on an on-call shift, and something comes up, that's "life getting in the way". And that's fine. I have separate issues, personally, with how many companies compensate employees for on-call work. But it is an on-call shift, a concept that exists in basically every industry where you're providing a timely service. (plumbers, airline pilots, systems administration...) Unless you're large enough to implement follow-the-sun, you're going to need someone to cover the evenings.
      - deleted 8y ago
        [deleted]
gberger 8y ago
Curious how PagerDuty handles incident response when PagerDuty itself is down.
- kenrose 8y ago
  I used to work at PD. When I was there, they followed a lot of (all?) of these guidelines. The app’s ownership was spread across teams, so depending which section was affected would page certain teams. If it was SEV-2 or higher, that would page an IC and the group of primary on-calls to begin triage. Other SMEs were looped in as necessary. The anti patterns section is quite authentic. PD had very healthy discussions internally about the topics covered (eg, when it made sense to stop paging everyone, how to make people feel it was OK to drop off the call if they weren’t adding anything). In terms of the actual “how do they page people if PD is down?”, they had some backup monitoring systems that could SMS / phone on-calls directly. As a piece of software though, PD is pretty resilient, so it was rare to have an outage that affected everything so badly that they had to rely on these secondary systems.
  - wbronitsky 8y ago
    I can verify this as well. I worked on a separate team from Ken, but while we were there, we faced a lot of issues that led the team to decide upon and codify these rules. They are pretty tried and true, if only because they came out of a lot of iteration and testing. Hope you are doing well, Ken!
- techslave 8y ago
  isn't that the article? why would PD write a summary of IRP that they use for ... what if not themselves?
  - gberger 8y ago
    I meant to ask “how do they page people if PD is down?”, which kenrose included in this reply.
C1sc0cat 8y ago
That's interesting it seems to assume there is no 24/7 ops coverage on site who triage before escalating the call to actual on call staff. This is from being the lead on call for the UK's tymnet billing system normally if I got called some initial triage
- sambull 8y ago
  Yea pretty much from my experience now in the US, most private companies use their salaried employees and wake them up multiple times a night as sometimes with customer directly on the other end of the line. Pagerduty enables this to work. This is considered now normal for smallish private orgs, shit my spot just got bought by a billions dollar defense contractor I still get woken up at 3am with a end user having printer issues.. and I work as a devops engineer.
  - softawre 8y ago
    If you're not exaggerating you need to nip that in the bud. Printer issues at 3am? Have two classes of alerts and send printer issues to somebody else, or worst case email notification for the morning.
    - sambull 8y ago
      Sev1 for us, end users are 24/7 manufacturers and they consider this a business stoppage, this would be SLA downtime. They shoved old school COTs into AWS and called it cloud. Which ugh yea.
  - wil421 8y ago
    Do your incidents have Priorites (Severity in the article)? At my company a printer would never be a P1 to P2 therefore no one gets paged at 3am due to printer issues. It’s not full proof and my team did get a P1 due to a field length being too short but it was during normal business hours. It was quickly changed to a P4 and the person who opened it got a lesson on Incident priorities.
    - 8y ago
TagAsDelusion 8y ago
reacting means you have not planned correctly this is fundamentally perverse
luhn 8y ago
One thing I've been wondering about is how smaller organizations handle on call. Looking at the roles laid out [1] and assuming an on-call schedule of 4 weeks off, 1 week secondary, and 1 week primary, that's a team of at least 25 people. For an organization of just a handful of engineers, how do they make on call work? A single on-call rotation would stretch the team to its limits, and it's likely that certain domains would only have a single Subject Matter Expert. [1] https://response.pagerduty.com/before/different_roles/ https://response.pagerduty.com/before/different_roles/
- zufallsheld 8y ago
  In my old team we managed our on call rotation for years with five persons. One primary, no secondary, 7 days in a row. Of course we had to arrange ourselves because of vacation and sick leave but we always managed. We also had an organizational rule that forbade being on call for more than ten days in a row or on two consecutive weekends.
  - simonsaidit 8y ago
    I was on call for years for a global lottery software and services provider in a 4 man rotation 1 week each. Took people 3-5 years to be ready to take on this responsibility. The last few years i was alone as people left with stress and i was 24/7 365. The pay was good but the day my new boss told me he didet get paid so i could escalate at nite and nobody could find a solution for months i started to look for new oppertunities. They hadent tried to prepare new people for years but managed to get a guy who was not ready to take the call but it also resulted in major incident and data loss. By the time i got out i had constant tinitus so i could barely hear on one ear from the stress so Im pretty happy Im done with it now but Sometimes miss the action ;)
    - fencepost 8y ago
      the day my new boss told me he didet get paid so i could escalate at nite Great, I want to go hourly, or if staying salaried I want to be paid as if hourly. That means I'm paid 33% for all time that I keep myself available while on-call, and paid 2x (or more?) for time spent actually responding because it's after-hours overtime. If I'm going to be 24/7/365 with no backup and get crap about "I'm not paid for you to escalate to me at night," you're going to pay through the nose.

246 points by blopeur 8y ago

Awesome. Will be reading these in short order. Can anyone recommend other good incident response resources (that are relevant in 2019)?

brooksgarrett 8y ago
Firefighter for over 10 years and an IT career spanning Red Team/Blue Team and now SRE, I can't recommend enough taking the FEMA Independent Study course in Incident Command. https://training.fema.gov/is/courseoverview.aspx?code=IS-100.c https://training.fema.gov/is/courseoverview.aspx?code=IS-100... Even if you don't adopt the system it will help you frame and understand how complex IR can be. Having a scalable system to a) grow with resources and b) grow with external interactions is crucial to have BEFORE you need it.
twunde 8y ago
Atlassian's handbook is pretty good: https://www.atlassian.com/software/jira/ops/handbook https://www.atlassian.com/software/jira/ops/handbook
richadams 8y ago
There are some good additional resources referenced in the docs here: https://response.pagerduty.com/resources/reading/ https://response.pagerduty.com/resources/reading/ Specifically, Google's SRE books are particularly useful (https://landing.google.com/sre/books/ https://landing.google.com/sre/books/) along with the book "Incident Management for Operations" (http://shop.oreilly.com/product/0636920036159.do http://shop.oreilly.com/product/0636920036159.do) and Etsy's Debriefing Facilitation Guide (http://extfiles.etsy.com/DebriefingFacilitationGuide.pdf http://extfiles.etsy.com/DebriefingFacilitationGuide.pdf). The book "Comparative Emergency Management" (https://training.fema.gov/hiedu/aemrc/booksdownload/compemmgmtbookproject/ https://training.fema.gov/hiedu/aemrc/booksdownload/compemmg...) is also quite interesting, as it compares the emergency management practices of about 30 different countries.

johnmarcus 8y ago

let me guess...it includes getting paged as a very important step. j/k, this is very well written. Good for sharing to those whom have not been on call at a good org before.

lifekaizen 8y ago

This is wonderful, especially the section on what didn't work ("Anti patterns," bottom of page). This one in particular feels like good advice to startup founders as their company grows: > Trying to take on multiple roles.# In past PagerDuty incidents, we've had instances where the Incident Commander has started to assume the Subject Matter Expert role and attempted to solve the problem themselves. This usually happens when the IC is an engineer in their day-to-day role. They are in an incident where the cause appears to be a system they know very well and have the requisite knowledge to fix. Wanting to solve the incident quickly, the IC will start to try and solve the problem. Sometimes you might get lucky and it will resolve the incident, but most of the time the immediately visible issue isn't necessarily the underlying cause of the incident. By the time that becomes apparent, you have an Incident Commander who is not paying attention to the other systems and is just focussed on the one problem in front of them. This effectively means there's no incident commander, as they would be busy trying to fix the problem. Inevitably, the problem turns out to be much bigger than anticipated and the response has become completely derailed.

cco 8y ago
An officer does not fire their weapon, they assess and respond to the situation.

uasm 8y ago

> "A guide to being on-call... we all have lives which might get in the way of on-call time" How dare those engineers live a life. They're on-call goddammit.

lwf 8y ago
Life gets in the way of scheduled work (on-call). The section you're quoting is about enabling on-call people to have lives without on-call constraints. It's encouraging people to be considerate and cover a colleague's shift if they want to go out and do something, and you're free.
- uasm 8y ago
  > "Life gets in the way of scheduled work (on-call)." Scheduling 24h work days will inevitably cause "life to get in the way of work". On-calls commonly take a week or two at a time. Stop de-legitimizing work-life seperation.
  - lwf 8y ago
    You're reading way more into both the PagerDuty guide and my comments than was intended. Work-life balance is great. Personally, I don't feel like I need to respond to queries on nights or weekends, take plenty of vacation, and feel empowered to push back at anybody going out of those bounds. Then again, I also work at a company where your colleagues will yell at you if you show up on work chat when you're supposed to be on PTO :) That said, if you're on an on-call shift, and something comes up, that's "life getting in the way". And that's fine. I have separate issues, personally, with how many companies compensate employees for on-call work. But it is an on-call shift, a concept that exists in basically every industry where you're providing a timely service. (plumbers, airline pilots, systems administration...) Unless you're large enough to implement follow-the-sun, you're going to need someone to cover the evenings.
    - deleted 8y ago
      [deleted]

gberger 8y ago

Curious how PagerDuty handles incident response when PagerDuty itself is down.

kenrose 8y ago
I used to work at PD. When I was there, they followed a lot of (all?) of these guidelines. The app’s ownership was spread across teams, so depending which section was affected would page certain teams. If it was SEV-2 or higher, that would page an IC and the group of primary on-calls to begin triage. Other SMEs were looped in as necessary. The anti patterns section is quite authentic. PD had very healthy discussions internally about the topics covered (eg, when it made sense to stop paging everyone, how to make people feel it was OK to drop off the call if they weren’t adding anything). In terms of the actual “how do they page people if PD is down?”, they had some backup monitoring systems that could SMS / phone on-calls directly. As a piece of software though, PD is pretty resilient, so it was rare to have an outage that affected everything so badly that they had to rely on these secondary systems.
- wbronitsky 8y ago
  I can verify this as well. I worked on a separate team from Ken, but while we were there, we faced a lot of issues that led the team to decide upon and codify these rules. They are pretty tried and true, if only because they came out of a lot of iteration and testing. Hope you are doing well, Ken!
techslave 8y ago
isn't that the article? why would PD write a summary of IRP that they use for ... what if not themselves?
- gberger 8y ago
  I meant to ask “how do they page people if PD is down?”, which kenrose included in this reply.

C1sc0cat 8y ago

That's interesting it seems to assume there is no 24/7 ops coverage on site who triage before escalating the call to actual on call staff. This is from being the lead on call for the UK's tymnet billing system normally if I got called some initial triage

sambull 8y ago
Yea pretty much from my experience now in the US, most private companies use their salaried employees and wake them up multiple times a night as sometimes with customer directly on the other end of the line. Pagerduty enables this to work. This is considered now normal for smallish private orgs, shit my spot just got bought by a billions dollar defense contractor I still get woken up at 3am with a end user having printer issues.. and I work as a devops engineer.
- softawre 8y ago
  If you're not exaggerating you need to nip that in the bud. Printer issues at 3am? Have two classes of alerts and send printer issues to somebody else, or worst case email notification for the morning.
  - sambull 8y ago
    Sev1 for us, end users are 24/7 manufacturers and they consider this a business stoppage, this would be SLA downtime. They shoved old school COTs into AWS and called it cloud. Which ugh yea.
- wil421 8y ago
  Do your incidents have Priorites (Severity in the article)? At my company a printer would never be a P1 to P2 therefore no one gets paged at 3am due to printer issues. It’s not full proof and my team did get a P1 due to a field length being too short but it was during normal business hours. It was quickly changed to a P4 and the person who opened it got a lesson on Incident priorities.
  - 8y ago

TagAsDelusion 8y ago

reacting means you have not planned correctly this is fundamentally perverse

luhn 8y ago

One thing I've been wondering about is how smaller organizations handle on call. Looking at the roles laid out [1] and assuming an on-call schedule of 4 weeks off, 1 week secondary, and 1 week primary, that's a team of at least 25 people. For an organization of just a handful of engineers, how do they make on call work? A single on-call rotation would stretch the team to its limits, and it's likely that certain domains would only have a single Subject Matter Expert. [1] https://response.pagerduty.com/before/different_roles/ https://response.pagerduty.com/before/different_roles/

zufallsheld 8y ago
In my old team we managed our on call rotation for years with five persons. One primary, no secondary, 7 days in a row. Of course we had to arrange ourselves because of vacation and sick leave but we always managed. We also had an organizational rule that forbade being on call for more than ten days in a row or on two consecutive weekends.
- simonsaidit 8y ago
  I was on call for years for a global lottery software and services provider in a 4 man rotation 1 week each. Took people 3-5 years to be ready to take on this responsibility. The last few years i was alone as people left with stress and i was 24/7 365. The pay was good but the day my new boss told me he didet get paid so i could escalate at nite and nobody could find a solution for months i started to look for new oppertunities. They hadent tried to prepare new people for years but managed to get a guy who was not ready to take the call but it also resulted in major incident and data loss. By the time i got out i had constant tinitus so i could barely hear on one ear from the stress so Im pretty happy Im done with it now but Sometimes miss the action ;)
  - fencepost 8y ago
    the day my new boss told me he didet get paid so i could escalate at nite Great, I want to go hourly, or if staying salaried I want to be paid as if hourly. That means I'm paid 33% for all time that I keep myself available while on-call, and paid 2x (or more?) for time spent actually responding because it's after-hours overtime. If I'm going to be 24/7/365 with no backup and get crap about "I'm not paid for you to escalate to me at night," you're going to pay through the nose.