3 ms·

I find their incident management times pretty impressive. 2 minutes to detect & alert, ~2+ minutes response, 10 minutes to initial triage, 15 minutes to interna

I find their incident management times pretty impressive. 2 minutes to detect & alert, ~2+ minutes response, 10 minutes to initial triage, 15 minutes to internal change control, 17 minutes to escalation & public communication, 19 minutes to major incident escalation & broad engagement, 73 minutes to remediation start & further public communication. Initial triage inside 10 minutes then change control, escalation, major incident, and public communication within another 9 minutes. That's hard to beat with humans in the loop.

londons_explore 8y ago
Most places I've seen have similar times to that, all except the public communication bit. Lots of companies are wary about letting engineers declare downtime directly - it could have costs down the line giving customers refunds etc. Getting legal, PR, and 3 levels of management to all agree on a message before it gets published is a recipe for the downtime notice taking hours to be published.
- donavanm 8y ago
  Do you have a NOC or similar dedicated group? Both initial engagement and their escalation path had a ~2 minute response time. Paging oncall team member, vpn connection, etc, takes ~5-10 in my experience. Similarly their incident manager apparently assessed the situation and further escalated in ~2 minutes as well. Thats pretty dang fast for coming up to speed on a as yet to be diagnosed major incident. WRT to public communication, yeah. Overly onerous and adhoc seems to be the norm. But we do have “analysts” publically using status post count as inverse evidence of reliability. Creates some real bad incentives for the actual customers.
  - londons_explore 8y ago
    For a core service, github is sure to have a 5 minute human SLA from dedicated SRE's who have an on call rotation etc. That means someone is already at their computer, logged in, and ready to go. They'll typically be investigating within 30 seconds. Probably within another 30 seconds that human sees many alerts coming in that don't look like false alarms and hits the 'escalate' button. If it's within office hours, anything that looks major will usually have a bunch of SRE's on the case within 3 or 4 minutes. Usually in various roles from 'I acknowledged this page, it's my job to fix it!' to 'I'm just laughing at your misfortune, but paying a little attention just incase the broken thing turns out to be that change I made yesterday and I have to take over mitigation efforts'.

londons_explore 8y ago

Most places I've seen have similar times to that, all except the public communication bit. Lots of companies are wary about letting engineers declare downtime directly - it could have costs down the line giving customers refunds etc. Getting legal, PR, and 3 levels of management to all agree on a message before it gets published is a recipe for the downtime notice taking hours to be published.

donavanm 8y ago
Do you have a NOC or similar dedicated group? Both initial engagement and their escalation path had a ~2 minute response time. Paging oncall team member, vpn connection, etc, takes ~5-10 in my experience. Similarly their incident manager apparently assessed the situation and further escalated in ~2 minutes as well. Thats pretty dang fast for coming up to speed on a as yet to be diagnosed major incident. WRT to public communication, yeah. Overly onerous and adhoc seems to be the norm. But we do have “analysts” publically using status post count as inverse evidence of reliability. Creates some real bad incentives for the actual customers.
- londons_explore 8y ago
  For a core service, github is sure to have a 5 minute human SLA from dedicated SRE's who have an on call rotation etc. That means someone is already at their computer, logged in, and ready to go. They'll typically be investigating within 30 seconds. Probably within another 30 seconds that human sees many alerts coming in that don't look like false alarms and hits the 'escalate' button. If it's within office hours, anything that looks major will usually have a bunch of SRE's on the case within 3 or 4 minutes. Usually in various roles from 'I acknowledged this page, it's my job to fix it!' to 'I'm just laughing at your misfortune, but paying a little attention just incase the broken thing turns out to be that change I made yesterday and I have to take over mitigation efforts'.