3 ms·
Most places I've seen have similar times to that, all except the public communication bit. Lots of companies are wary about letting engineers declare downtime
by londons_explore 8y ago
Most places I've seen have similar times to that, all except the public communication bit.
Lots of companies are wary about letting engineers declare downtime directly - it could have costs down the line giving customers refunds etc.
Getting legal, PR, and 3 levels of management to all agree on a message before it gets published is a recipe for the downtime notice taking hours to be published.
- donavanm 8y agoDo you have a NOC or similar dedicated group? Both initial engagement and their escalation path had a ~2 minute response time. Paging oncall team member, vpn connection, etc, takes ~5-10 in my experience. Similarly their incident manager apparently assessed the situation and further escalated in ~2 minutes as well. Thats pretty dang fast for coming up to speed on a as yet to be diagnosed major incident. WRT to public communication, yeah. Overly onerous and adhoc seems to be the norm. But we do have “analysts” publically using status post count as inverse evidence of reliability. Creates some real bad incentives for the actual customers.
- londons_explore 8y agoFor a core service, github is sure to have a 5 minute human SLA from dedicated SRE's who have an on call rotation etc. That means someone is already at their computer, logged in, and ready to go. They'll typically be investigating within 30 seconds. Probably within another 30 seconds that human sees many alerts coming in that don't look like false alarms and hits the 'escalate' button. If it's within office hours, anything that looks major will usually have a bunch of SRE's on the case within 3 or 4 minutes. Usually in various roles from 'I acknowledged this page, it's my job to fix it!' to 'I'm just laughing at your misfortune, but paying a little attention just incase the broken thing turns out to be that change I made yesterday and I have to take over mitigation efforts'.