4 ms·
I'm curious about big outages like this in big internet corps. Does anyone know if SREs in Europe fixed the problem or it relied on people in Mountain View? Wh
by x87678r 6y ago
I'm curious about big outages like this in big internet corps.
Does anyone know if SREs in Europe fixed the problem or it relied on people in Mountain View? When an outage this big hits do devs get involved?
I've worked as an SRE and it sucks to be fixing developer's bugs in the middle of the night.
- mikelward 6y agoYes, SRE teams typically have a sister SRE team in another continent and time zone.
- deleted 6y ago[deleted]
- asdfasgasdgasdg 6y agoI assume this is in the SRE book, but a tier one product like the identity service will have global SRE coverage (i.e. at least three SRE teams so that there is always an SRE group for whom it is daytime holding the pager). Devs are often involved in diagnosis, but are less often required for mitigation, as the mitigation is almost always to revert whatever change caused the problem. This is a simplification of course, but it gives an idea of the general pattern.
- smueller1234 6y agoI won't comment on the incident, but I can tell you that we have two, not three, SRE sibling teams each. That still gives awake-time coverage, but not working-hours coverage. We simply pay folks for the time spent oncall outside of working hours. (Google SRE)
- ryanobjc 6y ago4am is noon Europe time, so the sres in Europe would have gotten the page and been on top of their game. They fixed it pretty quick. Of course in a global outage, nothing is fast enough.
- guenthert 6y agoThey found the root cause within 20 minutes. I doubt one of the original developers could have been involved that quickly.