3 ms·
Google didn't have SRO. The closest thing was Traffic Team's oncall; Which was stressful, for sure, and hired some dedicated alcoholics, but it wasn't anything
by GauntletWizard 4y ago
Google didn't have SRO. The closest thing was Traffic Team's oncall; Which was stressful, for sure, and hired some dedicated alcoholics, but it wasn't anything like Facebooks (I knew people on both teams during my time at each). Google had a constant stream of alerts that someone was aware of and handling. Facebook had a constant stream of fire that someone was beating back the flames of.
SRO is madness. SRO is admitting that you're enough underwater and there's enough unexpected critical events to dedicate a team to burn out on it. There was a lot of spend in overcapacity and failover automation testing that kept it that way, and explicit mandates by a C-Suite and VP to keep keeping it that way. More importantly, though, it was baked into the architecture.
Everyone's talking this week about how Twitter will go up in flames in the next few days; I don't think so. I think they're closer to Google in that respect. There's a lot of automation that will keep things alive and a lot of redundancy that will handle the immediate problems of having a lot of the team leave. I actually think the company can continue to run indefinitely if they just follow the operations manuals and replace failed drives and machines. But I think there will be a major outage when they try to launch a new feature and fail.
- MrAwesome 4y agoAuthor here - a few months after the time of this story, we reoriented around a single purpose for the team, "Automate ourselves out of existence". It took a few months to remove all of the dependencies, tag all the alarms, train up all the team-specific oncalls, but we managed to do it, and I had the great honor of taking the very last SRO oncall on March 31, 2014 :)