7 ms·
When I was at Amazon in 2001 outages were done via everyone sitting at their desks on a POTS conference call. The engineers working the issue would be spread o
by lamontcg 5y ago
When I was at Amazon in 2001 outages were done via everyone sitting at their desks on a POTS conference call. The engineers working the issue would be spread out around building at least, and the people who were actually on-call would generally be at home. Remote hands were often in a datacenter that was on the other side of continent.
The whole idea of piling everyone into a "war room" doesn't really scale anyway.
The bigger problem here was just no out of band comms.
- kristjansson 5y agoYeah, knocking out their backup IRC (or maybe it was the authentication to the backup IRC?) along with everything else was the key.
- duskwuff 5y agoWord-of-mouth I've heard is that the backup IRC server was off-site working fine. The problem was that, because it was only used in emergencies and wasn't hooked into FB's central authentication, a lot of employees simply didn't remember how to log in to it.