3 ms·
A strangely prosaic failure. A large portion of the entries there seem to be the on-call engineer simply trying to figure out what the failover procedure actual
by _delirium 17y ago
A strangely prosaic failure. A large portion of the entries there seem to be the on-call engineer simply trying to figure out what the failover procedure actually is. Once he finally gets a copy of it, it takes 16 minutes to get back up:
9:53 AM - After engineering team consultation with the relevant
engineers, now online, the correct unplanned failover procedure
operations document is confirmed, and is ready to be used by
the oncall engineer. The actual unplanned failover procedure for
reads and writes begins.
10:09 AM - The unplanned failover procedure completes, without
any problems. Traffic resumes serving normally, read and write.
App Engine is considered up at this time.
- ww520 17y agoIt's probably just a matter of configuring the routers to redirect traffic to the backup data center. All the servers are already running in standby in the backup data center. The only thing tricky is to hold all writes on backup, and flush all pending updates from primary to backup. That they did: they made backup in readonly mode, and slowly turned on read/write after a while. I don't understand why they lost so much data. 0.00002% is a lot considering the dataset is really large. Shouldn't there be replication from primary to backup? I assume Google is rich enough to have multiple links and fatpipes.