3 ms·
Disclosure: I work on Google Cloud. I’ve always enjoyed this quote, but my problem with [the description of] this outage is the third-party dependency. Packet
by boulos 8y ago
Disclosure: I work on Google Cloud.
I’ve always enjoyed this quote, but my problem with [the description of] this outage is the third-party dependency.
Packets can’t get from your cloud provider to downstream users of CenturyLink? That’s fair.
Your cloud provider can’t send packets to/from CenturyLink, so they nuke your database? I literally don’t understand.
Is the service described actually a third-party service that’s been white boxed? (I mean this in the most honest way possible. I do not understand the details, and I found the article surprising).
- m0zg 8y agoYes, sounds like bad design to me. But the uber-problem here is with complexity. Google had its own share of screwups (most of which users never saw, but some of which they did), albeit not quite to the extent of destroying data at scale (and when GMail did destroy data at scale, it had tape backups, so it was never actually lost). The root cause of nearly all of these screwups is that large, complex systems can't be fully understood or observed, and that a good chunk of knowledge about such systems is institutional, rather than explicit. So from time to time people _will_ make assumptions that don't match reality, and reality will punish them for it. Which is what, I strongly suspect, happened in this case.