3 ms·

It’s still batshit insane to me that Google Fi had any infrastructure overlap with Gmail. When I was there these grand unified base layer systems everything els

It’s still batshit insane to me that Google Fi had any infrastructure overlap with Gmail. When I was there these grand unified base layer systems everything else was built on top of were a point of pride. From a risk perspective though it was sheer stupidity in retrospect. As Google continues to atrophy and suffers attrition of the original people that built those systems, the probability creeps up more and more that someone will one day cause a catastrophic global outage of systems that should share absolutely no dependencies but do because “it scales”. If Google made auto pilot for airplanes, it would all be on a centralized SaaS with white papers published in academic conferences about the elaborate systems designed to ensure server crashes don’t impact it. Nobody will ever complain about how all of those guarantees depend on chubby, big table, etc.

Guvante 3y ago
At my not Google job we talk about "what happens if a meteor hits a DC". We agree that that is so rare that as long as there are buttons we can push to recover after a reasonable timeframe that is an acceptable risk, we don't need a fully automatic way to recover from that. However your SRE teams needs a way to recover without intervention which is why there is talk of backups. BTW even using different cloud providers isn't enough to avoid a DC outage necessarily. No amount of redundancy can protect you from it beyond a ton of services which intentionally slice off access to the DC leading to the risk of that happening accidentally which is its own risk.
- kortilla 3y ago
  But that’s incredibly stupid. A meteor hitting a DC is an intentionally dumb way to eclipse the much more likely risks of thousands of other things that can wipe out a DC. I’ve been involved in such discussions and on the surface they seem reasonable but it turns into an easy reason to write of DC being wiped out. In comparison there are far more likely reasons for a DC to be wiped out effectively permanently. Data centers at the base of WTC 1 and 2 are good examples. The myriad of targeted attacks on the power grid are also prime examples of attacks that would cripple a data center for weeks if they were in the cross chairs. The Cascadia subduction zone has a much higher probability of wiping out all of them in the PNW simultaneously than a meteor hitting a single one. Looking at it from a different perspective, there are other teams at Google who understand how sensitive their data centers are and they act appropriately. The list of data center locations is not public. A motivated group with long range rifles purchase from Walmart could wipe out a Google data center. Google is very bad at modeling long tail events.
  - Guvante 3y ago
    Meteor isn't real. It is literally just a metaphor and more fun than "we lost power" or "egress died". Anybody claiming it is impossible is silly. You also won't lose a DC a year so you need some strict uptime guarantees to justify automatic roll over. Prepare to avoid dataloss but needing SRE to change over is fine if you can stomach a hour of downtime.
    - cutemonster 3y ago
      > Meteor isn't real Yes, just like "bus factor" isn't about buses

Guvante 3y ago

At my not Google job we talk about "what happens if a meteor hits a DC". We agree that that is so rare that as long as there are buttons we can push to recover after a reasonable timeframe that is an acceptable risk, we don't need a fully automatic way to recover from that. However your SRE teams needs a way to recover without intervention which is why there is talk of backups. BTW even using different cloud providers isn't enough to avoid a DC outage necessarily. No amount of redundancy can protect you from it beyond a ton of services which intentionally slice off access to the DC leading to the risk of that happening accidentally which is its own risk.

kortilla 3y ago
But that’s incredibly stupid. A meteor hitting a DC is an intentionally dumb way to eclipse the much more likely risks of thousands of other things that can wipe out a DC. I’ve been involved in such discussions and on the surface they seem reasonable but it turns into an easy reason to write of DC being wiped out. In comparison there are far more likely reasons for a DC to be wiped out effectively permanently. Data centers at the base of WTC 1 and 2 are good examples. The myriad of targeted attacks on the power grid are also prime examples of attacks that would cripple a data center for weeks if they were in the cross chairs. The Cascadia subduction zone has a much higher probability of wiping out all of them in the PNW simultaneously than a meteor hitting a single one. Looking at it from a different perspective, there are other teams at Google who understand how sensitive their data centers are and they act appropriately. The list of data center locations is not public. A motivated group with long range rifles purchase from Walmart could wipe out a Google data center. Google is very bad at modeling long tail events.
- Guvante 3y ago
  Meteor isn't real. It is literally just a metaphor and more fun than "we lost power" or "egress died". Anybody claiming it is impossible is silly. You also won't lose a DC a year so you need some strict uptime guarantees to justify automatic roll over. Prepare to avoid dataloss but needing SRE to change over is fine if you can stomach a hour of downtime.
  - cutemonster 3y ago
    > Meteor isn't real Yes, just like "bus factor" isn't about buses