5 ms·

The root cause was an issue in our automated quota management system which reduced capacity for Google's central identity management system, causing it to retur

The root cause was an issue in our automated quota management system which reduced capacity for Google's central identity management system, causing it to return errors globally. .... so much for limiting Blast Radius

anshumankmr 6y ago
What do you mean by blast radius?
- talonx 6y ago
  The area (in this case, number of services or regions) impacted by an outage.
- baskire 6y ago
  The fact one service had an issue. And not only did it take said service down globally, it also impacted dozens of other services...globally. A good infra architecture has the “blast radius” of any issue confined to only part of your infra fleet. Avoiding the global outage. Think of it like a navy ship. When a mussel breaches the hull, the ship is designed to contain the leak in a single section. Avoiding the entire ship sinking. Similar pattern is desired in your service and compute infrastructure. The outage google just faced is equivalent to a single missel taking down an entire navy fleet.
  - herodoturtle 6y ago
    I love this analogy, thanks for visuals. > The outage google just faced is equivalent to a single missel taking down an entire navy fleet. I really thought you were going to say "is equivalent to a single missile taking down an entire ship" - but your extreme analogy made even more sense in this context.
  - oh_sigh 6y ago
    How could one design a system that would not fail when something as critical and cross-cutting as capacity or identity management failed?
    - ZiiS 6y ago
      For example youtube should have defaulted to the logged out experiance. No one forced them to use the same identity platform for completly unrelated services. Gmail dose raise hard issues but people logged in should not have needed to recontact a auth service.
    - coddle-hark 6y ago
      You can limit the damage though; in theory there’s no reason why authentication needs to be centralised. You could split the backend up into multiple authentication services and “bucket” the auth requests per email address or something. So essentially make a bunch of different authentication services that can fail independently. I’m sure they have good reasons for not doing that though.
  - pjc50 6y ago
    > When a mussel breaches the hull I knew barnacles were a problem, I hadn't realised there were other dangerous bivalves. :)
    - baskire 6y ago
      Haha, that gave me a good laugh on my iPhone keyboard swipe typo.
  - joshxyz 6y ago
    Petition to have Blast Radius Awards per year among top companies pls
  - deleted 6y ago
    [deleted]
- deleted 6y ago
  [deleted]
deleted 6y ago
[deleted]
Ekaros 6y ago
If all of your services make use of one service, maybe that should be treated as very special case...
baskire 6y ago
Curious how Spotify and other enterprise GCP users have mitigated GCPs repeated global outages. This isn’t their first nor will it be their last.
- bobviolier 6y ago
  The Google Cloud was almost unaffected, during the outage you could not log into the cloud console or use tools like gcloud - but the services itself (vms, gke, pubsub, etc) kept working throughout the outage.
  - wdb 6y ago
    That's not correct, you wouldn't be able to connect these services after restart. E.g. you wouldn't be able to auth to Cloud SQL or Datastore. Stopping development
- jrockway 6y ago
  Services hosted on GCP didn't lose any traffic during this incident. This was a control plane outage, not a data plane outage. That means that if you needed to go in and adjust something to resolve your own outage, you were in trouble, but if your services were healthy, nothing bad happened. As the time of the outage increases, service health probably trends to zero without being able to manage the service... but for a few hours, it's not a disaster. Usually.
semi-extrinsic 6y ago
On the flipside, isn't this just a case where you are bound by the CAP theorem? If you want your central identity management database to be consistent and available, as I imagine Google do, it can't be parallel?
- simiones 6y ago
  I assume you meant "it can't be partition-tolerant"?
  - semi-extrinsic 6y ago
    Yes, sorry, sloppy of me.
cowsandmilk 6y ago
Note IAM is also global in AWS, perhaps the only global service other than those that are effectively global by requiring you to use us-east-1

anshumankmr 6y ago

What do you mean by blast radius?

talonx 6y ago
The area (in this case, number of services or regions) impacted by an outage.
baskire 6y ago
The fact one service had an issue. And not only did it take said service down globally, it also impacted dozens of other services...globally. A good infra architecture has the “blast radius” of any issue confined to only part of your infra fleet. Avoiding the global outage. Think of it like a navy ship. When a mussel breaches the hull, the ship is designed to contain the leak in a single section. Avoiding the entire ship sinking. Similar pattern is desired in your service and compute infrastructure. The outage google just faced is equivalent to a single missel taking down an entire navy fleet.
- herodoturtle 6y ago
  I love this analogy, thanks for visuals. > The outage google just faced is equivalent to a single missel taking down an entire navy fleet. I really thought you were going to say "is equivalent to a single missile taking down an entire ship" - but your extreme analogy made even more sense in this context.
- oh_sigh 6y ago
  How could one design a system that would not fail when something as critical and cross-cutting as capacity or identity management failed?
  - ZiiS 6y ago
    For example youtube should have defaulted to the logged out experiance. No one forced them to use the same identity platform for completly unrelated services. Gmail dose raise hard issues but people logged in should not have needed to recontact a auth service.
  - coddle-hark 6y ago
    You can limit the damage though; in theory there’s no reason why authentication needs to be centralised. You could split the backend up into multiple authentication services and “bucket” the auth requests per email address or something. So essentially make a bunch of different authentication services that can fail independently. I’m sure they have good reasons for not doing that though.
- pjc50 6y ago
  > When a mussel breaches the hull I knew barnacles were a problem, I hadn't realised there were other dangerous bivalves. :)
  - baskire 6y ago
    Haha, that gave me a good laugh on my iPhone keyboard swipe typo.
- joshxyz 6y ago
  Petition to have Blast Radius Awards per year among top companies pls
- deleted 6y ago
  [deleted]
deleted 6y ago
[deleted]

deleted 6y ago

[deleted]

Ekaros 6y ago

If all of your services make use of one service, maybe that should be treated as very special case...

baskire 6y ago

Curious how Spotify and other enterprise GCP users have mitigated GCPs repeated global outages. This isn’t their first nor will it be their last.

bobviolier 6y ago
The Google Cloud was almost unaffected, during the outage you could not log into the cloud console or use tools like gcloud - but the services itself (vms, gke, pubsub, etc) kept working throughout the outage.
- wdb 6y ago
  That's not correct, you wouldn't be able to connect these services after restart. E.g. you wouldn't be able to auth to Cloud SQL or Datastore. Stopping development
jrockway 6y ago
Services hosted on GCP didn't lose any traffic during this incident. This was a control plane outage, not a data plane outage. That means that if you needed to go in and adjust something to resolve your own outage, you were in trouble, but if your services were healthy, nothing bad happened. As the time of the outage increases, service health probably trends to zero without being able to manage the service... but for a few hours, it's not a disaster. Usually.

semi-extrinsic 6y ago

On the flipside, isn't this just a case where you are bound by the CAP theorem? If you want your central identity management database to be consistent and available, as I imagine Google do, it can't be parallel?

simiones 6y ago
I assume you meant "it can't be partition-tolerant"?
- semi-extrinsic 6y ago
  Yes, sorry, sloppy of me.

cowsandmilk 6y ago

Note IAM is also global in AWS, perhaps the only global service other than those that are effectively global by requiring you to use us-east-1