4 ms·

The fact one service had an issue. And not only did it take said service down globally, it also impacted dozens of other services...globally. A good infra arch

The fact one service had an issue. And not only did it take said service down globally, it also impacted dozens of other services...globally. A good infra architecture has the “blast radius” of any issue confined to only part of your infra fleet. Avoiding the global outage. Think of it like a navy ship. When a mussel breaches the hull, the ship is designed to contain the leak in a single section. Avoiding the entire ship sinking. Similar pattern is desired in your service and compute infrastructure. The outage google just faced is equivalent to a single missel taking down an entire navy fleet.

herodoturtle 6y ago
I love this analogy, thanks for visuals. > The outage google just faced is equivalent to a single missel taking down an entire navy fleet. I really thought you were going to say "is equivalent to a single missile taking down an entire ship" - but your extreme analogy made even more sense in this context.
oh_sigh 6y ago
How could one design a system that would not fail when something as critical and cross-cutting as capacity or identity management failed?
- ZiiS 6y ago
  For example youtube should have defaulted to the logged out experiance. No one forced them to use the same identity platform for completly unrelated services. Gmail dose raise hard issues but people logged in should not have needed to recontact a auth service.
- coddle-hark 6y ago
  You can limit the damage though; in theory there’s no reason why authentication needs to be centralised. You could split the backend up into multiple authentication services and “bucket” the auth requests per email address or something. So essentially make a bunch of different authentication services that can fail independently. I’m sure they have good reasons for not doing that though.
pjc50 6y ago
> When a mussel breaches the hull I knew barnacles were a problem, I hadn't realised there were other dangerous bivalves. :)
- baskire 6y ago
  Haha, that gave me a good laugh on my iPhone keyboard swipe typo.
joshxyz 6y ago
Petition to have Blast Radius Awards per year among top companies pls
deleted 6y ago
[deleted]

herodoturtle 6y ago

I love this analogy, thanks for visuals. > The outage google just faced is equivalent to a single missel taking down an entire navy fleet. I really thought you were going to say "is equivalent to a single missile taking down an entire ship" - but your extreme analogy made even more sense in this context.

oh_sigh 6y ago

How could one design a system that would not fail when something as critical and cross-cutting as capacity or identity management failed?

ZiiS 6y ago
For example youtube should have defaulted to the logged out experiance. No one forced them to use the same identity platform for completly unrelated services. Gmail dose raise hard issues but people logged in should not have needed to recontact a auth service.
coddle-hark 6y ago
You can limit the damage though; in theory there’s no reason why authentication needs to be centralised. You could split the backend up into multiple authentication services and “bucket” the auth requests per email address or something. So essentially make a bunch of different authentication services that can fail independently. I’m sure they have good reasons for not doing that though.

pjc50 6y ago

> When a mussel breaches the hull I knew barnacles were a problem, I hadn't realised there were other dangerous bivalves. :)

baskire 6y ago
Haha, that gave me a good laugh on my iPhone keyboard swipe typo.

joshxyz 6y ago

Petition to have Blast Radius Awards per year among top companies pls

deleted 6y ago

[deleted]