3 ms·

S3 is (for want of a better term) a legacy system in AWS, having been around since pretty much the beginning. A series of choices made when the idea that there

S3 is (for want of a better term) a legacy system in AWS, having been around since pretty much the beginning. A series of choices made when the idea that there would ever be more than two regions, leave them unfortunately stuck having to tie regions together to a limited degree (bucket names are global). Beyond those bucket mutations, everything else is purely regional scoped. I'm _really_ surprised to see that GCP has made the same mistakes though and had things tied together. I'm not convinced there's a good excuse. AWS/S3 is understandable. No one knew what things would look like. Not one engineer would have made the same decisions then if they'd known what the future was for the platform. Google came to the cloud market much later. Their engineers should have seen the issues with AWS and the problems that interdependence cause, and specifically architected isolated infrastructure (which from this post-mortem it seems like they didn't). That's both remarkable and disappointing to see.

tybit 8y ago
GCP may have come later than AWS but googles internal blob storage could of predated AWS for all we know.
nerpderp82 8y ago
The worlds infrastructure is inherently fragile. It should be difficult to take down these large distributed systems, but they are over-coupled.
- m0zg 8y ago
  It's very easy, apparently, to take them down by a configuration change. Happens several times a year and takes hours (and untold millions of dollars in productivity loss) to recover from.
bogomipz 8y ago
>"A series of choices made when the idea that there would ever be more than two regions, leave them unfortunately stuck having to tie regions together to a limited degree (bucket names are global)." Could you elaborate on those choices? Also what do you mean by they're "stuck having to tie regions together"?
- Twirrim 8y ago
  The main thing is that bucket namespace is global. There is only one bucket "foo" worldwide. That means that every bucket mutation event (e.g. bucket deletion or creation) requires co-ordination between regions. When that big us-east-1 region outage occurred a year or so back every other region worked fine except for bucket mutations.

tybit 8y ago

GCP may have come later than AWS but googles internal blob storage could of predated AWS for all we know.

nerpderp82 8y ago

The worlds infrastructure is inherently fragile. It should be difficult to take down these large distributed systems, but they are over-coupled.

m0zg 8y ago
It's very easy, apparently, to take them down by a configuration change. Happens several times a year and takes hours (and untold millions of dollars in productivity loss) to recover from.

bogomipz 8y ago

>"A series of choices made when the idea that there would ever be more than two regions, leave them unfortunately stuck having to tie regions together to a limited degree (bucket names are global)." Could you elaborate on those choices? Also what do you mean by they're "stuck having to tie regions together"?

Twirrim 8y ago
The main thing is that bucket namespace is global. There is only one bucket "foo" worldwide. That means that every bucket mutation event (e.g. bucket deletion or creation) requires co-ordination between regions. When that big us-east-1 region outage occurred a year or so back every other region worked fine except for bucket mutations.