4 ms·

Not sure why this gets downvoted. This is an entirely valid comments. It's getting increasingly obvious that despite being highly distributed, cloud systems ru

Not sure why this gets downvoted. This is an entirely valid comments. It's getting increasingly obvious that despite being highly distributed, cloud systems run by Google, Amazon and other big cloud companies are inherently centralized. If nothing else, they have centralized controls. A decentralized system can, of course, fail, but it would never have "configuration change" as the root cause.

Twirrim 8y ago
S3 is (for want of a better term) a legacy system in AWS, having been around since pretty much the beginning. A series of choices made when the idea that there would ever be more than two regions, leave them unfortunately stuck having to tie regions together to a limited degree (bucket names are global). Beyond those bucket mutations, everything else is purely regional scoped. I'm _really_ surprised to see that GCP has made the same mistakes though and had things tied together. I'm not convinced there's a good excuse. AWS/S3 is understandable. No one knew what things would look like. Not one engineer would have made the same decisions then if they'd known what the future was for the platform. Google came to the cloud market much later. Their engineers should have seen the issues with AWS and the problems that interdependence cause, and specifically architected isolated infrastructure (which from this post-mortem it seems like they didn't). That's both remarkable and disappointing to see.
- tybit 8y ago
  GCP may have come later than AWS but googles internal blob storage could of predated AWS for all we know.
- nerpderp82 8y ago
  The worlds infrastructure is inherently fragile. It should be difficult to take down these large distributed systems, but they are over-coupled.
  - m0zg 8y ago
    It's very easy, apparently, to take them down by a configuration change. Happens several times a year and takes hours (and untold millions of dollars in productivity loss) to recover from.
- bogomipz 8y ago
  >"A series of choices made when the idea that there would ever be more than two regions, leave them unfortunately stuck having to tie regions together to a limited degree (bucket names are global)." Could you elaborate on those choices? Also what do you mean by they're "stuck having to tie regions together"?
  - Twirrim 8y ago
    The main thing is that bucket namespace is global. There is only one bucket "foo" worldwide. That means that every bucket mutation event (e.g. bucket deletion or creation) requires co-ordination between regions. When that big us-east-1 region outage occurred a year or so back every other region worked fine except for bucket mutations.
uluyol 8y ago
Sure it can. Let's back up a bit to something physical as an example. If every car on the road were self driving, that would form a decentralized and distributed system. Each car independently monitors and reacts to its surroundings. That doesn't prevent accidents between two cars, nor does it prevent huge cascading accidents that can cause highway closures. Decentralization often makes these scenarios harder to avoid, not easier. There are simply more ways that things can interact (and therefore break). To avoid these failures in any setting, you need to have provable containment of failures while assuming part of your network is malicious. That's asking for a lot more than decentralization.
- gambler 8y ago
  Like I said, a decentralized system can fail, but the root cause of such a failure cannot be "configuration change". If a configuration change is sufficient to break your system then it's not truly decentralized. Let's take your car example. If you update your car's firmware and get in an accident, it's not a system-level failure. If manufacturer sends a wireless update and all their autonomous cars start to bump into one another, it's not really a decentralized system anymore - updates are centralized. If you update your car, it crashes, sends a wireless signal that affects another car, and it crashes and so on... This is an example of a cascading failure in a fully decentralized system, but the real root cause would be cars interfering with one another, not your firmware update.
  - uluyol 8y ago
    If a manufacturer with 10 percent market share sends out an update that causes their vehicles to suddenly turn right, that would ruin conditions for everyone.
    - matt_kantor 8y ago
      If that's the case, then the conclusion based on the upthread reasoning is that this system isn't decentralized enough when any single manufacturer has a 10 percent market share.
- 8y ago

Twirrim 8y ago

S3 is (for want of a better term) a legacy system in AWS, having been around since pretty much the beginning. A series of choices made when the idea that there would ever be more than two regions, leave them unfortunately stuck having to tie regions together to a limited degree (bucket names are global). Beyond those bucket mutations, everything else is purely regional scoped. I'm _really_ surprised to see that GCP has made the same mistakes though and had things tied together. I'm not convinced there's a good excuse. AWS/S3 is understandable. No one knew what things would look like. Not one engineer would have made the same decisions then if they'd known what the future was for the platform. Google came to the cloud market much later. Their engineers should have seen the issues with AWS and the problems that interdependence cause, and specifically architected isolated infrastructure (which from this post-mortem it seems like they didn't). That's both remarkable and disappointing to see.

tybit 8y ago
GCP may have come later than AWS but googles internal blob storage could of predated AWS for all we know.
nerpderp82 8y ago
The worlds infrastructure is inherently fragile. It should be difficult to take down these large distributed systems, but they are over-coupled.
- m0zg 8y ago
  It's very easy, apparently, to take them down by a configuration change. Happens several times a year and takes hours (and untold millions of dollars in productivity loss) to recover from.
bogomipz 8y ago
>"A series of choices made when the idea that there would ever be more than two regions, leave them unfortunately stuck having to tie regions together to a limited degree (bucket names are global)." Could you elaborate on those choices? Also what do you mean by they're "stuck having to tie regions together"?
- Twirrim 8y ago
  The main thing is that bucket namespace is global. There is only one bucket "foo" worldwide. That means that every bucket mutation event (e.g. bucket deletion or creation) requires co-ordination between regions. When that big us-east-1 region outage occurred a year or so back every other region worked fine except for bucket mutations.

uluyol 8y ago

Sure it can. Let's back up a bit to something physical as an example. If every car on the road were self driving, that would form a decentralized and distributed system. Each car independently monitors and reacts to its surroundings. That doesn't prevent accidents between two cars, nor does it prevent huge cascading accidents that can cause highway closures. Decentralization often makes these scenarios harder to avoid, not easier. There are simply more ways that things can interact (and therefore break). To avoid these failures in any setting, you need to have provable containment of failures while assuming part of your network is malicious. That's asking for a lot more than decentralization.

gambler 8y ago
Like I said, a decentralized system can fail, but the root cause of such a failure cannot be "configuration change". If a configuration change is sufficient to break your system then it's not truly decentralized. Let's take your car example. If you update your car's firmware and get in an accident, it's not a system-level failure. If manufacturer sends a wireless update and all their autonomous cars start to bump into one another, it's not really a decentralized system anymore - updates are centralized. If you update your car, it crashes, sends a wireless signal that affects another car, and it crashes and so on... This is an example of a cascading failure in a fully decentralized system, but the real root cause would be cars interfering with one another, not your firmware update.
- uluyol 8y ago
  If a manufacturer with 10 percent market share sends out an update that causes their vehicles to suddenly turn right, that would ruin conditions for everyone.
  - matt_kantor 8y ago
    If that's the case, then the conclusion based on the upthread reasoning is that this system isn't decentralized enough when any single manufacturer has a 10 percent market share.
8y ago