5 ms·

Can someone explain how does WW outage happen? My assumption is anysite at Amazon scale is atleast designed to be regional partitioned.

• xwolfi 5y ago
  There must be a silly single source of failure, like a lambda giving the time of day or a marketing tracker pixel. Yourkit, a java profiler, stopped working after a certain date in 2018 and had to be patched on thousands of users in my company alone. Could be some such time based constraint on a dependency loaded by all components for instance. You can regionally partition but if the poison pill was inserted years ago and rolled out everywhere with a time trigger...
• paxys 5y ago
  There are always bottlenecks and single points of failure no matter how resilient the design in. And even if there are multiple regions, the same bit of offending code could have been deployed to all of them.
• KeepFlying 5y ago
  It could be that all global systems happen to rely on some kind of single point of failure that the team never really realized before. Or if they did realize it they didn't realize the potential impact. Or it was a code change that was able to be validated and rolled out to all regions but then some totally independent interaction caused it to fail afterward in a unique way. I'm banking on surprising singe point of failure though. I really hope we get a detailed post mortem here.
  • dehrmann 5y ago
    > Or it was a code change Probably not; I don't know specifically for Amazon, but most places avoid weekend code changes.
    • alisonkisk 5y ago
      > some totally independent interaction caused it to fail afterward
  • MrStonedOne 5y ago
    Operational consistency is a single point of failure.