5 ms·

The postmortem for this should make for a good read. How does storage go down in eleven regions at once?

• deleted 12y ago
  [deleted]
• ohyesyodo 12y ago
  Just apply same buggy network patch to all DCs at once? They use software networking so causing something like this should be easy. Or mess up network routing for *.blob.core.windows.net which pretty much all of Azure relies on.
  • icebraining 12y ago
    Isn't applying the same patch everywhere at once a major anti-pattern?
    • davis 12y ago
      Yes.
    • ohyesyodo 12y ago
      Turns out this was exactly what happened - they applied a buggy patch to all data centers at once by mistake.
• deleted 12y ago
  [deleted]
• photorized 12y ago
  I suspect a human config error. I don't see how else multiple services, in multiple regions, can all be affected at once. Looking forward to the post mortem.