2 ms·

You were in a position to actually cut off production zones with live traffic at Amazon scale and test the recovery?

• bravetraveler 1y ago
  Yes, it was something we would do to maintain certain contracts. Sounds crazy, isn't: they used a significant portion of the capacity, anyway. They brought the auditors. Real People would notice/care, but financially, it didn't matter. Contract said the edge had to be lost for a moment/restored. I've played both Incident Manager and SRE in this routine. edit: Less often we'd do a more thorough test: power loss/full recovery. We'd disconnect more regularly given the simplicity.