A similar thing happened to a Fortune 500 client of mine when a payment gateway decided to make a production change to their API without notice and without depl

A similar thing happened to a Fortune 500 client of mine when a payment gateway decided to make a production change to their API without notice and without deploy to their sandbox first. Another case is the E911 system used by either Vacaville, CA or Solano County that cannot correctly parse GPS information from Verizon Wireless's system, leading to an inability to find callers. It is likely still an unresolved safety issue that will never be fixed. Here are some important principles in this kind of critical system engineering: 0. Waterflow development model 1. Formal verification 2. Staging env 3. "Space Shuttle" control system quorum with deploy new code to one system of three for a month 4. Externally-coordinated change control approval, deployment and notification processes, including subscriber-published specifications 5. All unexpected software conditions raise human attention to relevant stakeholders, similar to Taiichi Ohno's "Stop the Line"