A few years ago we had an outage that cost us quite a bit of money [1]. The root cause was the deletion of a test record from the database we query (and run by

A few years ago we had an outage that cost us quite a bit of money [1]. The root cause was the deletion of a test record from the database we query (and run by the Oligarchist Phone Company) but the deletion had a domino effect on our service because of the way the error was treated by our side (the test record was never supposed to be deleted because it's used to ensure the database we query is up and running [2][3]). From what I understand, we were told the record would never get removed, and we relied upon that. Could our error handling been better? Yes. Could the customer's procedures been better to prevent the record from being deleted? Yes. It was a mistake on both parties. Who should have gotten fired? [1] Very scary SLAs with the Oligarchist Phone Company. [2] It's actually a DNS query. NAPTR records to be precise. Search on that if you are interested. [3] It was also important because that was the only record we could query and not get charged for it. Everything in the telephony world cross-company (and even cross-department) gets charged. Somebody makes money; somebody pays money.