I used to work support for a SMS aggregator. The core business was delivering texts to mobile networks. Mobile networks break all the time, it turns out. And ca

I used to work support for a SMS aggregator. The core business was delivering texts to mobile networks. Mobile networks break all the time, it turns out. And can't really be replaced - there's only one T-Mobile. Now, when they died or sent garbage, our software wouldn't crash. But our service would stop functioning. And alarms would go off, and we would have to confirm why, and call them, and ask them if they knew if it had stopped working and why. (The follow up was indeed to always ask to be added to the mailing list that would let us know this ahead of time. These proved remarkably unhelpful. In once case, we ended up setting up having to set up a mobile number to receive SMS alerts - this was apparently the only way they would notify anyone?) And reasonably often actual engineers would be needed to be woken up. And in several cases, change the parsing behaviour so we could handle sudden unexpected changes in the format they returned. Yes, in the middle of the night. Our system wasn't perfect, but I'm not sure our problem was simply a lack of system or unit tests. During the day, we largely dealt with more minor customer complaints, and ongoing maintenance, outages and other nonsense fromt he carriers. Engineers would often have to dig into these edge cases too, and there was a nominated maintainer to look at this stuff, to let the rest of the team add features and work on long term fixes. But sometimes you just need someone to wedge LargeCustomer's encoding settings because they can't figure out how to properly specify it on their end.