2 ms·

Yet! The 911 outage described in the article was the result of crossing a threshold that was - logically - dead simple both in how it worked and how it was to b

Yet! The 911 outage described in the article was the result of crossing a threshold that was - logically - dead simple both in how it worked and how it was to be understood. The issue was that normal operation of the system pushed through that threshold. The problem here wasn't that the software was changed in a way that made it perform unexpectedly. I don't have a silver bullet, or at least not a pithy one. But I don't see how set theory & etc could have led to any different outcome here.

tetha 9y ago
In my book, operating software is a different skill set from writing software. In the 911 outage, my first thought was: Alright, we have a server accepting calls, and dispatching calls. If there is too much of a difference between incoming and dispatched calls, there is a problem. Maybe it's a capacity problem and we don't have enough 911 operators, maybe it's a software problem, maybe we're dealing with a DoS. I don't need to know or understand anything about the dispatch software, and I'm pretty sure I can monitor, measure and alert these metrics independently from the software. We can twist, turn, push and shuffle Dev, Ops and DevOps around, but good operators largely try to mitigate failure and risk, including their own limited imagination how much infrastructure and software could fail at once. That's a very different skill than trying to make a single component in a network bug-free.