3 ms·
The actual document is available as a PDF from the FCC, here: http://transition.fcc.gov/Daily_Releases/Daily_Business/2014/db1017/DOC-330012A1.pdf http://trans
by drostie 12y ago
The actual document is available as a PDF from the FCC, here:
http://transition.fcc.gov/Daily_Releases/Daily_Business/2014/db1017/DOC-330012A1.pdf http://transition.fcc.gov/Daily_Releases/Daily_Business/2014...
I do not see the '40 million' number mentioned in that document, but there are a lot of other legal documents which may be important here.
The key problem is that these states offloaded their 911 calls to Intrado, a company which specializes in looking up the closest actual agency which can respond (called a "PSAP" in the jargon). Intrado had to engineer for two time-domain multiplexed (TDM) systems, basically one normal and one legacy. The legacy system served a CAMA[1] system of 911 PSAPs, CAMA just means that the system automatically logged every number and time.
Because it was servicing this system which had log numbers, it wrote that CAMA's information to some sort of database. (It's a little more complicated than that but basically they needed to make unique IDs as part of the CAMA's protocol.) The counter could only generate N entries. When it tried to handle more entries, some warning-level notification was raised by software, severe enough to crash the application but not severe enough to emit a warning that would alert Intrado's IT staff. All CAMA-routed destination traffic from this Intrado server was affected.
It took several hours before they figured out what was wrong, in part because some other 911 system went down at the same time for an unrelated reason and they figured wrongly that they were related. Once they realized what was going on, they rerouted to a backup server which didn't have a full log file. This second server was able to handle 911 calls while the original bug was fixed. About 6,600 calls did not make it through the system and were dropped (in the form of a busy signal?) rather than routed. They could have switched over to the backup server immediately, but (a) they missed the warnings that their software was emitting, and (b) they didn't understand that the server was the problem.
[1] http://en.wikipedia.org/wiki/Automatic_message_accounting http://en.wikipedia.org/wiki/Automatic_message_accounting