3 ms·

The example however (911 outage) is a poor one a traditional telco could not make that mistake with POTS they talk in major failures (ie losing a switch) as a o

The example however (911 outage) is a poor one a traditional telco could not make that mistake with POTS they talk in major failures (ie losing a switch) as a once in a generation or two.

Animats 9y ago
In the history of the Bell System, no electromechanical exchange was ever totally down for more than half an hour for any reason other than an natural disaster.
- Periodic 9y ago
  The reason cited seems much more a case of moving call-management into a high-level software system that didn't have the same sort of rigor that telecom systems traditionally have. Maybe it did and someone missed this? It's hard to know. In short it feels a lot more like a management failure than a failure of software.
  - walshemj 9y ago
    Probably a lowest cost bidder and not helped by the US devolved government if this had happened in say the UK the papers would have a field day and the government would force the telcos to fix it stat!
- Animats 9y ago
  Someday I should write up how that was done in modern terminology. You can read the "Number 5 Crossbar" documents, but the terminology is archaic.[1] Crossbar offices consisted of a dumb switching fabric and lots of microservices. The switching fabric did the actual connecting, but it was told what to connect by other hardware. Each microservice was implemented on special-purpose hardware, and there were always at least two units of each type. Any unit of a type could do the job, and units of a type were used in rotation. Microservice units included "originating registers", which parsed dial digits, "markers", which took parsed dial digits and routed calls through the switch fabric, "senders", which transmitted dial digits to other exchanges for inter-exchange calls, "incoming registers", which received dial digits from other exchanges, and "translators", which looked up routing info from read-only memory devices. There were also trouble recorders, billing recorders, trouble alarms, and other auxiliary services. Every service unit had a hardware time limit. If a unit took longer than the allowed worst case time, the unit got a hard reset, and a trouble event was logged. This prevented system hangs. Failures of part of the switching fabric could only take down a few lines. Failures of one microservice unit of a group just slowed the system down. Retry policy was "try one microservice unit, if it fails, try a second one, then give up and log an error". If a retry with a different unit didn't work, further retries were unlikely to help. All microservice units were stateless. At the end of each transaction, they went back to their ground state. So they couldn't get into a bad state. All state was in the switching fabric. All microservice units were replaceable and hot-pluggable, so maintenance didn't require downtime. This architecture was very robust in practice. It's worth knowing about today. [1] http://etler.com/docs/Crossbar/ http://etler.com/docs/Crossbar/
  - tmccrmck 9y ago
    Please, please, please do this!

The example however (911 outage) is a poor one a traditional telco could not make that mistake with POTS they talk in major failures (ie losing a switch) as a once in a generation or two.

Animats 9y ago

In the history of the Bell System, no electromechanical exchange was ever totally down for more than half an hour for any reason other than an natural disaster.

Periodic 9y ago
The reason cited seems much more a case of moving call-management into a high-level software system that didn't have the same sort of rigor that telecom systems traditionally have. Maybe it did and someone missed this? It's hard to know. In short it feels a lot more like a management failure than a failure of software.
- walshemj 9y ago
  Probably a lowest cost bidder and not helped by the US devolved government if this had happened in say the UK the papers would have a field day and the government would force the telcos to fix it stat!
Animats 9y ago
Someday I should write up how that was done in modern terminology. You can read the "Number 5 Crossbar" documents, but the terminology is archaic.[1] Crossbar offices consisted of a dumb switching fabric and lots of microservices. The switching fabric did the actual connecting, but it was told what to connect by other hardware. Each microservice was implemented on special-purpose hardware, and there were always at least two units of each type. Any unit of a type could do the job, and units of a type were used in rotation. Microservice units included "originating registers", which parsed dial digits, "markers", which took parsed dial digits and routed calls through the switch fabric, "senders", which transmitted dial digits to other exchanges for inter-exchange calls, "incoming registers", which received dial digits from other exchanges, and "translators", which looked up routing info from read-only memory devices. There were also trouble recorders, billing recorders, trouble alarms, and other auxiliary services. Every service unit had a hardware time limit. If a unit took longer than the allowed worst case time, the unit got a hard reset, and a trouble event was logged. This prevented system hangs. Failures of part of the switching fabric could only take down a few lines. Failures of one microservice unit of a group just slowed the system down. Retry policy was "try one microservice unit, if it fails, try a second one, then give up and log an error". If a retry with a different unit didn't work, further retries were unlikely to help. All microservice units were stateless. At the end of each transaction, they went back to their ground state. So they couldn't get into a bad state. All state was in the switching fabric. All microservice units were replaceable and hot-pluggable, so maintenance didn't require downtime. This architecture was very robust in practice. It's worth knowing about today. [1] http://etler.com/docs/Crossbar/ http://etler.com/docs/Crossbar/
- tmccrmck 9y ago
  Please, please, please do this!