5 ms·

Ok, I have something to add here for the following reasons: * I build big call centers for a living * I'm a really big Telecom Nerd You might be wondering wh

Ok, I have something to add here for the following reasons: * I build big call centers for a living * I'm a really big Telecom Nerd You might be wondering why something as important as 911 is not automated or accessible via any other mechanism; why is it that the most critical service in the country runs on a system that hasn't materially changed since 1984? For one, the Public Switched Telephone Network (PSTN) is extremely reliable, authentic because it's logically addressed, and ubiquitous. The only real flaw is the addressing, which, because of Caller ID, can be faked, but that's another story (See Wikipedia on Swatting). So, there's a huge movement to reshape 911 service to take advantage of modern things like the internet, SMS, GPS and all different kinds of connectivity. The way 911 works right now is sort of a hodgepodge all over the place depending on which vendor the municipality has selected for the Public Service Access Point. There is not one standardized 911 system across the country, which may or may not be a good thing depending on your perspective. So everyone can agree fixing 911 is a great idea, BUT why hasn't it happened already? I can point to a couple potential reasons: 1) It's a really big contract; supposing PSAPs get standardized across the board, we're talking about many millions, potentially billions of dollars. That means lobbying and negotiation and time (oh by the way Sprint is currently leading the charge but I find it unlikely that the US will let a foreign-owned conglomerate manage their backbone emergency services). 2) The current vendors see nothing wrong with the existing solution, and have projects like what has happened in New York to stand against the tide of change. Their service contracts provide a big incentive to keep those boxes in those cities. So I think there's a lot of reasons why 911 is as bad as it is, most of it isn't technical, but political, as with most old ridiculous infrastructures. It's amazing how not planning for obsolescence almost automatically results in political conflict down the road.

gcb0 13y ago
I doubt about #2. #1 is the only issue. As with everything gov, too much money leads to under the table deals. It's ridiculous that we have 200 people constantly updating code for my sound card in Linux at 2 different projects, but no one can give out time to improve a vital system like this which eats billions of my taxes. ...just because someone went to college with someone else or somebody got golf membership somewhere for free.
- josh2600 13y ago
  Just an FYI, 2600hz, where I work, is building an free/open-source operating system for Telecom that could easily support nationwide IP-based 911 service. I just don't have any lobbying money :).
  - voltagex_ 13y ago
    I love the reference, is it Asterisk based?
    - josh2600 13y ago
      A little more complicated than that :). So Here's a brief overview of what we're doing and why we're doing it this way. Asterisk doesn't scale without a lot of cruft and it's not accessible. At the high end of the market there's only gigantic switches from companies like Cisco/Alcatel etc; and those are $$$. So we see a market space inbetween cheapo asterisk and high-end cisco. The crux of what we're doing is a lot of work in AMQP (RabbitMQ) to disintermediate the underlying infrastructure from the services. The point is that we don't ultimately want to care what underlying components you run on your network, only that you have components that can be consumed. Think of it as TelecomSDN from a management perspective and a distributed switch from an infrastructure perspective. The stack as we distribute it today is: * Kamailio (Border Controller) * RabbitMQ (AMQP messaging bus) * FreeSWITCH (Open-source switch) * BigCouch (Distributed database with Dynamo Quorum to help deal with Network Partitions) That's the gist of it; all on CentOS and all open-source. We may close source some modules as we go forward, but right now just about everything is open. Using a few clicks in AWS you can run an infrastructure capable of competing against AT&T. There's a lot of telecom infrastructure that exists which can route calls. There aren't a lot of telecom infrastructures that have built-in fault tolerance and linear scaling. EDIT: I should note that we've developed custom modules for both Kamailio and FreeSWITCH for our use case. We had to write a multi-threading module for Erlang for FreeSWITCH, as an example.
      - voltagex_ 13y ago
        Blergh, Kamailio. Will it scale down to a Raspberry Pi? I realise I'm completely not your target market but I need something that does VoIP better than my router does it.
gregd 13y ago
You mention Sprint as leading the charge and they've got issues of their own. I can't find an article at the moment, but Sprint is a provider of PTT/cell phone services to a local Portland, OR area police force and that police department is being forced to look for another vendor because Sprint has proven to be really, really unreliable.
- josh2600 13y ago
  Sprint as two networks right now. One is the legacy iDen network they acquired in the Nextel deal, the other is their Sprint/Clearwire 4G WiMAX/LTE monstrosity. The iDen network is basically analog, so transactions like presence and push to talk are binary and the state is always transmittable. The same cannot be said of IP. This is a problem we've spent a lot of time working on and I can guarantee none of the Big Telcos have a solution yet. So the answer is, the iDen network's PTT is basically analog and easy to manage whereas the LTE network is IP but difficult to manage. They want people off of iDen because running two networks is a massive waste of money.
  - kayfox 13y ago
    > The iDen network is basically analog, You mean the Integrated Digital Enhanced Network is analog, wait what? The actual problem why noone has come up with a solid way to support PTT on cell phones other than iDEN phones is because iDEN is a rare breed of radio system called a Cellular Trunked Radio System, it was designed to provide a trunked radio system that could scale to large footprints. Unfortunately it never caught on with anyone but Nextel and a few other providers, most likely because of the need to totally replace other systems. Project 25 did not have this issue because agencies could gradually transition from analog conventional and analog trunked systems to digital Project 25 trunked radio systems or hybrids thereof. So, iDEN was intended to primarily support trunked radio like features, Direct Connect and Group Connect are two of those, several bolt on upgrades later provided us with Nationwide Direct Connect and Nationwide Group Connect. This is why it performs so well at those (infact, from what I have seen, outperforming existing trunked radio systems).
    - 13y ago
fredley 13y ago
What's amazing is that this story seems eerily similar to when they upgraded the call centres for the London Ambulance service 21 years ago, and a more recent failure in 2006. You would have thought anyone dealing with these systems would have read that case study at some point? https://en.wikipedia.org/wiki/London_Ambulance_Service#System_failures https://en.wikipedia.org/wiki/London_Ambulance_Service#Syste...
dkarl 13y ago
The supposed advantage of a hodgepodge is that somewhere the right approach has been tried and is working well, and pressure will mount on cities with inferior systems to either match or adopt the better systems. Where, if anywhere, is 911 being done well, and how slowly or quickly will the successful approaches spread? Do you think this dynamic will lead to places like New York having faster and more reliable 911 systems, or should they concentrate on their own efforts?
- josh2600 13y ago
  911 isn't being done well anywhere IMHO. If you look at the way 911 works, there's no location information unless you use the cellular radio (and this assumes your carrier is using the cell tower to anchor your location). If you use IP to call, let's say via an application like RingCentral, how does the 911 service know whether you're in San Francisco or New York? It doesn't. This isn't pie in the sky fantasy, this is actually how all over the top VoIP connections work. You pick a location and that's where your 911 calls originate. It doesn't matter whether you're using your SIP client in New York, your 911 call will still go to a San Francisco 911 center because your office address is in SF. That's not good. So IMHO the right approach is a top-down rewrite and rolling that sucker out nationwide. Normally, we'd see one vendor start to dominate over other vendors, and while there has definitely been some consolidation in the 911 market, the fact that these services are federally mandated (Read: Allocated budget automatically) there's not a huge incentive for municipalities to shop around. These systems are essentially free as far as they're concerned. Part of the reason we're not on IP is because the federal dollars can't be legally spent on IP in many places (gotta love lobbying). In short, the correct approach, IMHO, is IP-enabled PSAPs with GPS information for location combined with reform efforts in congress. The logically addressed world of the phone network is a fantasy and our emergency services should model reality instead of our dreams.
  - jimbokun 13y ago
    "So IMHO the right approach is a top-down rewrite..." This is almost always an extremely bad idea: http://www.joelonsoftware.com/articles/fog0000000069.html http://www.joelonsoftware.com/articles/fog0000000069.html I have no inside knowledge, but wouldn't be surprised at all if many of the problems of the new NYC 911 system is due to a decision to rewrite the code from scratch, losing much of the knowledge and bug fixes embedded in the old system's source code.
deleted 13y ago
[deleted]

gcb0 13y ago

I doubt about #2. #1 is the only issue. As with everything gov, too much money leads to under the table deals. It's ridiculous that we have 200 people constantly updating code for my sound card in Linux at 2 different projects, but no one can give out time to improve a vital system like this which eats billions of my taxes. ...just because someone went to college with someone else or somebody got golf membership somewhere for free.

josh2600 13y ago
Just an FYI, 2600hz, where I work, is building an free/open-source operating system for Telecom that could easily support nationwide IP-based 911 service. I just don't have any lobbying money :).
- voltagex_ 13y ago
  I love the reference, is it Asterisk based?
  - josh2600 13y ago
    A little more complicated than that :). So Here's a brief overview of what we're doing and why we're doing it this way. Asterisk doesn't scale without a lot of cruft and it's not accessible. At the high end of the market there's only gigantic switches from companies like Cisco/Alcatel etc; and those are $$$. So we see a market space inbetween cheapo asterisk and high-end cisco. The crux of what we're doing is a lot of work in AMQP (RabbitMQ) to disintermediate the underlying infrastructure from the services. The point is that we don't ultimately want to care what underlying components you run on your network, only that you have components that can be consumed. Think of it as TelecomSDN from a management perspective and a distributed switch from an infrastructure perspective. The stack as we distribute it today is: * Kamailio (Border Controller) * RabbitMQ (AMQP messaging bus) * FreeSWITCH (Open-source switch) * BigCouch (Distributed database with Dynamo Quorum to help deal with Network Partitions) That's the gist of it; all on CentOS and all open-source. We may close source some modules as we go forward, but right now just about everything is open. Using a few clicks in AWS you can run an infrastructure capable of competing against AT&T. There's a lot of telecom infrastructure that exists which can route calls. There aren't a lot of telecom infrastructures that have built-in fault tolerance and linear scaling. EDIT: I should note that we've developed custom modules for both Kamailio and FreeSWITCH for our use case. We had to write a multi-threading module for Erlang for FreeSWITCH, as an example.
    - voltagex_ 13y ago
      Blergh, Kamailio. Will it scale down to a Raspberry Pi? I realise I'm completely not your target market but I need something that does VoIP better than my router does it.

gregd 13y ago

You mention Sprint as leading the charge and they've got issues of their own. I can't find an article at the moment, but Sprint is a provider of PTT/cell phone services to a local Portland, OR area police force and that police department is being forced to look for another vendor because Sprint has proven to be really, really unreliable.

josh2600 13y ago
Sprint as two networks right now. One is the legacy iDen network they acquired in the Nextel deal, the other is their Sprint/Clearwire 4G WiMAX/LTE monstrosity. The iDen network is basically analog, so transactions like presence and push to talk are binary and the state is always transmittable. The same cannot be said of IP. This is a problem we've spent a lot of time working on and I can guarantee none of the Big Telcos have a solution yet. So the answer is, the iDen network's PTT is basically analog and easy to manage whereas the LTE network is IP but difficult to manage. They want people off of iDen because running two networks is a massive waste of money.
- kayfox 13y ago
  > The iDen network is basically analog, You mean the Integrated Digital Enhanced Network is analog, wait what? The actual problem why noone has come up with a solid way to support PTT on cell phones other than iDEN phones is because iDEN is a rare breed of radio system called a Cellular Trunked Radio System, it was designed to provide a trunked radio system that could scale to large footprints. Unfortunately it never caught on with anyone but Nextel and a few other providers, most likely because of the need to totally replace other systems. Project 25 did not have this issue because agencies could gradually transition from analog conventional and analog trunked systems to digital Project 25 trunked radio systems or hybrids thereof. So, iDEN was intended to primarily support trunked radio like features, Direct Connect and Group Connect are two of those, several bolt on upgrades later provided us with Nationwide Direct Connect and Nationwide Group Connect. This is why it performs so well at those (infact, from what I have seen, outperforming existing trunked radio systems).
  - 13y ago

fredley 13y ago

What's amazing is that this story seems eerily similar to when they upgraded the call centres for the London Ambulance service 21 years ago, and a more recent failure in 2006. You would have thought anyone dealing with these systems would have read that case study at some point? https://en.wikipedia.org/wiki/London_Ambulance_Service#System_failures https://en.wikipedia.org/wiki/London_Ambulance_Service#Syste...

dkarl 13y ago

The supposed advantage of a hodgepodge is that somewhere the right approach has been tried and is working well, and pressure will mount on cities with inferior systems to either match or adopt the better systems. Where, if anywhere, is 911 being done well, and how slowly or quickly will the successful approaches spread? Do you think this dynamic will lead to places like New York having faster and more reliable 911 systems, or should they concentrate on their own efforts?

josh2600 13y ago
911 isn't being done well anywhere IMHO. If you look at the way 911 works, there's no location information unless you use the cellular radio (and this assumes your carrier is using the cell tower to anchor your location). If you use IP to call, let's say via an application like RingCentral, how does the 911 service know whether you're in San Francisco or New York? It doesn't. This isn't pie in the sky fantasy, this is actually how all over the top VoIP connections work. You pick a location and that's where your 911 calls originate. It doesn't matter whether you're using your SIP client in New York, your 911 call will still go to a San Francisco 911 center because your office address is in SF. That's not good. So IMHO the right approach is a top-down rewrite and rolling that sucker out nationwide. Normally, we'd see one vendor start to dominate over other vendors, and while there has definitely been some consolidation in the 911 market, the fact that these services are federally mandated (Read: Allocated budget automatically) there's not a huge incentive for municipalities to shop around. These systems are essentially free as far as they're concerned. Part of the reason we're not on IP is because the federal dollars can't be legally spent on IP in many places (gotta love lobbying). In short, the correct approach, IMHO, is IP-enabled PSAPs with GPS information for location combined with reform efforts in congress. The logically addressed world of the phone network is a fantasy and our emergency services should model reality instead of our dreams.
- jimbokun 13y ago
  "So IMHO the right approach is a top-down rewrite..." This is almost always an extremely bad idea: http://www.joelonsoftware.com/articles/fog0000000069.html http://www.joelonsoftware.com/articles/fog0000000069.html I have no inside knowledge, but wouldn't be surprised at all if many of the problems of the new NYC 911 system is due to a decision to rewrite the code from scratch, losing much of the knowledge and bug fixes embedded in the old system's source code.

deleted 13y ago

[deleted]