10 ms·

My hell as a 911 operator

223 points by DavidChouinard 13y ago

ck2 13y ago
Lowest bidder? No one loses their job for someone dying? Sounds like a perfect storm. I'd love to see an itemization for $88 Million and how many vacation homes were bought.
- thezach 13y ago
  They don't always take lowest bidder.... sometimes it really just boils down to who the contractor knows, or what favors they've done for people (even if they are incompetant)
  - gregd 13y ago
    This is generally not true. What it generally boils down to with government contracts is a really sweet dog and pony show from the RFP process. Generally people lack the critical thinking skills necessary to determine if a company can really fill their needs. They don't know the difference between a COTS system and one built from the ground up. They don't know the difference between a hosted solution and one running on their own servers. I would say very rarely does it come down to simply being the lowest bidder or nepotism.
    - hga 13y ago
      Sometimes it does, due to the (near) total ignorance of what this is all about. 10 years ago I was briefly employed by a contractor for a state government that was working on a system a bit like this, although not hardly so life critical. The original system had been written and customized by the one of the big names like Lockheed or CSC and was pretty good as these things go ... but the state put continued maintenance out for bid and gave it to another company. With no transition, and really counterproductive budgeting. I.e. around a year previously another guy was hired to do what I did, and let go after a month or two because of money. What was I hired to do? The most difficult job of software archaeology I've ever attempted, not counting one code base that was impossible (https://news.ycombinator.com/item?id=6092960 https://news.ycombinator.com/item?id=6092960). The original programmers, probably on site, used SCCS (!) ... until they stopped some months before producing the binaries running on the system. Other minor details: the system ran on DEC Alpha hardware and their version of UNIX(TM), and of course by then it was a doomed system with no migration path for binaries. The state budgeted the project and made its milestones based on estimates made by another consulting company, and those were not based on reality. I bailed when it became clear there was no possible way to meet the state's demands, e.g. a point when I realized that if there was a way to recreate the production binaries I wasn't going to be able to find it, and they were assuming the sources were in a lot better shape.
    - lostlogin 13y ago
      Just to throw in a situation I'm in at the moment. A public hospital has put out a tender for a new MRI scanner. I know who is likely cheapest (GE) and I know who is likely the most expensive (Siemens). I suspect Philips falls nearer the higher end, but will be in between. I have nothing to do with the purchase but work with those who do. I'd bet everything I own that GE don't win as the corners cut and unreliability are not winning any favours. Their scanner ticks all the boxes, but in the worst possible way. Multi-transmit RF? Yes! (But it doesn't work). Wide bore? Yes! (But you can't scan any further laterally than with a narrow bore). Propeller type imaging? Yes! (But scanner crashes, weird artifacts arise or the utterly crap interface gets in the way). The list goes on for probably another 50 points. Philips and Siemens have faults, sure. But they are more the result of compromises and are stylistic where the system is that way because someone chose to make it like that. Not because someone screwed up, forgot, didn't check, didn't know etc. some places chose quality over price, and this publicly owned MRI scanner will be in this category.
- moogleii 13y ago
  Sounds like it could fit perfectly with NYC's current "See something crooked" campaign: http://www.therepublic.com/view/story/6f91a149db2641cc84c25c2ad7621f7b/NY--See-Something-Crooked http://www.therepublic.com/view/story/6f91a149db2641cc84c25c...
- mattmcknight 13y ago
  Let's not forget that the developers of the system are potentially handing the system over to another party for O&M, didn't have flexibility in choosing what software to use, didn't get to prioritize the requirements and blue buttons were equal with system stability. They may have even had a separate contract do the design spec, and IV&V. The diffusion of responsibility combined with the rigidity of the procurement process is such that it is often impossible to achieve the flexibility needed to devise an optimal solution.
- draugadrotten 13y ago
  > Lowest bidder? No one loses their job for someone dying? Government is not as bad as you think. It's worse. Lowest bidder really causes a lot of problems. My company recently bid for a contract for an EU agency. We were not chosen, because the winning bid offered senior staff at a rate of 0.1 euro per hour. The bloody idiots at Government procurement calculated the average as (senior+junior)/2 and awarded the contract to these crooks with the flawed bid as their average was the lowest. It's not unreasonable to assume they will debit enough extra junior hours to offset the senior time. I am not even allowed to tell media about this, by contract. I don't think it's a rare event. Just my €0.02.
  - VMG 13y ago
    Please find a way to wikileak it.
- lucb1e 13y ago
  > No one loses their job for someone dying? They should. I think suing them over the deaths of hundreds of people and having the company that made this system go bankrupt is the very least they can do. Then the secret service can come in and crack some more necks. They got paid 88 million US dollars. For that you can make a system that Just Works or be honest up front if you can't get it done (on time).
kylec 13y ago
This seems like a great opportunity for someone to develop an open source high-availability, fault-tolerant 911 dispatch system. Being able to get first responders where they're needed as fast as possible would provide a tremendous public good.
- richardv 13y ago
  That's not how things work... You can't just create some system and see governments flock to you. There are years of consulting and specification requirements. Linux is open source... Office Libre... Go convince a public office to switch to Ubuntu.. Saving money is often at the bottom of their focus.. and then you have competing interests releasing figures that show it is more costly to switch to free alternatives. Public spending is a corrupt cycle.
  - tomsaffell 13y ago
    >There are years of ... Yes, First they ignore you, then they laugh at you, then they fight you, then you win. Sorry, I know it's trite, but there's some truth there. Being ignored for years is not necessarily a reason not to do something. In some cases it's the very best reason to do it. [EDIT] - would those people down-voting please explain why?
    - doki_pen 13y ago
      I don't see how you could develop such a system without partnership from some government. How would you test it?
      - coldarchon 13y ago
        You have just read a perfect description what this program needs, from there it's just attributes ..
      - tomsaffell 13y ago
        Maybe you could partner with just one small government to start. Of all the municipalities in the world, you'd think at-least one would be open to the idea, even if in tandem. Maybe it could be started by a person who works (or worked) at the dispatch center, and is also a programmer (there must be at-least one of those in the world too). The point is that NYC laughing/ignoring/fighting you is not a good enough reason not to start. (and who's to say that NYC would do any of those things, given the current state of affairs)
deleted 13y ago
[deleted]
gregd 13y ago
Having been involved in long, arduous government procurement processes, I can assure you that it will leave you shaking your head and wondering how ANYTHING in government gets done. The trip from RFP to signed contract is maddening and has probably driven more than one government worker to an alcoholic stupor, wishing for the sweet embrace of death. On a side note, I'm no longer a government employee. My name is Greg and I'm an alcoholic....
- greghinch 13y ago
  When I was freelancing, I did some government work. I was flabbergasted by the slow pace at which things moved, and that I had to create accounts in 3 separate systems to submit and get paid for even very basic invoices. Government is a place that is ripe for disruption, but the walls around it are so high, I don't know how to really breach them. There is a huge machine in place of contractors and lobbyists which essentially maintain a strict control over all IT contracts that come out of the public sector. Massive corporate entities with an entrenched web of influencers at every level of government, ensuring that only a select few even have an opportunity to realistically compete for the jobs. The amount of overspending that could be eliminated with the disruption of those entities probably represents a significant portion of local, state, and national budgets, not to mention the likely better solutions that could be presented by folks more accustomed to working efficiently. I'm not necessarily saying the 20 somethings who were going to make "Snapchat for dogs" could do a better job as government contractors. But certainly someone who designed infrastructure for Twitter might have some interesting perspective on how to manage 911 traffic.
  - jdavis703 13y ago
    This kind of disruption can happen. When I was in school I worked at a small 8(a) (women or minority owned federal contractor) firm. All the developers and designers were under 30 and we built all kinds of CRUD apps for the federal government. Now I work at a startup and the real main difference is that everything moves much faster. Not everyone can start an 8(a) firm but we could get legislation passed that creates a more competitive way for any small business to work with the federal government.
DanielBMarkham 13y ago
Pre-9-11, I spent some time doing some high-level analysis, talking to users and administrators, of a nation-wide EMS system for an entire country (Sorry, can't say which) It was a U.S. firm interested in figuring out exactly how much it would cost to create a national system so they could bid on it. This does not have to be a very complicated system at all. It's basically a real-time workflow system. Information comes in, is assigned a type and other data, and depending on the values is routed to various other people, which also tag and update it. In a way, it's almost like a bug-tracking system. Tickets get made. Tickets get worked on. Tickets get resolved. Now combine that 10K problem with a large rollout, lots of administrative BS, and $2 Billion? It's wonder the damn thing works at all. A group of hackers could cobble together something over a week with better uptime and more reliability than I'm inferring from this article. The technical complexity of the problem has nothing at all to do with the political and legal maneuvering required to get and execute a large government contract.
- gregd 13y ago
  "This does not have to be a very complicated system at all. It's basically a real-time workflow system. Information comes in, is assigned a type and other data, and depending on the values is routed to various other people, which also tag and update it. " So basically SharePoint?
  - vxNsr 13y ago
    Exactly what I was thinking. The only question is how many tickets can Sharepoint take per minute?
- mjn 13y ago
  The technical complexity of the problem has nothing at all to do with the political and legal maneuvering required to get and execute a large government contract. Or a large contract with any organization, as far as I can tell. In universities, the whole education-ware thing is a mess, with huge piles of contractor dollars going to junk like Blackboard, who manage to sell things to high-level administrators. By far the best education-ware software I've used is stuff that was developed for almost nothing by comparison: 1) a course-specific tool a student made as a senior project; and 2) a wiki. In industry, I have some third-hand knowledge of a large procurement project at a petrochemical company that spent somewhere in the 9 or 10 figures on document digitization and didn't deliver any document digitization.
siculars 13y ago
Twitter can handle north of 4k tweets per second on virtually entirely open source software and these morons can't record phone calls. If this is not a glaring example of government mismanagement and waste I don't know what is. Where are the criminal indictments?
- hga 13y ago
  Errr, people keep citing Twitter; doesn't anyone remember the Fail Whale?
  - thezach 13y ago
    i just fail whaled last week for 4 hours straight.
- epochwolf 13y ago
  And twitter was buggier than the 911 software for the first two years of it's existence.
  - lucb1e 13y ago
    Good thing no lives depended on it. You'd think they test 911 at more than the designed capacity before bring it into practice.
- thezach 13y ago
  Comparing twitter to a 911 call center is like comparing a monochrome display to a 4k display 911 call centers have to interface with many different computer systems (for example the National Criminal Information center, department of motor vehicles, FBI CJIS, in house databases, court records systems, mapping, computer aided dispatch, weather information, etc. Many of these systems (the NCIC, and CJIS have very strict requirements on what can and can not connect to their systems, and on top of that who can use a terminal that connects to their systems). Its not an easy task. However comparing it to twitter is just ... wow
  - 13y ago
joyeuse6701 13y ago
A lot of posts here talk about open source projects or the relative simplicity of implementing something like this at it's core: I agree with the idea though I understand that emergency systems require a high level of reliability similar to mil-spec hardware I imagine. If there was any justification for that price of 88+ mil, that would be it. Looks like that went right out the damn window. All systems have bugs, I think we can all agree on that, but I'd like to know that they followed the most sensical procedure to rolling this out. Did they test it prior to release? How did they test it. Did they bother with unit tests at the lowest level? Did they perform thorough integration tests? Did they consider rolling out incrementally (if possible) the new system so that they wouldn't be completely left out in the rain when it failed? What contingency plans if any did they build into the system upon failure? What was wrong with the old system, and what did the new system promise to deliver? It's a lot of questions I wish more journalism would answer. On another note. It's interesting to think about open source. As some have mentioned, it's difficult to imagine the feasibility of actually doing an open source version and getting it adopted (though that may be because I don't know how EMS stuff is structured). Though I think all would benefit from a source code release of this system so we could actually crowdsource a better friggin' version because this one clearly isn't doing its job...since May.
- doorhammer 13y ago
  I work in the call center branch of a fairly large corporation. Obviously we're not saving lives and dispatching fire trucks, but for some reason it seems like we're dead set against doing incremental rollouts and user testing in any meaningful way. Don't get me wrong, I see things get to user testing phases and we have reasonable ways of logging bugs and problems, but almost every project I've seen had kept on trucking to meet an arbitrary deadline regardless of how poorly any of the testing goes, then the system hits the floor, chaos ensues and everyone runs around in a panic until someone managers to staple together a solution. Kind of depressing that it looks to be the same in the government with calls that are a little more important than ecommerce sales. I want to say that the guy that is in charge of the 911 phone network spoke a few times at a call center conference I recently went to.
  - 13y ago
300bps 13y ago
Holy hyperbole. Definitely seems like this worker is frustrated beyond sense or has an agenda with saying things like: They give rookie operators two months training, and after that just throw ’em in. They’re not ready. Two months training is not enough training? This is implausible. At its core, they're answering the phone, determining the reason for a call and typing information from the call into a computer. They're not driving out to a site and performing cardiac bypass on people having a heart attack. The whole thing seemed targeted toward eliciting an emotional response rather than a rational one.
- eksith 13y ago
  >Two months training is not enough training? You've never answered the phone, have you? I've worked in customer service and (while that's not exactly any way on par with what these operators go through) there's an entire volume of stuff to keep in mind before going in. Being an operator involves engaging in a fundamentally unnatural mindset that's often completely antithetical to what your normal responses would be. No matter who's screaming, shooting, dying, it's still your job to remain calm, clear and transfer as much information as possible to emergency workers. Try listening to a few 911 calls. There are tons on the interwebs (LiveLeak has a whole lot). Be prepared to lose sleep for a day or two.
  - 300bps 13y ago
    You've never answered the phone, have you? I was the CTO of a large publicly traded insurance company that had 3 call centers across the country and thousands of sales reps and customer service agents. I designed user interfaces and system architecture that enabled them to rapidly quote and service prospective customers and existing customers. I spent 1 day per week in one of our call centers every week with a headset on listening in on calls eight hours per day. I did that for 10 years. I know call centers and I most likely know them better than you do. I think considering someone being given two months training "throwing them in" to be hyperbole.
    - eksith 13y ago
      None of that meets much of the training necessary to save lives. The insurance company has to keep current customers and make sure to gain new ones. That's it. Only monetary loss or gain. The callers at 911 centers (and I'm guessing you still haven't listened to any of these calls) have to handle different systems (with no ability to modify the UIs) contact different services, handle vital data and filter it out from non-vital ones when anything critical lost would mean deaths (as it had been). E.G. To become familiar with police, emergency medical and fire codes. Respond to caller situations appropriately, calmly and above all else, cogently. This is a far greater burden than any insurance operator will ever face in the line of work. It's most definitely not hyperbole.
thebiglebrewski 13y ago
Where can we start the discussion to start building a better version of this?
- deleted 13y ago
  [deleted]
z0ol 13y ago
Guess they're running Windows.
snitko 13y ago
Now imagine for a moment 911 line was a private company. People subscribe to it, pay a monthly fee, call it, wait to get through and don't receive help in time. What do you think people would do then? Stop paying and likely file a lawsuit. Good luck doing this with a government.
- loopdoend 13y ago
  This would of course tie in with the private police force and private fire service, right? And when the customers stop paying and sue the company into oblivion, who handles the calls? Do you prevent monopolies from forming so no one company becomes responsible for all 911 call traffic? That doesn't sound very libertarian to me. Let's presume we're interfacing with public services here, just for shits and giggles. What happens when grandma Jean forgets to pay the bill one month? Does she lose access to those public services when she needs them most because her 911 is redirecting to a busy tone?
  - barnabask 13y ago
    Here's an example of that kind of situation happening for a fire department in Tennassee: http://thinkprogress.org/politics/2010/10/04/122193/county-firefighters-subscription/ http://thinkprogress.org/politics/2010/10/04/122193/county-f...
    - loopdoend 13y ago
      That policy is about as backwards and fucked up as the idea of privatizing a 911 service. Those "firefighters" should be ashamed. As a counterpoint, I refer you to all the other counties in the US where receiving this vital service isn't contingent on your bills being paid. I can see these people losing their home to a fire because they failed to acquire home insurance, but I can't see how moral people would stand by and let a house burn to the ground over an unpaid bill.
      - gknoy 13y ago
        I'm not so sure. It makes some sense: Here is a service we are happy to provide, in a sense a form of insurance. The point of it is that the community pitches in, to cover the catastrophic case that we hope no one has. If you decide not to pay into such a cooperative scheme, you're abusing the trust of all of your neighbors if you expect them to save your home anyway. This is part of the reason we often use taxes to cover the cost of things like this. On the other hand, in this case they had already deployed, and it would have been nicer if there had been SOME price that they had thought of as the "fee" for service to non-subscribers. 5k, 10k, etc: better than losing the house. Once you advertise something like that, though, people will stop subscribing, and decide to gamble that they won't have a catastrophe. The price has to either be high enough to discourage that, or high enough that the fire department can cover its operating costs by servicing X fires per year.
GigabyteCoin 13y ago
The sensationalism is high in this article. Operators had to record 911 calls with pen and paper I know I work all day on a computer but I wouldn't consider using pen and paper a hindrance.
- gregd 13y ago
  Have you ever been in a 911 call center? Many, many, many things are computer driven, not the least of which is interfacing with the various agencies one may have to deal with on a per call basis. Then there's "whose jurisdiction is this"? Kind of hard to determine with pen and paper. Who's dispatched to what? Who's covering which stations now? How many calls are in the queue? The list goes on and on. Having to record things on pen and paper is hardly sensationalism. It's more or less a worse case scenario in the 911 dispatching arena...
  - yaddayadda 13y ago
    And handwriting can be (a) drastically slower and (b) less legible (therefore more prone to mistakes or lost information). In a 911 environment, either of these alone can cost lives. Throw in all of gregd's additional variables, and it would be a nightmare of the sort described in the article.
  - GigabyteCoin 13y ago
    I'm just saying that 20 years ago handwriting was the only option, and police still managed to get the job done.
    - gregd 13y ago
      Oh boy. In 1993 handwriting wasn't the only option. And we're talking about 911 dispatching, not just police. Fire, medics, police, dog catchers, tow trucks etc., are all involved now.
      - jessaustin 13y ago
        Successful management of any endeavor requires prioritization. If dog catchers and tow trucks are in the same bullet list as ambulances, they're doing something wrong.
ErikAugust 13y ago
"...and Intergraph [the Alabama company that designed the new system]." http://www.intergraph.com/ http://www.intergraph.com/ - Flash header on company's homepage hangs on Chrome. Not a good start.
- hga 13y ago
  They used to be a really hot hardware and I think software (e.g. CAD) company, even did a lot of Chipper RISC chip work (http://en.wikipedia.org/wiki/Clipper_architecture http://en.wikipedia.org/wiki/Clipper_architecture ; not to be confused with the Clinton administration cryptographic debacle). Then they exited the hardware business in 2000 and among other things became a patent troll: https://www.google.com/search?q=intergraph+lawsuit https://www.google.com/search?q=intergraph+lawsuit Which of course is an even worse sign.
  - ErikAugust 13y ago
    At least they have been keeping their website's copyright up to date: " © Copyright <SCRIPT>  </SCRIPT> "
    - VMG 13y ago
      Why does website quality always seem to correlate so strongly with the company it's representing?
      - briandear 13y ago
        Funny how that works! A company that can't even get a website right probably has a few things wrong. First it's likely they take their customers for granted and don't care. However, it's likely the guys at the top actually think they have a good website. Companies like that make every decision by committee and as long as the boxes are checked, the 'product' works.
  - gdubs 13y ago
jjindev 13y ago
Large organizations have a tendency to demand custom solutions, and then they want to throw the switch all at once. That is pretty much opposite the bottom up, self-organization, of HN themes. If you could "bubble up" a "good enough" 911 system, it probably would be reliable and probably would scale. But the contract will be given to the company which will say they can provide a big custom solution on a deadline. It's almost worse if they believe it themselves. A dishonest vendor would at least understand the scale of the problem(s).
readme 13y ago
looks like someone needs 911.io -- replacement emergency 911 system using mongodb, node.js, and twitter bootstrap. ultra reliable, super scalable
- talklittle 13y ago
  For posterity: 911.io doesn't exist. Parent is just thinking out loud.
- ams6110 13y ago
  Fad technologies do not a reliable, scalable system make.
  - briandear 13y ago
    So apparently the existing legacy technologies make reliable and scalable systems? A 'fad' technology that works is better than old technology that doesn't.
    - ams6110 13y ago
      I'm just saying, you can't just take a currently popular software stack and think "problem solved."
  - jrockway 13y ago
    You may want to adjust your sarcasm detector.
    - ams6110 13y ago
      Yeah, perhaps.
- deleted 13y ago
  [deleted]
nostromo 13y ago
What if we made 911 into a voucher program. When signing up for a phone, you must pick a 911 provider. The cost is fixed, so you're not choosing based on saving money, but based on quality of service. Each provider would be audited and would publish response times, etc. You could choose the government program described in this article, or go with someone more technologically advanced.
- dotBen 13y ago
  Because the beneficiary of the 911 provider isn't likely to be you, it's the person who needs the emergency who probably can't come to the phone. If someone needs to call 911 for me, I want to know it's regulated and up to quality - not based on whether the person who happened to pick their provider actually bothered to research a good one, etc
  - nostromo 13y ago
    Did the system described in the article sound like a good one to you? Mandated monopolies have a way of discouraging innovation and depressing quality.
    - Udo 13y ago
      Monopolies? 911 dispatch centers should really not be a battleground for competing commercial entities. They're a public service. You may disagree with the general idea of a public service, but it's really one of the very few areas where government can do (and frequently does) things for the good of everyone. Call centers might be bad sometimes regionally, but that doesn't mean they can't work in principle. For example, in my home country (Germany), there is a public dispatch system for all emergency services and after having had to use it repeatedly for a range of things from routine to disaster response, I have to say it works really well. And I wouldn't say Germany is exactly the paragon of efficiency when it comes to public services either. So with a little bit of political will, it's achievable - as it should be.
  - acheron 13y ago
    Because if government bureaucrats "regulate" something that means it has to be good, right?
Ind007 13y ago
Any idea on which technology the current system is built upon?
- ams6110 13y ago
  I will go out on a wild limb and say Windows servers and either SQL Server or Oracle database.
  - brokenparser 13y ago
    Hate to break it to you, but that sounds more than adequate. I would choose differently if given the choice, but it's not a valid excuse for pausing 3 minutes to generate a job number by quite a long shot.
    - ams6110 13y ago
      I actually agree, I wasn't making a good/bad comment, just speculating. Very few government contractors would bid a linux or open-source-software-based system. I do agree, a well designed and implemented system running on Oracle and Windows can certainly handle something like 911 dispatch and do it well.
josh2600 13y ago
Ok, I have something to add here for the following reasons: * I build big call centers for a living * I'm a really big Telecom Nerd You might be wondering why something as important as 911 is not automated or accessible via any other mechanism; why is it that the most critical service in the country runs on a system that hasn't materially changed since 1984? For one, the Public Switched Telephone Network (PSTN) is extremely reliable, authentic because it's logically addressed, and ubiquitous. The only real flaw is the addressing, which, because of Caller ID, can be faked, but that's another story (See Wikipedia on Swatting). So, there's a huge movement to reshape 911 service to take advantage of modern things like the internet, SMS, GPS and all different kinds of connectivity. The way 911 works right now is sort of a hodgepodge all over the place depending on which vendor the municipality has selected for the Public Service Access Point. There is not one standardized 911 system across the country, which may or may not be a good thing depending on your perspective. So everyone can agree fixing 911 is a great idea, BUT why hasn't it happened already? I can point to a couple potential reasons: 1) It's a really big contract; supposing PSAPs get standardized across the board, we're talking about many millions, potentially billions of dollars. That means lobbying and negotiation and time (oh by the way Sprint is currently leading the charge but I find it unlikely that the US will let a foreign-owned conglomerate manage their backbone emergency services). 2) The current vendors see nothing wrong with the existing solution, and have projects like what has happened in New York to stand against the tide of change. Their service contracts provide a big incentive to keep those boxes in those cities. So I think there's a lot of reasons why 911 is as bad as it is, most of it isn't technical, but political, as with most old ridiculous infrastructures. It's amazing how not planning for obsolescence almost automatically results in political conflict down the road.
- gcb0 13y ago
  I doubt about #2. #1 is the only issue. As with everything gov, too much money leads to under the table deals. It's ridiculous that we have 200 people constantly updating code for my sound card in Linux at 2 different projects, but no one can give out time to improve a vital system like this which eats billions of my taxes. ...just because someone went to college with someone else or somebody got golf membership somewhere for free.
chestnut-tree 13y ago
Are there any examples of complex, large-scale government IT projects that have succeeded, come in on cost and time and worked better than what they replaced? I can only think of one example: the UK's Government Digital Service (who built www.gov.uk). Admittedly not a mission-critical project that saves lives, but they actually managed to bring the hundreds of different government websites under one site, re-wrote and re-rorganised the content so it was easier to find and read, and actually improved considerably on what was in place before. It's an exception rather than the rule though to most government or local council projects in the UK. Any other examples of success? (There must be some)
crististm 13y ago
wtf emergency system costs one billion dollars???
- crististm 13y ago
  For those who don't understand my amazement: they build aircraft carriers for this amount of money! Don't tell me it's the same amount of engineering, knowledge and effort in building an emergency response system?
slashCJ 13y ago
It would be fun to have hackathon to design a better 911 system - I'd bet 24 hours worth of work would produce some amazing stuff (probably not super-ultra reliable production ready, but some cool stuff that could save lives)
e40 13y ago
The NY Post? Really??
jowiar 13y ago
I am tired of HN (and the rest of the country) taking shots at government employees. They're talented. They work hard. They take pride in their work. They do a damn good job. If you look at any good technology product that's come out of government, it's almost always been a creation of federal employees: CFPB. Data.gov. Healthcare.gov. Some neat things coming out of the OPM. Accumulo. There are Github repositories: https://github.com/GSA https://github.com/GSA-OCSIT https://github.com/cfpb https://github.com/opengovplatform To quote Deputy CIO of CFPB Matthew Burton: Got two out-of-the-blue emails today asking "who is behind http://CFPB.gov's great graphic design." I'm proud to say: civil servants" [1] On the other hand, feel free to tear apart the contracting system all you want. The system of "who can check all the boxes for the lowest price" is an unmitigated train wreck. It basically mandates a waterfall-designed usability disaster. The sooner we can tear apart that system, the better (and I say that as someone who has spent a majority of my career as a government contractor). [1]: https://twitter.com/matthewburton/status/359747466339418113 https://twitter.com/matthewburton/status/359747466339418113
- LekkoscPiwa 13y ago
  yeah, sure just have more forms for folks taking 911 calls -- I call it the Government efficiency!
  - moocowduckquack 13y ago
    Is funny to watch people decry state enterprise as being always doomed and wasteful, on the internet.
    - LekkoscPiwa 13y ago
      I don't. I think 911 should be run by the Government. I just see the difference between 911 calls I had in the US and their equivalent in Eastern Europe (Poland). To exaggarate: in the US you're like: "Help! Fast! Somebody's dying here! That's the address right here:..." Dispatcher:" What is your name, last name, shoe size, and how do you like the weather? Wait! Too fast! Typying it down!". Poland:"what is the address? what's happening" Bam, dispatch!
blackprawn 13y ago
So let me get this straight: You (and I) truly believe that the telcos have proof positive location on our landlines and cells, and this data is collected whole hog by the govt, but BUT somehow they cant locate a caller having a heart attack on long island. You know, the govt has the systems it deserves, and I really dont care to hear whining about pencil and paper. It takes no longer to write down on pencil and paper a 911 call and theres a mandatory handoff anyway. This has nothing to do with the folks working the phones not being up to par. They are. It has everything to do with big telcos and govt contractors holding taxpayers hostage. We dont need them. Use the pencil and paper, make a phone call to the next dept. That takes the same amount of time as data entry. Grow up, be a human being and stop pretending that life and death situations are some variant of farmville.
- deleted 13y ago
  [deleted]
EpicEng 13y ago
This sort of thing doesn't surprise me. Anecdotally, I've heard more than a few people bemoan a new POS system. Because I write software for a living, I usually prod them for details. Even though their old system was some DOS based thing from 1991, it worked. It was fast and simple. Now they have some new, enterprisey, WPF + EF behemoth that is buggy and slow. Sure, new software will almost always have more bugs than the system it is replacing; the old system, whatever its flaws (perceived or actual), was battle-hardened. It was probably written in a time when speed was more important than the technology stack it used. These people don't care so much about a nicer looking interface. They need something that works and stays out of their way so that they can do their job.
znowi 13y ago
Our call volume is too high for this system. Starting pay for a 911 operator is about $26,000 Yea, obviously not as high a priority as stock exchange market where milliseconds matter!
imonkey 13y ago
I see Windows blue taskbar on the photo. HOW ITS POSSIBLE TO RUN SUCH IMPORTANT SYSTEM ON THE FG WINDOWS??????? Answer me please!
- rdouble 13y ago
  I hate to break this to you, but airplane systems, medical systems, banking systems, fuel pump systems and a lot of other systems all run on Windows.
  - imonkey 13y ago
    Now why they wonder of these problems? I'm old Unix user btw.
  - realo 13y ago
    Windows should NEVER be used in anything mission critical. Windows programmers and their 'Just reboot the machine' mindset should NEVER be allowed anywhere near a mission critical application. Why? Here is your answer, straight from the original article: ======= ... Intergraph keeps telling us, “Shut your computer down and reboot. It’ll just take a minute.” How many lives can you lose in a minute? ... =======
    - ams6110 13y ago
      Nonsense. This has nothing to do with Windows and everything to do with poorly-implemented software running on it. I have seen Java-based systems running on Linux that stall/hang and run agonizingly slowly under what should not be demanding levels of use. Bloated, heavyweight software stacks exist on all platforms, and they are favorites in government contractor architectures.
umrashrf 13y ago
There is a similar movie. The Call. http://www.imdb.com/title/tt1911644/ http://www.imdb.com/title/tt1911644/
rlu 13y ago
People always say being an ATC is the most stressful job that you can have. But this sounds worse, even if the system worked properly. Does anyone know starting salaries for air traffic controllers?
- VLM 13y ago
  $9/hr during training (seriously!), $33K at your first assignment. Pay follows a "public school teacher" trajectory, or at least like the trejectory in my district. Such that you'll start your career eating ramen and living in your car, but by retirement you'll be hauling in $100K or so. The FAA workforce is gray. No other way to say it. So you tend to see ridiculous averages like your local TRACON might pay on average $80K. However as a noob you'd only get $30K or so... you need 30 years in to get rich. Most pilots get to know ATC people for obvious professional reasons. You can probably get interesting anecdotes from a private pilot. Also more than a few ATC personnel ARE private pilots. Helps to understand the other side of the radio, so to speak. I am told there is a narrow window where you're "too old" to be an ATC and get mandatory retirement, yet still young enough to be a CFI, and this is where gray haired flight instructors come from. I do not know if this is true or not. In other words mandatory retirement age for CFI > ATC but not by much.
nfriedly 13y ago
Want to make a difference? Check out http://codeforamerica.org/ http://codeforamerica.org/
rlpb 13y ago
The London Ambulance Service had similar problems when switching to a different system back in 1992. It's actually an incredibly well studied case of an IT project failure. It's studied as a textbook case in just about every IT project management course in the UK. Interesting that NYC is repeating what sounds like exactly the same mistakes. http://en.wikipedia.org/wiki/London_Ambulance_Service#System_failures http://en.wikipedia.org/wiki/London_Ambulance_Service#System... (or just Google for "london ambulance service case study")
tikwidd 13y ago
Seems like the US military would be a good candidate for taking care of 911 dispatch.
peter_tonoli 13y ago
It seems to be that previous massive failures by companies such as Intergraph are quickly forgotten about. In Victoria, Australia, there ended up being a Royal Commission - and Intergraph didn't seek an extension to their contract as a result due to the massive fallout <http://www.abc.net.au/pm/stories/s181316.htm> http://www.abc.net.au/pm/stories/s181316.htm>, and <http://www.abc.net.au/worldtoday/stories/s427907.htm> http://www.abc.net.au/worldtoday/stories/s427907.htm>
- emmelaich 13y ago
  Yep, I expected to hear that Intergraph were responsible. 12 plus years after disasters in NZ and Australia, (the latter resulting in a Royal Commission) they are still at it. Some links: (Australian) Victorian Police Corruption http://www.smuggled.com/Inte1.htm http://www.smuggled.com/Inte1.htm NZ Parliamentary Record: http://www.vdig.net/hansard/content.jsp?id=76345 http://www.vdig.net/hansard/content.jsp?id=76345 "This year [2000] we see the very two people who were involved in the police force who were making all the recommendations about Intergraph, go to work for Intergraph."
ballard 13y ago
Wow. It's a new case study on how process risk equals 1970's-style failures: Big Design Upfront and failure to manage risk. Gambling with people's lives instead of ironing the bugs out in a mock environment, going live on a limited basis (smaller center/group first) and not letting the consultants drive the product. The operators should've had final say on production worthiness. The arrogance of top-down "solutions" that don't work. Other observation: results are often inversely proportional to budget, up to a point. Spending instead of thinking is not a solution.
stretchwithme 13y ago
The miracle of over-centralization at work.
Kilo-byte 13y ago
Intergraph keeps telling us, “Shut your computer down and reboot. It’ll just take a minute. Glad I don't trust my life on such a broken system.

223 points by DavidChouinard 13y ago

ck2 13y ago

Lowest bidder? No one loses their job for someone dying? Sounds like a perfect storm. I'd love to see an itemization for $88 Million and how many vacation homes were bought.

thezach 13y ago
They don't always take lowest bidder.... sometimes it really just boils down to who the contractor knows, or what favors they've done for people (even if they are incompetant)
- gregd 13y ago
  This is generally not true. What it generally boils down to with government contracts is a really sweet dog and pony show from the RFP process. Generally people lack the critical thinking skills necessary to determine if a company can really fill their needs. They don't know the difference between a COTS system and one built from the ground up. They don't know the difference between a hosted solution and one running on their own servers. I would say very rarely does it come down to simply being the lowest bidder or nepotism.
  - hga 13y ago
    Sometimes it does, due to the (near) total ignorance of what this is all about. 10 years ago I was briefly employed by a contractor for a state government that was working on a system a bit like this, although not hardly so life critical. The original system had been written and customized by the one of the big names like Lockheed or CSC and was pretty good as these things go ... but the state put continued maintenance out for bid and gave it to another company. With no transition, and really counterproductive budgeting. I.e. around a year previously another guy was hired to do what I did, and let go after a month or two because of money. What was I hired to do? The most difficult job of software archaeology I've ever attempted, not counting one code base that was impossible (https://news.ycombinator.com/item?id=6092960 https://news.ycombinator.com/item?id=6092960). The original programmers, probably on site, used SCCS (!) ... until they stopped some months before producing the binaries running on the system. Other minor details: the system ran on DEC Alpha hardware and their version of UNIX(TM), and of course by then it was a doomed system with no migration path for binaries. The state budgeted the project and made its milestones based on estimates made by another consulting company, and those were not based on reality. I bailed when it became clear there was no possible way to meet the state's demands, e.g. a point when I realized that if there was a way to recreate the production binaries I wasn't going to be able to find it, and they were assuming the sources were in a lot better shape.
  - lostlogin 13y ago
    Just to throw in a situation I'm in at the moment. A public hospital has put out a tender for a new MRI scanner. I know who is likely cheapest (GE) and I know who is likely the most expensive (Siemens). I suspect Philips falls nearer the higher end, but will be in between. I have nothing to do with the purchase but work with those who do. I'd bet everything I own that GE don't win as the corners cut and unreliability are not winning any favours. Their scanner ticks all the boxes, but in the worst possible way. Multi-transmit RF? Yes! (But it doesn't work). Wide bore? Yes! (But you can't scan any further laterally than with a narrow bore). Propeller type imaging? Yes! (But scanner crashes, weird artifacts arise or the utterly crap interface gets in the way). The list goes on for probably another 50 points. Philips and Siemens have faults, sure. But they are more the result of compromises and are stylistic where the system is that way because someone chose to make it like that. Not because someone screwed up, forgot, didn't check, didn't know etc. some places chose quality over price, and this publicly owned MRI scanner will be in this category.
moogleii 13y ago
Sounds like it could fit perfectly with NYC's current "See something crooked" campaign: http://www.therepublic.com/view/story/6f91a149db2641cc84c25c2ad7621f7b/NY--See-Something-Crooked http://www.therepublic.com/view/story/6f91a149db2641cc84c25c...
mattmcknight 13y ago
Let's not forget that the developers of the system are potentially handing the system over to another party for O&M, didn't have flexibility in choosing what software to use, didn't get to prioritize the requirements and blue buttons were equal with system stability. They may have even had a separate contract do the design spec, and IV&V. The diffusion of responsibility combined with the rigidity of the procurement process is such that it is often impossible to achieve the flexibility needed to devise an optimal solution.
draugadrotten 13y ago
> Lowest bidder? No one loses their job for someone dying? Government is not as bad as you think. It's worse. Lowest bidder really causes a lot of problems. My company recently bid for a contract for an EU agency. We were not chosen, because the winning bid offered senior staff at a rate of 0.1 euro per hour. The bloody idiots at Government procurement calculated the average as (senior+junior)/2 and awarded the contract to these crooks with the flawed bid as their average was the lowest. It's not unreasonable to assume they will debit enough extra junior hours to offset the senior time. I am not even allowed to tell media about this, by contract. I don't think it's a rare event. Just my €0.02.
- VMG 13y ago
  Please find a way to wikileak it.
lucb1e 13y ago
> No one loses their job for someone dying? They should. I think suing them over the deaths of hundreds of people and having the company that made this system go bankrupt is the very least they can do. Then the secret service can come in and crack some more necks. They got paid 88 million US dollars. For that you can make a system that Just Works or be honest up front if you can't get it done (on time).

kylec 13y ago

This seems like a great opportunity for someone to develop an open source high-availability, fault-tolerant 911 dispatch system. Being able to get first responders where they're needed as fast as possible would provide a tremendous public good.

richardv 13y ago
That's not how things work... You can't just create some system and see governments flock to you. There are years of consulting and specification requirements. Linux is open source... Office Libre... Go convince a public office to switch to Ubuntu.. Saving money is often at the bottom of their focus.. and then you have competing interests releasing figures that show it is more costly to switch to free alternatives. Public spending is a corrupt cycle.
- tomsaffell 13y ago
  >There are years of ... Yes, First they ignore you, then they laugh at you, then they fight you, then you win. Sorry, I know it's trite, but there's some truth there. Being ignored for years is not necessarily a reason not to do something. In some cases it's the very best reason to do it. [EDIT] - would those people down-voting please explain why?
  - doki_pen 13y ago
    I don't see how you could develop such a system without partnership from some government. How would you test it?
    - coldarchon 13y ago
      You have just read a perfect description what this program needs, from there it's just attributes ..
    - tomsaffell 13y ago
      Maybe you could partner with just one small government to start. Of all the municipalities in the world, you'd think at-least one would be open to the idea, even if in tandem. Maybe it could be started by a person who works (or worked) at the dispatch center, and is also a programmer (there must be at-least one of those in the world too). The point is that NYC laughing/ignoring/fighting you is not a good enough reason not to start. (and who's to say that NYC would do any of those things, given the current state of affairs)

deleted 13y ago

[deleted]

gregd 13y ago

Having been involved in long, arduous government procurement processes, I can assure you that it will leave you shaking your head and wondering how ANYTHING in government gets done. The trip from RFP to signed contract is maddening and has probably driven more than one government worker to an alcoholic stupor, wishing for the sweet embrace of death. On a side note, I'm no longer a government employee. My name is Greg and I'm an alcoholic....

greghinch 13y ago
When I was freelancing, I did some government work. I was flabbergasted by the slow pace at which things moved, and that I had to create accounts in 3 separate systems to submit and get paid for even very basic invoices. Government is a place that is ripe for disruption, but the walls around it are so high, I don't know how to really breach them. There is a huge machine in place of contractors and lobbyists which essentially maintain a strict control over all IT contracts that come out of the public sector. Massive corporate entities with an entrenched web of influencers at every level of government, ensuring that only a select few even have an opportunity to realistically compete for the jobs. The amount of overspending that could be eliminated with the disruption of those entities probably represents a significant portion of local, state, and national budgets, not to mention the likely better solutions that could be presented by folks more accustomed to working efficiently. I'm not necessarily saying the 20 somethings who were going to make "Snapchat for dogs" could do a better job as government contractors. But certainly someone who designed infrastructure for Twitter might have some interesting perspective on how to manage 911 traffic.
- jdavis703 13y ago
  This kind of disruption can happen. When I was in school I worked at a small 8(a) (women or minority owned federal contractor) firm. All the developers and designers were under 30 and we built all kinds of CRUD apps for the federal government. Now I work at a startup and the real main difference is that everything moves much faster. Not everyone can start an 8(a) firm but we could get legislation passed that creates a more competitive way for any small business to work with the federal government.

DanielBMarkham 13y ago

Pre-9-11, I spent some time doing some high-level analysis, talking to users and administrators, of a nation-wide EMS system for an entire country (Sorry, can't say which) It was a U.S. firm interested in figuring out exactly how much it would cost to create a national system so they could bid on it. This does not have to be a very complicated system at all. It's basically a real-time workflow system. Information comes in, is assigned a type and other data, and depending on the values is routed to various other people, which also tag and update it. In a way, it's almost like a bug-tracking system. Tickets get made. Tickets get worked on. Tickets get resolved. Now combine that 10K problem with a large rollout, lots of administrative BS, and $2 Billion? It's wonder the damn thing works at all. A group of hackers could cobble together something over a week with better uptime and more reliability than I'm inferring from this article. The technical complexity of the problem has nothing at all to do with the political and legal maneuvering required to get and execute a large government contract.

gregd 13y ago
"This does not have to be a very complicated system at all. It's basically a real-time workflow system. Information comes in, is assigned a type and other data, and depending on the values is routed to various other people, which also tag and update it. " So basically SharePoint?
- vxNsr 13y ago
  Exactly what I was thinking. The only question is how many tickets can Sharepoint take per minute?
mjn 13y ago
The technical complexity of the problem has nothing at all to do with the political and legal maneuvering required to get and execute a large government contract. Or a large contract with any organization, as far as I can tell. In universities, the whole education-ware thing is a mess, with huge piles of contractor dollars going to junk like Blackboard, who manage to sell things to high-level administrators. By far the best education-ware software I've used is stuff that was developed for almost nothing by comparison: 1) a course-specific tool a student made as a senior project; and 2) a wiki. In industry, I have some third-hand knowledge of a large procurement project at a petrochemical company that spent somewhere in the 9 or 10 figures on document digitization and didn't deliver any document digitization.

siculars 13y ago

Twitter can handle north of 4k tweets per second on virtually entirely open source software and these morons can't record phone calls. If this is not a glaring example of government mismanagement and waste I don't know what is. Where are the criminal indictments?

hga 13y ago
Errr, people keep citing Twitter; doesn't anyone remember the Fail Whale?
- thezach 13y ago
  i just fail whaled last week for 4 hours straight.
epochwolf 13y ago
And twitter was buggier than the 911 software for the first two years of it's existence.
- lucb1e 13y ago
  Good thing no lives depended on it. You'd think they test 911 at more than the designed capacity before bring it into practice.
thezach 13y ago
Comparing twitter to a 911 call center is like comparing a monochrome display to a 4k display 911 call centers have to interface with many different computer systems (for example the National Criminal Information center, department of motor vehicles, FBI CJIS, in house databases, court records systems, mapping, computer aided dispatch, weather information, etc. Many of these systems (the NCIC, and CJIS have very strict requirements on what can and can not connect to their systems, and on top of that who can use a terminal that connects to their systems). Its not an easy task. However comparing it to twitter is just ... wow
- 13y ago

joyeuse6701 13y ago

A lot of posts here talk about open source projects or the relative simplicity of implementing something like this at it's core: I agree with the idea though I understand that emergency systems require a high level of reliability similar to mil-spec hardware I imagine. If there was any justification for that price of 88+ mil, that would be it. Looks like that went right out the damn window. All systems have bugs, I think we can all agree on that, but I'd like to know that they followed the most sensical procedure to rolling this out. Did they test it prior to release? How did they test it. Did they bother with unit tests at the lowest level? Did they perform thorough integration tests? Did they consider rolling out incrementally (if possible) the new system so that they wouldn't be completely left out in the rain when it failed? What contingency plans if any did they build into the system upon failure? What was wrong with the old system, and what did the new system promise to deliver? It's a lot of questions I wish more journalism would answer. On another note. It's interesting to think about open source. As some have mentioned, it's difficult to imagine the feasibility of actually doing an open source version and getting it adopted (though that may be because I don't know how EMS stuff is structured). Though I think all would benefit from a source code release of this system so we could actually crowdsource a better friggin' version because this one clearly isn't doing its job...since May.

doorhammer 13y ago
I work in the call center branch of a fairly large corporation. Obviously we're not saving lives and dispatching fire trucks, but for some reason it seems like we're dead set against doing incremental rollouts and user testing in any meaningful way. Don't get me wrong, I see things get to user testing phases and we have reasonable ways of logging bugs and problems, but almost every project I've seen had kept on trucking to meet an arbitrary deadline regardless of how poorly any of the testing goes, then the system hits the floor, chaos ensues and everyone runs around in a panic until someone managers to staple together a solution. Kind of depressing that it looks to be the same in the government with calls that are a little more important than ecommerce sales. I want to say that the guy that is in charge of the 911 phone network spoke a few times at a call center conference I recently went to.
- 13y ago

300bps 13y ago

Holy hyperbole. Definitely seems like this worker is frustrated beyond sense or has an agenda with saying things like: They give rookie operators two months training, and after that just throw ’em in. They’re not ready. Two months training is not enough training? This is implausible. At its core, they're answering the phone, determining the reason for a call and typing information from the call into a computer. They're not driving out to a site and performing cardiac bypass on people having a heart attack. The whole thing seemed targeted toward eliciting an emotional response rather than a rational one.

eksith 13y ago
>Two months training is not enough training? You've never answered the phone, have you? I've worked in customer service and (while that's not exactly any way on par with what these operators go through) there's an entire volume of stuff to keep in mind before going in. Being an operator involves engaging in a fundamentally unnatural mindset that's often completely antithetical to what your normal responses would be. No matter who's screaming, shooting, dying, it's still your job to remain calm, clear and transfer as much information as possible to emergency workers. Try listening to a few 911 calls. There are tons on the interwebs (LiveLeak has a whole lot). Be prepared to lose sleep for a day or two.
- 300bps 13y ago
  You've never answered the phone, have you? I was the CTO of a large publicly traded insurance company that had 3 call centers across the country and thousands of sales reps and customer service agents. I designed user interfaces and system architecture that enabled them to rapidly quote and service prospective customers and existing customers. I spent 1 day per week in one of our call centers every week with a headset on listening in on calls eight hours per day. I did that for 10 years. I know call centers and I most likely know them better than you do. I think considering someone being given two months training "throwing them in" to be hyperbole.
  - eksith 13y ago
    None of that meets much of the training necessary to save lives. The insurance company has to keep current customers and make sure to gain new ones. That's it. Only monetary loss or gain. The callers at 911 centers (and I'm guessing you still haven't listened to any of these calls) have to handle different systems (with no ability to modify the UIs) contact different services, handle vital data and filter it out from non-vital ones when anything critical lost would mean deaths (as it had been). E.G. To become familiar with police, emergency medical and fire codes. Respond to caller situations appropriately, calmly and above all else, cogently. This is a far greater burden than any insurance operator will ever face in the line of work. It's most definitely not hyperbole.

thebiglebrewski 13y ago

Where can we start the discussion to start building a better version of this?

deleted 13y ago
[deleted]

z0ol 13y ago

Guess they're running Windows.

snitko 13y ago

Now imagine for a moment 911 line was a private company. People subscribe to it, pay a monthly fee, call it, wait to get through and don't receive help in time. What do you think people would do then? Stop paying and likely file a lawsuit. Good luck doing this with a government.

loopdoend 13y ago
This would of course tie in with the private police force and private fire service, right? And when the customers stop paying and sue the company into oblivion, who handles the calls? Do you prevent monopolies from forming so no one company becomes responsible for all 911 call traffic? That doesn't sound very libertarian to me. Let's presume we're interfacing with public services here, just for shits and giggles. What happens when grandma Jean forgets to pay the bill one month? Does she lose access to those public services when she needs them most because her 911 is redirecting to a busy tone?
- barnabask 13y ago
  Here's an example of that kind of situation happening for a fire department in Tennassee: http://thinkprogress.org/politics/2010/10/04/122193/county-firefighters-subscription/ http://thinkprogress.org/politics/2010/10/04/122193/county-f...
  - loopdoend 13y ago
    That policy is about as backwards and fucked up as the idea of privatizing a 911 service. Those "firefighters" should be ashamed. As a counterpoint, I refer you to all the other counties in the US where receiving this vital service isn't contingent on your bills being paid. I can see these people losing their home to a fire because they failed to acquire home insurance, but I can't see how moral people would stand by and let a house burn to the ground over an unpaid bill.
    - gknoy 13y ago
      I'm not so sure. It makes some sense: Here is a service we are happy to provide, in a sense a form of insurance. The point of it is that the community pitches in, to cover the catastrophic case that we hope no one has. If you decide not to pay into such a cooperative scheme, you're abusing the trust of all of your neighbors if you expect them to save your home anyway. This is part of the reason we often use taxes to cover the cost of things like this. On the other hand, in this case they had already deployed, and it would have been nicer if there had been SOME price that they had thought of as the "fee" for service to non-subscribers. 5k, 10k, etc: better than losing the house. Once you advertise something like that, though, people will stop subscribing, and decide to gamble that they won't have a catastrophe. The price has to either be high enough to discourage that, or high enough that the fire department can cover its operating costs by servicing X fires per year.

GigabyteCoin 13y ago

The sensationalism is high in this article. Operators had to record 911 calls with pen and paper I know I work all day on a computer but I wouldn't consider using pen and paper a hindrance.

gregd 13y ago
Have you ever been in a 911 call center? Many, many, many things are computer driven, not the least of which is interfacing with the various agencies one may have to deal with on a per call basis. Then there's "whose jurisdiction is this"? Kind of hard to determine with pen and paper. Who's dispatched to what? Who's covering which stations now? How many calls are in the queue? The list goes on and on. Having to record things on pen and paper is hardly sensationalism. It's more or less a worse case scenario in the 911 dispatching arena...
- yaddayadda 13y ago
  And handwriting can be (a) drastically slower and (b) less legible (therefore more prone to mistakes or lost information). In a 911 environment, either of these alone can cost lives. Throw in all of gregd's additional variables, and it would be a nightmare of the sort described in the article.
- GigabyteCoin 13y ago
  I'm just saying that 20 years ago handwriting was the only option, and police still managed to get the job done.
  - gregd 13y ago
    Oh boy. In 1993 handwriting wasn't the only option. And we're talking about 911 dispatching, not just police. Fire, medics, police, dog catchers, tow trucks etc., are all involved now.
    - jessaustin 13y ago
      Successful management of any endeavor requires prioritization. If dog catchers and tow trucks are in the same bullet list as ambulances, they're doing something wrong.

ErikAugust 13y ago

"...and Intergraph [the Alabama company that designed the new system]." http://www.intergraph.com/ http://www.intergraph.com/ - Flash header on company's homepage hangs on Chrome. Not a good start.

hga 13y ago
They used to be a really hot hardware and I think software (e.g. CAD) company, even did a lot of Chipper RISC chip work (http://en.wikipedia.org/wiki/Clipper_architecture http://en.wikipedia.org/wiki/Clipper_architecture ; not to be confused with the Clinton administration cryptographic debacle). Then they exited the hardware business in 2000 and among other things became a patent troll: https://www.google.com/search?q=intergraph+lawsuit https://www.google.com/search?q=intergraph+lawsuit Which of course is an even worse sign.
- ErikAugust 13y ago
  At least they have been keeping their website's copyright up to date: " © Copyright <SCRIPT>  </SCRIPT> "
  - VMG 13y ago
    Why does website quality always seem to correlate so strongly with the company it's representing?
    - briandear 13y ago
      Funny how that works! A company that can't even get a website right probably has a few things wrong. First it's likely they take their customers for granted and don't care. However, it's likely the guys at the top actually think they have a good website. Companies like that make every decision by committee and as long as the boxes are checked, the 'product' works.
- gdubs 13y ago

jjindev 13y ago

Large organizations have a tendency to demand custom solutions, and then they want to throw the switch all at once. That is pretty much opposite the bottom up, self-organization, of HN themes. If you could "bubble up" a "good enough" 911 system, it probably would be reliable and probably would scale. But the contract will be given to the company which will say they can provide a big custom solution on a deadline. It's almost worse if they believe it themselves. A dishonest vendor would at least understand the scale of the problem(s).

readme 13y ago

looks like someone needs 911.io -- replacement emergency 911 system using mongodb, node.js, and twitter bootstrap. ultra reliable, super scalable

talklittle 13y ago
For posterity: 911.io doesn't exist. Parent is just thinking out loud.
ams6110 13y ago
Fad technologies do not a reliable, scalable system make.
- briandear 13y ago
  So apparently the existing legacy technologies make reliable and scalable systems? A 'fad' technology that works is better than old technology that doesn't.
  - ams6110 13y ago
    I'm just saying, you can't just take a currently popular software stack and think "problem solved."
- jrockway 13y ago
  You may want to adjust your sarcasm detector.
  - ams6110 13y ago
    Yeah, perhaps.
deleted 13y ago
[deleted]

nostromo 13y ago

What if we made 911 into a voucher program. When signing up for a phone, you must pick a 911 provider. The cost is fixed, so you're not choosing based on saving money, but based on quality of service. Each provider would be audited and would publish response times, etc. You could choose the government program described in this article, or go with someone more technologically advanced.

dotBen 13y ago
Because the beneficiary of the 911 provider isn't likely to be you, it's the person who needs the emergency who probably can't come to the phone. If someone needs to call 911 for me, I want to know it's regulated and up to quality - not based on whether the person who happened to pick their provider actually bothered to research a good one, etc
- nostromo 13y ago
  Did the system described in the article sound like a good one to you? Mandated monopolies have a way of discouraging innovation and depressing quality.
  - Udo 13y ago
    Monopolies? 911 dispatch centers should really not be a battleground for competing commercial entities. They're a public service. You may disagree with the general idea of a public service, but it's really one of the very few areas where government can do (and frequently does) things for the good of everyone. Call centers might be bad sometimes regionally, but that doesn't mean they can't work in principle. For example, in my home country (Germany), there is a public dispatch system for all emergency services and after having had to use it repeatedly for a range of things from routine to disaster response, I have to say it works really well. And I wouldn't say Germany is exactly the paragon of efficiency when it comes to public services either. So with a little bit of political will, it's achievable - as it should be.
- acheron 13y ago
  Because if government bureaucrats "regulate" something that means it has to be good, right?

Ind007 13y ago

Any idea on which technology the current system is built upon?

ams6110 13y ago
I will go out on a wild limb and say Windows servers and either SQL Server or Oracle database.
- brokenparser 13y ago
  Hate to break it to you, but that sounds more than adequate. I would choose differently if given the choice, but it's not a valid excuse for pausing 3 minutes to generate a job number by quite a long shot.
  - ams6110 13y ago
    I actually agree, I wasn't making a good/bad comment, just speculating. Very few government contractors would bid a linux or open-source-software-based system. I do agree, a well designed and implemented system running on Oracle and Windows can certainly handle something like 911 dispatch and do it well.

josh2600 13y ago

Ok, I have something to add here for the following reasons: * I build big call centers for a living * I'm a really big Telecom Nerd You might be wondering why something as important as 911 is not automated or accessible via any other mechanism; why is it that the most critical service in the country runs on a system that hasn't materially changed since 1984? For one, the Public Switched Telephone Network (PSTN) is extremely reliable, authentic because it's logically addressed, and ubiquitous. The only real flaw is the addressing, which, because of Caller ID, can be faked, but that's another story (See Wikipedia on Swatting). So, there's a huge movement to reshape 911 service to take advantage of modern things like the internet, SMS, GPS and all different kinds of connectivity. The way 911 works right now is sort of a hodgepodge all over the place depending on which vendor the municipality has selected for the Public Service Access Point. There is not one standardized 911 system across the country, which may or may not be a good thing depending on your perspective. So everyone can agree fixing 911 is a great idea, BUT why hasn't it happened already? I can point to a couple potential reasons: 1) It's a really big contract; supposing PSAPs get standardized across the board, we're talking about many millions, potentially billions of dollars. That means lobbying and negotiation and time (oh by the way Sprint is currently leading the charge but I find it unlikely that the US will let a foreign-owned conglomerate manage their backbone emergency services). 2) The current vendors see nothing wrong with the existing solution, and have projects like what has happened in New York to stand against the tide of change. Their service contracts provide a big incentive to keep those boxes in those cities. So I think there's a lot of reasons why 911 is as bad as it is, most of it isn't technical, but political, as with most old ridiculous infrastructures. It's amazing how not planning for obsolescence almost automatically results in political conflict down the road.

gcb0 13y ago
I doubt about #2. #1 is the only issue. As with everything gov, too much money leads to under the table deals. It's ridiculous that we have 200 people constantly updating code for my sound card in Linux at 2 different projects, but no one can give out time to improve a vital system like this which eats billions of my taxes. ...just because someone went to college with someone else or somebody got golf membership somewhere for free.

chestnut-tree 13y ago

Are there any examples of complex, large-scale government IT projects that have succeeded, come in on cost and time and worked better than what they replaced? I can only think of one example: the UK's Government Digital Service (who built www.gov.uk). Admittedly not a mission-critical project that saves lives, but they actually managed to bring the hundreds of different government websites under one site, re-wrote and re-rorganised the content so it was easier to find and read, and actually improved considerably on what was in place before. It's an exception rather than the rule though to most government or local council projects in the UK. Any other examples of success? (There must be some)

crististm 13y ago

wtf emergency system costs one billion dollars???

crististm 13y ago
For those who don't understand my amazement: they build aircraft carriers for this amount of money! Don't tell me it's the same amount of engineering, knowledge and effort in building an emergency response system?

slashCJ 13y ago

It would be fun to have hackathon to design a better 911 system - I'd bet 24 hours worth of work would produce some amazing stuff (probably not super-ultra reliable production ready, but some cool stuff that could save lives)

e40 13y ago

The NY Post? Really??

jowiar 13y ago

I am tired of HN (and the rest of the country) taking shots at government employees. They're talented. They work hard. They take pride in their work. They do a damn good job. If you look at any good technology product that's come out of government, it's almost always been a creation of federal employees: CFPB. Data.gov. Healthcare.gov. Some neat things coming out of the OPM. Accumulo. There are Github repositories: https://github.com/GSA https://github.com/GSA-OCSIT https://github.com/cfpb https://github.com/opengovplatform To quote Deputy CIO of CFPB Matthew Burton: Got two out-of-the-blue emails today asking "who is behind http://CFPB.gov's great graphic design." I'm proud to say: civil servants" [1] On the other hand, feel free to tear apart the contracting system all you want. The system of "who can check all the boxes for the lowest price" is an unmitigated train wreck. It basically mandates a waterfall-designed usability disaster. The sooner we can tear apart that system, the better (and I say that as someone who has spent a majority of my career as a government contractor). [1]: https://twitter.com/matthewburton/status/359747466339418113 https://twitter.com/matthewburton/status/359747466339418113

LekkoscPiwa 13y ago
yeah, sure just have more forms for folks taking 911 calls -- I call it the Government efficiency!
- moocowduckquack 13y ago
  Is funny to watch people decry state enterprise as being always doomed and wasteful, on the internet.
  - LekkoscPiwa 13y ago
    I don't. I think 911 should be run by the Government. I just see the difference between 911 calls I had in the US and their equivalent in Eastern Europe (Poland). To exaggarate: in the US you're like: "Help! Fast! Somebody's dying here! That's the address right here:..." Dispatcher:" What is your name, last name, shoe size, and how do you like the weather? Wait! Too fast! Typying it down!". Poland:"what is the address? what's happening" Bam, dispatch!

blackprawn 13y ago

So let me get this straight: You (and I) truly believe that the telcos have proof positive location on our landlines and cells, and this data is collected whole hog by the govt, but BUT somehow they cant locate a caller having a heart attack on long island. You know, the govt has the systems it deserves, and I really dont care to hear whining about pencil and paper. It takes no longer to write down on pencil and paper a 911 call and theres a mandatory handoff anyway. This has nothing to do with the folks working the phones not being up to par. They are. It has everything to do with big telcos and govt contractors holding taxpayers hostage. We dont need them. Use the pencil and paper, make a phone call to the next dept. That takes the same amount of time as data entry. Grow up, be a human being and stop pretending that life and death situations are some variant of farmville.

deleted 13y ago
[deleted]

EpicEng 13y ago

This sort of thing doesn't surprise me. Anecdotally, I've heard more than a few people bemoan a new POS system. Because I write software for a living, I usually prod them for details. Even though their old system was some DOS based thing from 1991, it worked. It was fast and simple. Now they have some new, enterprisey, WPF + EF behemoth that is buggy and slow. Sure, new software will almost always have more bugs than the system it is replacing; the old system, whatever its flaws (perceived or actual), was battle-hardened. It was probably written in a time when speed was more important than the technology stack it used. These people don't care so much about a nicer looking interface. They need something that works and stays out of their way so that they can do their job.

znowi 13y ago

Our call volume is too high for this system. Starting pay for a 911 operator is about $26,000 Yea, obviously not as high a priority as stock exchange market where milliseconds matter!

imonkey 13y ago

I see Windows blue taskbar on the photo. HOW ITS POSSIBLE TO RUN SUCH IMPORTANT SYSTEM ON THE FG WINDOWS??????? Answer me please!

rdouble 13y ago
I hate to break this to you, but airplane systems, medical systems, banking systems, fuel pump systems and a lot of other systems all run on Windows.
- imonkey 13y ago
  Now why they wonder of these problems? I'm old Unix user btw.
- realo 13y ago
  Windows should NEVER be used in anything mission critical. Windows programmers and their 'Just reboot the machine' mindset should NEVER be allowed anywhere near a mission critical application. Why? Here is your answer, straight from the original article: ======= ... Intergraph keeps telling us, “Shut your computer down and reboot. It’ll just take a minute.” How many lives can you lose in a minute? ... =======
  - ams6110 13y ago
    Nonsense. This has nothing to do with Windows and everything to do with poorly-implemented software running on it. I have seen Java-based systems running on Linux that stall/hang and run agonizingly slowly under what should not be demanding levels of use. Bloated, heavyweight software stacks exist on all platforms, and they are favorites in government contractor architectures.

umrashrf 13y ago

There is a similar movie. The Call. http://www.imdb.com/title/tt1911644/ http://www.imdb.com/title/tt1911644/

rlu 13y ago

People always say being an ATC is the most stressful job that you can have. But this sounds worse, even if the system worked properly. Does anyone know starting salaries for air traffic controllers?

VLM 13y ago
$9/hr during training (seriously!), $33K at your first assignment. Pay follows a "public school teacher" trajectory, or at least like the trejectory in my district. Such that you'll start your career eating ramen and living in your car, but by retirement you'll be hauling in $100K or so. The FAA workforce is gray. No other way to say it. So you tend to see ridiculous averages like your local TRACON might pay on average $80K. However as a noob you'd only get $30K or so... you need 30 years in to get rich. Most pilots get to know ATC people for obvious professional reasons. You can probably get interesting anecdotes from a private pilot. Also more than a few ATC personnel ARE private pilots. Helps to understand the other side of the radio, so to speak. I am told there is a narrow window where you're "too old" to be an ATC and get mandatory retirement, yet still young enough to be a CFI, and this is where gray haired flight instructors come from. I do not know if this is true or not. In other words mandatory retirement age for CFI > ATC but not by much.

nfriedly 13y ago

Want to make a difference? Check out http://codeforamerica.org/ http://codeforamerica.org/

rlpb 13y ago

The London Ambulance Service had similar problems when switching to a different system back in 1992. It's actually an incredibly well studied case of an IT project failure. It's studied as a textbook case in just about every IT project management course in the UK. Interesting that NYC is repeating what sounds like exactly the same mistakes. http://en.wikipedia.org/wiki/London_Ambulance_Service#System_failures http://en.wikipedia.org/wiki/London_Ambulance_Service#System... (or just Google for "london ambulance service case study")

tikwidd 13y ago

Seems like the US military would be a good candidate for taking care of 911 dispatch.

peter_tonoli 13y ago

It seems to be that previous massive failures by companies such as Intergraph are quickly forgotten about. In Victoria, Australia, there ended up being a Royal Commission - and Intergraph didn't seek an extension to their contract as a result due to the massive fallout <http://www.abc.net.au/pm/stories/s181316.htm> http://www.abc.net.au/pm/stories/s181316.htm>, and <http://www.abc.net.au/worldtoday/stories/s427907.htm> http://www.abc.net.au/worldtoday/stories/s427907.htm>

emmelaich 13y ago
Yep, I expected to hear that Intergraph were responsible. 12 plus years after disasters in NZ and Australia, (the latter resulting in a Royal Commission) they are still at it. Some links: (Australian) Victorian Police Corruption http://www.smuggled.com/Inte1.htm http://www.smuggled.com/Inte1.htm NZ Parliamentary Record: http://www.vdig.net/hansard/content.jsp?id=76345 http://www.vdig.net/hansard/content.jsp?id=76345 "This year [2000] we see the very two people who were involved in the police force who were making all the recommendations about Intergraph, go to work for Intergraph."

ballard 13y ago

Wow. It's a new case study on how process risk equals 1970's-style failures: Big Design Upfront and failure to manage risk. Gambling with people's lives instead of ironing the bugs out in a mock environment, going live on a limited basis (smaller center/group first) and not letting the consultants drive the product. The operators should've had final say on production worthiness. The arrogance of top-down "solutions" that don't work. Other observation: results are often inversely proportional to budget, up to a point. Spending instead of thinking is not a solution.

stretchwithme 13y ago

The miracle of over-centralization at work.

Kilo-byte 13y ago

Intergraph keeps telling us, “Shut your computer down and reboot. It’ll just take a minute. Glad I don't trust my life on such a broken system.