3 ms·

A lot of posts here talk about open source projects or the relative simplicity of implementing something like this at it's core: I agree with the idea though I

A lot of posts here talk about open source projects or the relative simplicity of implementing something like this at it's core: I agree with the idea though I understand that emergency systems require a high level of reliability similar to mil-spec hardware I imagine. If there was any justification for that price of 88+ mil, that would be it. Looks like that went right out the damn window. All systems have bugs, I think we can all agree on that, but I'd like to know that they followed the most sensical procedure to rolling this out. Did they test it prior to release? How did they test it. Did they bother with unit tests at the lowest level? Did they perform thorough integration tests? Did they consider rolling out incrementally (if possible) the new system so that they wouldn't be completely left out in the rain when it failed? What contingency plans if any did they build into the system upon failure? What was wrong with the old system, and what did the new system promise to deliver? It's a lot of questions I wish more journalism would answer. On another note. It's interesting to think about open source. As some have mentioned, it's difficult to imagine the feasibility of actually doing an open source version and getting it adopted (though that may be because I don't know how EMS stuff is structured). Though I think all would benefit from a source code release of this system so we could actually crowdsource a better friggin' version because this one clearly isn't doing its job...since May.

doorhammer 13y ago
I work in the call center branch of a fairly large corporation. Obviously we're not saving lives and dispatching fire trucks, but for some reason it seems like we're dead set against doing incremental rollouts and user testing in any meaningful way. Don't get me wrong, I see things get to user testing phases and we have reasonable ways of logging bugs and problems, but almost every project I've seen had kept on trucking to meet an arbitrary deadline regardless of how poorly any of the testing goes, then the system hits the floor, chaos ensues and everyone runs around in a panic until someone managers to staple together a solution. Kind of depressing that it looks to be the same in the government with calls that are a little more important than ecommerce sales. I want to say that the guy that is in charge of the 911 phone network spoke a few times at a call center conference I recently went to.
- gregd 13y ago
  "but almost every project I've seen had kept on trucking to meet an arbitrary deadline regardless of how poorly any of the testing goes, then the system hits the floor" And the reason for this is simple..the one person who can pull the plug on a project of this magnitude, won't. Simply because s/he doesn't want to have to answer to having already dropped 88 million with nothing to show for it. There becomes a single person to blame at this point, whereas a failed system that gets implemented, suddenly allows for MANY people to be thrown under the bus.
  - hga 13y ago
    Even worse, this is a small, at least in $$$ part of a $2 billion total upgrade of emergency services. And it's the "customer's" entry into the rest of those systems.
  - doorhammer 13y ago
    Totally agree. Even in projects at work, the worst possible thing people think they could do is admit that something they're working on doesn't work the right the first time. I will say, though, that in the case of most projects I've worked on that are failing at some point, we do have something to show for it. We just have things that need to be fixed. Even a totally failed try is something you know doesn't work. It seems like people equate admitting any failure as admitting total failure. It's one of the only reasons I like my job. My boss is comfortable saying "this didn't work, that's fine, let's figure out why it didn't work and make something that does." I recently had a project to reduce a certain call-type to our call centers. We implemented a new system in one part of the company after doing some research into the call drivers, and did a follow up a few months later. My followup showed nothing had changed. I don't have any problem saying that. It just means we need to look into which part of the plan failed, why, and how we can fix it. Either way we'll have a better understanding of the problem. I'd rather do that than fudge the followup, keep the problem, and pretend everything was fixed.
    - gregd 13y ago
      "My boss is comfortable saying "this didn't work, that's fine, let's figure out why it didn't work and make something that does."" The taxpayer is much less forgiving about "failures" and tend to trot out pitchforks and torches at the drop of a hat. They're much more guided by perception than actual facts. So for them, there's very little room for a distinction to be made between utter failure, failure, almost failure, not quite a failure, maybe a failure, no failure...
ampersandy 13y ago
I recently attended a lecture about developing shutdown software for a nuclear power plant. The development process involved three key stages: Formal Requirements Documents, a Software Design Document, and finally, Coding. Each step in the process is accompanied by formal verification processes and an audit which produces a Hazard Analysis Report. Code also goes through code review and verification. I wonder if Intergraph employed a testing plan quite as thorough. Development cycle diagram: http://i.imgur.com/RaBSNHN.png http://i.imgur.com/RaBSNHN.png (from the below PDF, page 4) The entire paper: http://procon.bg/system/files/28.18_Lawford_Wassyng.pdf http://procon.bg/system/files/28.18_Lawford_Wassyng.pdf
- ams6110 13y ago
  I think that mil-spec or nuclear QC standards are probably overkill for civilian systems such as 911. Granted there should be some high standards in place, but the systems are a) not secret/classified and b) not as high liability. Yes, failure to handle a 911 call is serious problem, but not like a failed nuclear reactor shutdown in terms of cost, number of people affected, and as a last resort, ability to fall back to a paper-and-pen backup system.
  - ampersandy 13y ago
    I agree that this process might be overkill for a civilian system, but it seems to me that an ongoing failure to respond efficiently to 911 calls (over the course of weeks, months, years) is quite worthy of a development process that reflects the life or death nature of the software's purpose. Regardless, hopefully Intergraph fixes their software quickly so that the operators aren't put under more stress than they need to be.

doorhammer 13y ago

I work in the call center branch of a fairly large corporation. Obviously we're not saving lives and dispatching fire trucks, but for some reason it seems like we're dead set against doing incremental rollouts and user testing in any meaningful way. Don't get me wrong, I see things get to user testing phases and we have reasonable ways of logging bugs and problems, but almost every project I've seen had kept on trucking to meet an arbitrary deadline regardless of how poorly any of the testing goes, then the system hits the floor, chaos ensues and everyone runs around in a panic until someone managers to staple together a solution. Kind of depressing that it looks to be the same in the government with calls that are a little more important than ecommerce sales. I want to say that the guy that is in charge of the 911 phone network spoke a few times at a call center conference I recently went to.

gregd 13y ago
"but almost every project I've seen had kept on trucking to meet an arbitrary deadline regardless of how poorly any of the testing goes, then the system hits the floor" And the reason for this is simple..the one person who can pull the plug on a project of this magnitude, won't. Simply because s/he doesn't want to have to answer to having already dropped 88 million with nothing to show for it. There becomes a single person to blame at this point, whereas a failed system that gets implemented, suddenly allows for MANY people to be thrown under the bus.
- hga 13y ago
  Even worse, this is a small, at least in $$$ part of a $2 billion total upgrade of emergency services. And it's the "customer's" entry into the rest of those systems.
- doorhammer 13y ago
  Totally agree. Even in projects at work, the worst possible thing people think they could do is admit that something they're working on doesn't work the right the first time. I will say, though, that in the case of most projects I've worked on that are failing at some point, we do have something to show for it. We just have things that need to be fixed. Even a totally failed try is something you know doesn't work. It seems like people equate admitting any failure as admitting total failure. It's one of the only reasons I like my job. My boss is comfortable saying "this didn't work, that's fine, let's figure out why it didn't work and make something that does." I recently had a project to reduce a certain call-type to our call centers. We implemented a new system in one part of the company after doing some research into the call drivers, and did a follow up a few months later. My followup showed nothing had changed. I don't have any problem saying that. It just means we need to look into which part of the plan failed, why, and how we can fix it. Either way we'll have a better understanding of the problem. I'd rather do that than fudge the followup, keep the problem, and pretend everything was fixed.
  - gregd 13y ago
    "My boss is comfortable saying "this didn't work, that's fine, let's figure out why it didn't work and make something that does."" The taxpayer is much less forgiving about "failures" and tend to trot out pitchforks and torches at the drop of a hat. They're much more guided by perception than actual facts. So for them, there's very little room for a distinction to be made between utter failure, failure, almost failure, not quite a failure, maybe a failure, no failure...

ampersandy 13y ago

I recently attended a lecture about developing shutdown software for a nuclear power plant. The development process involved three key stages: Formal Requirements Documents, a Software Design Document, and finally, Coding. Each step in the process is accompanied by formal verification processes and an audit which produces a Hazard Analysis Report. Code also goes through code review and verification. I wonder if Intergraph employed a testing plan quite as thorough. Development cycle diagram: http://i.imgur.com/RaBSNHN.png http://i.imgur.com/RaBSNHN.png (from the below PDF, page 4) The entire paper: http://procon.bg/system/files/28.18_Lawford_Wassyng.pdf http://procon.bg/system/files/28.18_Lawford_Wassyng.pdf

ams6110 13y ago
I think that mil-spec or nuclear QC standards are probably overkill for civilian systems such as 911. Granted there should be some high standards in place, but the systems are a) not secret/classified and b) not as high liability. Yes, failure to handle a 911 call is serious problem, but not like a failed nuclear reactor shutdown in terms of cost, number of people affected, and as a last resort, ability to fall back to a paper-and-pen backup system.
- ampersandy 13y ago
  I agree that this process might be overkill for a civilian system, but it seems to me that an ongoing failure to respond efficiently to 911 calls (over the course of weeks, months, years) is quite worthy of a development process that reflects the life or death nature of the software's purpose. Regardless, hopefully Intergraph fixes their software quickly so that the operators aren't put under more stress than they need to be.