5 ms·

Hyperbole aside ("... an act which would border on criminal negligence if it was done in the private sector and someone was harmed ..." - what does that even me

Hyperbole aside ("... an act which would border on criminal negligence if it was done in the private sector and someone was harmed ..." - what does that even mean? So all of us who have shipped buggy software for our customers are borderline criminals?) - this doesn't surprise me in having dealt with the VA. They have legacy upon legacy upon legacy, with all sorts of fun limitations like not being able to have a "\t" in your content because that'll screw up their backend which relies on tab-delimited data. Health care in the US is playing catch up technology wise to almost every industry. And not for lack of technology, but for lack of political will power. My favourite example of this was trying to deploy an app within the VA that was written in Django. I was told "Python is not on the list of acceptable languages." So we came back to them and said, "Good news everyone, we ported it to Java." Of course, it was just Jython, but that's the sort of stuff you encounter. Multiply this by the complexity involved in trying to herd all these cats into one backend like healthcare.gov and it was doomed to fail.

Spearchucker 13y ago
Such projects are doomed to fail because Big.corp, as well as USA.gov doesn't get that they're not using the right people. High profile project? Want it to work? Hire the right people. Want the right people? Pay whatever it takes. I've been involved in more such projects than I care to remember, and the problem is always the same. A project manager with rudimentary delivery process knowledge owns a large technology project. What's needed is a technically astute lead that knows how to abstract away from delicate backend dependencies, knows that some projects need big design up front, and knows people that have the specializations he or she doesn't. .Gov projects unfortunately are turf wars, where people scramble for a piece of the cake because money smells good, and success is someone else's problem
- hga 13y ago
  Pournelle's Iron Law of Bureaucracy tells us this will almost always be the case (http://en.wikipedia.org/wiki/Jerry_Pournelle#Iron_Law_of_Bureaucracy http://en.wikipedia.org/wiki/Jerry_Pournelle#Iron_Law_of_Bur...): "In any bureaucracy, the people devoted to the benefit of the bureaucracy itself always get in control and those dedicated to the goals the bureaucracy is supposed to accomplish have less and less influence, and sometimes are eliminated entirely." And this amplified restatement: "...in any bureaucratic organization there will be two kinds of people: those who work to further the actual goals of the organization, and those who work for the organization itself. Examples in education would be teachers who work and sacrifice to teach children, vs. union representatives who work to protect any teacher including the most incompetent. The Iron Law states that in all cases, the second type of person will always gain control of the organization, and will always write the rules under which the organization functions."
  - tunesmith 13y ago
    I guess the alternative is to subject the implementers to a truckload of meetings. Implementers don't like that. I can see a pretty clear cause-effect chain here. You let the implementers focus on implementing. There becomes an agreed-upon definition of what sort of matters are in the implementer's sphere of focus - that anything outside that "sphere" is a distraction to the work they need to focus on. Usually those items outside the sphere are a step up in abstraction - product requirements, etc. The implementers then start to get blocked by matters outside their sphere, and they rely on people outside their "sphere" to get them unblocked. People are assigned to roles to handle the issues that are outside the implementer sphere... and they are usually not implementer (because someone with implementing skills is needed on the implementing team) and presto, bureaucracy.
    - hga 13y ago
      You have some interesting musings, but let's reify this to one particular "political" (especially in the pejorative sense we use that) decision: In CMS's integration testing in the week before launch, they tried 200 simultaneous simulated logins, and the system seized up. It was decided to launch anyway, knowing the system was non-functional. By contrast Oregon and Hawaii declined to launch their state exchanges on Oct 1st. I submit that those three decisions are intrinsically political and you can't change that.
      - spikels 13y ago
        I think the administration was worried if they didn't launch on schedule at the start of the shutdown/ceiling fight they might have to accept house republicans demands of a delay in the ACA. Apparently they were gambling the problems wouldn't be discovered until after the shutdown/ceiling fight was over. They won the gamble and the ACA survived but the problems remain and fixing them looks challenging.
        hga 13y ago
        That's for completely supporting my point that this was a political decision. But I'd say the jury is out on the wisdom of this decision. A Healthcare.gov launch delay would have joined a bunch of other delays and waivers, and ... well, you and I don't seem to live in the same universe, I don't see any scenario where Obama and the Senate Dems would have been forced to accept anything they didn't want. There's in fact more than a little worry from the liberal side of things that this most recent crushing of the eGOP was a step too far, e.g. http://pjmedia.com/richardfernandez/2013/10/16/the-third-party/ http://pjmedia.com/richardfernandez/2013/10/16/the-third-par... which I'm just starting to try to grok.
- dasil003 13y ago
  Even without being cynical about the motivation of the players: it's the unknown unknowns that eat them alive, even if they have the best of intentions. It's also a tough problem to solve because very competent people don't want to work on a boondoggle, and any large government IT project has boondoggle written all over it before it even goes out for bid. I wonder if Obama asked for any advice from some of his campaign people. If he didn't before I'm sure he is now.
  - lifeisstillgood 13y ago
    that last paragraph is the most interesting.
  - Spearchucker 13y ago
    I don't buy the unknown unknowns. Big system integration problems follow patterns, and an experienced tech will recognise them. The problem with any half-assed strategy is that people gloss over the knowns, which is fatal. First thing anyone should do on a large (actually any) project is to assess the current situation[1]. When you discover a big nasty piece of legacy you need to integrate with, you look at how it works. The rabbit hole must be followed to it's end, or something will bite you. Blaming unknown unknowns in IT is the same as telling the teacher that the dog ate your homework. https://www.wittenburg.co.uk/Entry.aspx?id=46870dcd-70cb-4ef8-98ed-91c94714fe03 https://www.wittenburg.co.uk/Entry.aspx?id=46870dcd-70cb-4ef...
    - dasil003 13y ago
      It was a poor choice of words on my part. I meant to suggest that they were knowable unknowns, but just that the people involved didn't have a clue.
  - PeterisP 13y ago
    The project is full with 'known knowns' that have bitten them very hard; their excuses may be valid if they were harmed by the unkowns while handling the basic stuff at least somewhat appropriately - which they have not.
- Iftheshoefits 13y ago
  It's actually deliberate that they're not using the right people. It has very little to do with accidental incompetence, mistaken application of domain knowledge, knowledge applied from incorrect domains, or inexperienced project managers and engineers (though the latter can be symptoms of the real issue). This is the result of the correct observation indicated by the last sentence in your comment.
lgieron 13y ago
Another reason is the fact that these large projects rewrite business processes of the organisation (instead of merely automating them). This obviously opens up a huge turf war on how the post-deployment world is going to look like (esp. if there are to be layoffs). Additionally, in .gov, the business processes are always at least partially defined by law, which in consequence needs to be changed. The actual software project is the easy part.
skylan_q 13y ago
an act which would border on criminal negligence if it was done in the private sector and someone was harmed For example, someone thought they got coverage for something and they didn't. That mis-communication can lead to actual harm when it comes to medical and financial issues.
- hga 13y ago
  The good news is that the head of the new management of this project, the fix-it czar Jeffrey Zients, is saying the very top item of his punch list is to stop feeding garbage to the insurance companies. The bad news it that it's taken this long for anyone in authority to even admit there's a major problem there, when there's only 7 weeks left to correct it for those who lose their insurance Jan 1. Someone in another discussion brought up the Space Shuttle accidents. I'll bet you more people will die because of this monumental monstrosity ... except they didn't volunteer for something obviously dangerous.
  - danielweber 13y ago
    While I agree with the rest of what you said, it's jumping to a few conclusions to say that lots of people will die because of this. People will think they are insured when they aren't, but the usual outcome of that is bankruptcy -- or a bunch of ugly and public lawsuits over who pays the bill.
    - hga 13y ago
      I'm making a wild guess than more than 14 people will fall through the cracks and die. Certainly such stories have been used to sell Obamacare. Ah, I just remembered that economists can predict over large populations how many will prematurely die if you remove X dollars from the population. I forget the now obsolete figures, but I'll bet on that basis alone there will be that many casualties. Any time the visible foot of government makes such big steps people will die, it's a given.
      - danielweber 13y ago
fit2rule 13y ago
>what does that even mean? It means that if you are writing health-critical software, where lives are involved, and you deliver low quality dreck (as in this case), then you should be held criminally liable. There is a lot to be said for SIL-4. (http://en.wikipedia.org/wiki/Safety_Integrity_Level http://en.wikipedia.org/wiki/Safety_Integrity_Level) It seems that someone goofed by not giving healthcare.gov a SIL-4 requirement ..
- hga 13y ago
  Would that level of quality be possible in 3.5 years, given all the other constraints? I agree about the lives, BTW, this screw up will kill people: https://news.ycombinator.com/item?id=6622634 https://news.ycombinator.com/item?id=6622634
  - fit2rule 13y ago
    To answer your question: I believe it would be, yes. There's nothing 'hard' about SIL-4 programming - its just a process that you have to follow in order to ensure the quality of your software.. Of course, SIL-4 may not be the solution .. but it sure seems to me that there is no way healthcare.gov would have gotten through the process if SIL-4 was applied to it..
- PeterisP 13y ago
  Well, but healthcare.gov is not health-critical software, just as most other health related tools. Software on an X-ray machine is health-critical, but the software used for billing X-rays is not.
  - fit2rule 13y ago
    I don't know about that - this system is supposed to help people track their expenses on drugs and treatment, and so on .. so to me it seems its not 'just' a billing software package, but rather a healthcare management system. I'd certainly feel better about using it if it had some sort of safety-critical level certification, but then again I don't have the problem of needing healthcare in America ..

Spearchucker 13y ago

Such projects are doomed to fail because Big.corp, as well as USA.gov doesn't get that they're not using the right people. High profile project? Want it to work? Hire the right people. Want the right people? Pay whatever it takes. I've been involved in more such projects than I care to remember, and the problem is always the same. A project manager with rudimentary delivery process knowledge owns a large technology project. What's needed is a technically astute lead that knows how to abstract away from delicate backend dependencies, knows that some projects need big design up front, and knows people that have the specializations he or she doesn't. .Gov projects unfortunately are turf wars, where people scramble for a piece of the cake because money smells good, and success is someone else's problem

hga 13y ago
Pournelle's Iron Law of Bureaucracy tells us this will almost always be the case (http://en.wikipedia.org/wiki/Jerry_Pournelle#Iron_Law_of_Bureaucracy http://en.wikipedia.org/wiki/Jerry_Pournelle#Iron_Law_of_Bur...): "In any bureaucracy, the people devoted to the benefit of the bureaucracy itself always get in control and those dedicated to the goals the bureaucracy is supposed to accomplish have less and less influence, and sometimes are eliminated entirely." And this amplified restatement: "...in any bureaucratic organization there will be two kinds of people: those who work to further the actual goals of the organization, and those who work for the organization itself. Examples in education would be teachers who work and sacrifice to teach children, vs. union representatives who work to protect any teacher including the most incompetent. The Iron Law states that in all cases, the second type of person will always gain control of the organization, and will always write the rules under which the organization functions."
- tunesmith 13y ago
  I guess the alternative is to subject the implementers to a truckload of meetings. Implementers don't like that. I can see a pretty clear cause-effect chain here. You let the implementers focus on implementing. There becomes an agreed-upon definition of what sort of matters are in the implementer's sphere of focus - that anything outside that "sphere" is a distraction to the work they need to focus on. Usually those items outside the sphere are a step up in abstraction - product requirements, etc. The implementers then start to get blocked by matters outside their sphere, and they rely on people outside their "sphere" to get them unblocked. People are assigned to roles to handle the issues that are outside the implementer sphere... and they are usually not implementer (because someone with implementing skills is needed on the implementing team) and presto, bureaucracy.
  - hga 13y ago
    You have some interesting musings, but let's reify this to one particular "political" (especially in the pejorative sense we use that) decision: In CMS's integration testing in the week before launch, they tried 200 simultaneous simulated logins, and the system seized up. It was decided to launch anyway, knowing the system was non-functional. By contrast Oregon and Hawaii declined to launch their state exchanges on Oct 1st. I submit that those three decisions are intrinsically political and you can't change that.
    - spikels 13y ago
      I think the administration was worried if they didn't launch on schedule at the start of the shutdown/ceiling fight they might have to accept house republicans demands of a delay in the ACA. Apparently they were gambling the problems wouldn't be discovered until after the shutdown/ceiling fight was over. They won the gamble and the ACA survived but the problems remain and fixing them looks challenging.
      - hga 13y ago
        That's for completely supporting my point that this was a political decision. But I'd say the jury is out on the wisdom of this decision. A Healthcare.gov launch delay would have joined a bunch of other delays and waivers, and ... well, you and I don't seem to live in the same universe, I don't see any scenario where Obama and the Senate Dems would have been forced to accept anything they didn't want. There's in fact more than a little worry from the liberal side of things that this most recent crushing of the eGOP was a step too far, e.g. http://pjmedia.com/richardfernandez/2013/10/16/the-third-party/ http://pjmedia.com/richardfernandez/2013/10/16/the-third-par... which I'm just starting to try to grok.
dasil003 13y ago
Even without being cynical about the motivation of the players: it's the unknown unknowns that eat them alive, even if they have the best of intentions. It's also a tough problem to solve because very competent people don't want to work on a boondoggle, and any large government IT project has boondoggle written all over it before it even goes out for bid. I wonder if Obama asked for any advice from some of his campaign people. If he didn't before I'm sure he is now.
- lifeisstillgood 13y ago
  that last paragraph is the most interesting.
- Spearchucker 13y ago
  I don't buy the unknown unknowns. Big system integration problems follow patterns, and an experienced tech will recognise them. The problem with any half-assed strategy is that people gloss over the knowns, which is fatal. First thing anyone should do on a large (actually any) project is to assess the current situation[1]. When you discover a big nasty piece of legacy you need to integrate with, you look at how it works. The rabbit hole must be followed to it's end, or something will bite you. Blaming unknown unknowns in IT is the same as telling the teacher that the dog ate your homework. https://www.wittenburg.co.uk/Entry.aspx?id=46870dcd-70cb-4ef8-98ed-91c94714fe03 https://www.wittenburg.co.uk/Entry.aspx?id=46870dcd-70cb-4ef...
  - dasil003 13y ago
    It was a poor choice of words on my part. I meant to suggest that they were knowable unknowns, but just that the people involved didn't have a clue.
- PeterisP 13y ago
  The project is full with 'known knowns' that have bitten them very hard; their excuses may be valid if they were harmed by the unkowns while handling the basic stuff at least somewhat appropriately - which they have not.
Iftheshoefits 13y ago
It's actually deliberate that they're not using the right people. It has very little to do with accidental incompetence, mistaken application of domain knowledge, knowledge applied from incorrect domains, or inexperienced project managers and engineers (though the latter can be symptoms of the real issue). This is the result of the correct observation indicated by the last sentence in your comment.

lgieron 13y ago

Another reason is the fact that these large projects rewrite business processes of the organisation (instead of merely automating them). This obviously opens up a huge turf war on how the post-deployment world is going to look like (esp. if there are to be layoffs). Additionally, in .gov, the business processes are always at least partially defined by law, which in consequence needs to be changed. The actual software project is the easy part.

skylan_q 13y ago

an act which would border on criminal negligence if it was done in the private sector and someone was harmed For example, someone thought they got coverage for something and they didn't. That mis-communication can lead to actual harm when it comes to medical and financial issues.

hga 13y ago
The good news is that the head of the new management of this project, the fix-it czar Jeffrey Zients, is saying the very top item of his punch list is to stop feeding garbage to the insurance companies. The bad news it that it's taken this long for anyone in authority to even admit there's a major problem there, when there's only 7 weeks left to correct it for those who lose their insurance Jan 1. Someone in another discussion brought up the Space Shuttle accidents. I'll bet you more people will die because of this monumental monstrosity ... except they didn't volunteer for something obviously dangerous.
- danielweber 13y ago
  While I agree with the rest of what you said, it's jumping to a few conclusions to say that lots of people will die because of this. People will think they are insured when they aren't, but the usual outcome of that is bankruptcy -- or a bunch of ugly and public lawsuits over who pays the bill.
  - hga 13y ago
    I'm making a wild guess than more than 14 people will fall through the cracks and die. Certainly such stories have been used to sell Obamacare. Ah, I just remembered that economists can predict over large populations how many will prematurely die if you remove X dollars from the population. I forget the now obsolete figures, but I'll bet on that basis alone there will be that many casualties. Any time the visible foot of government makes such big steps people will die, it's a given.
    - danielweber 13y ago

fit2rule 13y ago

>what does that even mean? It means that if you are writing health-critical software, where lives are involved, and you deliver low quality dreck (as in this case), then you should be held criminally liable. There is a lot to be said for SIL-4. (http://en.wikipedia.org/wiki/Safety_Integrity_Level http://en.wikipedia.org/wiki/Safety_Integrity_Level) It seems that someone goofed by not giving healthcare.gov a SIL-4 requirement ..

hga 13y ago
Would that level of quality be possible in 3.5 years, given all the other constraints? I agree about the lives, BTW, this screw up will kill people: https://news.ycombinator.com/item?id=6622634 https://news.ycombinator.com/item?id=6622634
- fit2rule 13y ago
  To answer your question: I believe it would be, yes. There's nothing 'hard' about SIL-4 programming - its just a process that you have to follow in order to ensure the quality of your software.. Of course, SIL-4 may not be the solution .. but it sure seems to me that there is no way healthcare.gov would have gotten through the process if SIL-4 was applied to it..
PeterisP 13y ago
Well, but healthcare.gov is not health-critical software, just as most other health related tools. Software on an X-ray machine is health-critical, but the software used for billing X-rays is not.
- fit2rule 13y ago
  I don't know about that - this system is supposed to help people track their expenses on drugs and treatment, and so on .. so to me it seems its not 'just' a billing software package, but rather a healthcare management system. I'd certainly feel better about using it if it had some sort of safety-critical level certification, but then again I don't have the problem of needing healthcare in America ..