5 ms·

I don't see how giving 20 startups $15 million each would have led to success instead of failure. Even if you found a better company to do the implementation t

by GVIrish 13y ago

I don't see how giving 20 startups $15 million each would have led to success instead of failure. Even if you found a better company to do the implementation than CGI Federal, there were several enormous problems that no company would have had any control over. 1. Requirements were delayed so much that development didn't start until March of this year. That spells doom for a system with this kind of complexity, regardless of who the implementor is. You need several months of functional, load, and integration testing so that effectively means you would only have had 4 to 5 months to code healthcare.gov. And that's assuming there weren't any big requirements changes. 2. The people responsible for integration (Center for Medicare/Medicaid Services) had no large IT project integration experience. These are people that thought 1 week of full integration testing would be enough. 3. Healthcare.gov had to integrate with legacy systems from the IRS, Medicare/Medicaid, Social Security, in addition to the various state exchanges. Any number of those systems could have serious flaws that would make it extremely difficult to interface with. On any project a poorly implemented legacy system can dramatically affect the effort needed to be successful. Again, even the best companies would've had a sizable challenge dealing with that. 4. One of the biggest challenges in government IT is the customer. The government decision makers often don't know enough about software engineering to make sensible decisions on requirements, timelines, testing, you name it. In this case there was the added political pressure of, "This cannot fail," even though it should've been clear at least a year ago that there's no way they were going to make the deadline. But you get people who think that you can just deploy and fix it as you go along. Or you get people that think you can just add developers to make up for lost time. 5. And what does being a "startup" have to do with it anyway? Either a software company can do the work, or they can't. Whether they are a startup or an established entity really has nothing to do with it.

integraton 13y ago
It looks like you and jroseattle are demonstrating the main two competing views about how to approach new software projects: 1) deterministic, controlled, strong planning, avoidance of failure vs 2) nondeterministic, flexible, embracing uncertainty, expect and handle failure gracefully. There are many different versions of this in technology: one big expensive server vs many commodity servers, waterfall vs agile, ACID vs BASE, BigCorp in-house R&D vs distributing risk across startups. Large parts of the technology world operate according to the latter model, and they do so for a variety of very valid reasons. Obviously, the government and government contractors do not. jroseattle's comment presents a speculative model for how to apply a distributed, fault-tolerant model to this kind of technology project.
- GVIrish 13y ago
  I'm not saying that you can't or shouldn't distribute risk across many small projects, I'm saying that in this case doing so would not have helped you at all. If you had a competition to build a MVP of healthcare.gov where the requirements didn't get delivered until 48 hours before it was due, then the requirements changed substantially 30 minutes before the cutoff time, none of the teams would have succeeded.
jroseattle 13y ago
The distribution among startups is about spreading risk for the project. The existing approach was one integrator, and they failed. Because there are so many reasons that are provided for why this had no chance of success, i.e. "Even if you found a better company to do the implementation", it makes strategic sense to me to try multiple approaches. But I would contend the problems weren't unmanageable. Problematic, yes, but manageable. I will try to address your points: 1. Requirements delay - sure, specifics were delayed. But, there was a basic premise for what this system would entail, and core work could progress without incurring much risk or waste. (I'm unsure when project work by CGI was started versus development work.) 2. IT integration experience - this is dependency management 101. If this is a recognized risk, then CGI should have raised the red flag. This looks like very poor project management. 3. Dependencies - yes, there is a possibility that external dependencies can affect your system. You design and architect for these scenarios, including failure. 4. Technical know-how - I would not expect anyone in government to know one iota about software development. CGI's job, in this scenario, is to elicit those requirements in a way that's useful. 5. Good point, but this project would benefit from some competition. Why not let more teams try this, rather than wrapping it all up with an entity who's only proven competency is simply receiving more contracts?
- GVIrish 13y ago
  The integrator in this case was the government (CMS is a division of HHS), there's no getting around them as a single point of failure. 1. Having a basic idea of what the system will do is helpful, but in a system that has a lot of very specific business logic that has to be defined and HAS to be right in order to comply with the law, you're not going to get very far without good requirements. Furthermore, depending on how the contract was structured, CGI Federal may not have been allowed to start development work until it was authorized by the customer. 2. CGI may very well have raised multiple red flags, and CMS and HHS could have promptly ignored them. That is one of the maddening things about government IT. You can tell the decision makers that they are heading for disaster and they can simply ignore you. 3. Let's say you plan for every one of your dependencies to fail. What useful product could you deliver in that case? If you don't get certain information you cannot give the user a health insurance product according to the law. This is system where you cannot reasonable ignore a component if you can't get it to work in time. Let's say the interface with the IRS is going to take longer than you thought. You can't just cut that out or substitute dummy data. That would amount to failure to deliver a functioning product. 4. Again, you can elicit as hard as you want, if they don't give you the requirements because the law is still up in the air, or because the president might change his mind, or because someone wants to meet with all the stakeholders in the other agencies first, there is absolutely nothing you can do. I've been on projects where the stakeholder with the final say couldn't be bothered to give their input until the software was done. I've also been on projects where the decision maker prevented us from getting the input of the employees that would actually be using the application. Sometimes the people you're dealing with are not rational actors. 5. Fixing the contract award process is certainly part of the solution. Normally many teams could submit a bid for the work, but due to the insane time scale it was skipped for healthcare.gov. I think having a proof of concept competition after some sort of initial proposal selection gate could be useful. It could sort of be like how the military has contractors build prototypes for weapons systems competitions.

by GVIrish 13y ago

integraton 13y ago

It looks like you and jroseattle are demonstrating the main two competing views about how to approach new software projects: 1) deterministic, controlled, strong planning, avoidance of failure vs 2) nondeterministic, flexible, embracing uncertainty, expect and handle failure gracefully. There are many different versions of this in technology: one big expensive server vs many commodity servers, waterfall vs agile, ACID vs BASE, BigCorp in-house R&D vs distributing risk across startups. Large parts of the technology world operate according to the latter model, and they do so for a variety of very valid reasons. Obviously, the government and government contractors do not. jroseattle's comment presents a speculative model for how to apply a distributed, fault-tolerant model to this kind of technology project.

GVIrish 13y ago
I'm not saying that you can't or shouldn't distribute risk across many small projects, I'm saying that in this case doing so would not have helped you at all. If you had a competition to build a MVP of healthcare.gov where the requirements didn't get delivered until 48 hours before it was due, then the requirements changed substantially 30 minutes before the cutoff time, none of the teams would have succeeded.

jroseattle 13y ago

The distribution among startups is about spreading risk for the project. The existing approach was one integrator, and they failed. Because there are so many reasons that are provided for why this had no chance of success, i.e. "Even if you found a better company to do the implementation", it makes strategic sense to me to try multiple approaches. But I would contend the problems weren't unmanageable. Problematic, yes, but manageable. I will try to address your points: 1. Requirements delay - sure, specifics were delayed. But, there was a basic premise for what this system would entail, and core work could progress without incurring much risk or waste. (I'm unsure when project work by CGI was started versus development work.) 2. IT integration experience - this is dependency management 101. If this is a recognized risk, then CGI should have raised the red flag. This looks like very poor project management. 3. Dependencies - yes, there is a possibility that external dependencies can affect your system. You design and architect for these scenarios, including failure. 4. Technical know-how - I would not expect anyone in government to know one iota about software development. CGI's job, in this scenario, is to elicit those requirements in a way that's useful. 5. Good point, but this project would benefit from some competition. Why not let more teams try this, rather than wrapping it all up with an entity who's only proven competency is simply receiving more contracts?

GVIrish 13y ago
The integrator in this case was the government (CMS is a division of HHS), there's no getting around them as a single point of failure. 1. Having a basic idea of what the system will do is helpful, but in a system that has a lot of very specific business logic that has to be defined and HAS to be right in order to comply with the law, you're not going to get very far without good requirements. Furthermore, depending on how the contract was structured, CGI Federal may not have been allowed to start development work until it was authorized by the customer. 2. CGI may very well have raised multiple red flags, and CMS and HHS could have promptly ignored them. That is one of the maddening things about government IT. You can tell the decision makers that they are heading for disaster and they can simply ignore you. 3. Let's say you plan for every one of your dependencies to fail. What useful product could you deliver in that case? If you don't get certain information you cannot give the user a health insurance product according to the law. This is system where you cannot reasonable ignore a component if you can't get it to work in time. Let's say the interface with the IRS is going to take longer than you thought. You can't just cut that out or substitute dummy data. That would amount to failure to deliver a functioning product. 4. Again, you can elicit as hard as you want, if they don't give you the requirements because the law is still up in the air, or because the president might change his mind, or because someone wants to meet with all the stakeholders in the other agencies first, there is absolutely nothing you can do. I've been on projects where the stakeholder with the final say couldn't be bothered to give their input until the software was done. I've also been on projects where the decision maker prevented us from getting the input of the employees that would actually be using the application. Sometimes the people you're dealing with are not rational actors. 5. Fixing the contract award process is certainly part of the solution. Normally many teams could submit a bid for the work, but due to the insane time scale it was skipped for healthcare.gov. I think having a proof of concept competition after some sort of initial proposal selection gate could be useful. It could sort of be like how the military has contractors build prototypes for weapons systems competitions.