9 ms·
How Healthcare.gov Botched $600M Worth of Contracts
- reustle 11y agoThis seems to put most of the blame on the gov workers managing the contracts, and very little on the contracted companies themselves. I would find it hard to believe that they also weren't absolutely sure what they were milking.
- kctess5 11y agoIt sounds like the people hiring the contractors didn't exactly do their due diligence. > unclear who is really in charge, sometimes it's no one > vague deliverables from people who don't know what they're talking about > tons of cash, regularly, with no end in sight Sounds like an incredibly profitable and easy contract. It's an ethical grey area, but it's not surprising that companies took advantage.
- OliverJones 11y ago"easy contract?" I doubt it. Ten million users on the go-live date? Interchange of health-care data from system to system? Getting Oracle's identity / authentication management system integrated as a door-one requirement? Hostile congresscritters doing everything in their power to torpedo the project? The only difference between this and a typical giant cluster#uck is the vast and visible scale.
- humanrebar 11y ago> I would find it hard to believe that they also weren't absolutely sure what they were milking. It was pretty clear that the project was doomed to start with. Maybe some companies didn't bid because they knew that, leaving only companies that were either naive or willing to get paid for a broken product and then blamed and abused after. In other words, I completely disagree. The buck absolutely needs to stop at the people putting out the contracts and spending the money.
- tssva 11y agoThis is one of several IG audits covering the healthcare marketplace roll out. The purpose of this particular audit was to review the government's management of the contractors.
- jakozaur 11y agoLesson for government: Software is not a typical infrastructure project (e.g. bridge). It is almost impossible to specify all requirement upfront. That's why waterfall methodology is prone to spectacular failures. Training government officials how to manage those project can help, but I doubt it will make a difference. It can rather give false sense of "fixing it" without actually fixing it. I would rather focus on hiring competent ppl and give them freedom to do that right. Such as: https://www.whitehouse.gov/digital/united-states-digital-service https://www.whitehouse.gov/digital/united-states-digital-ser...
- x5n1 11y agoCompetent people with a long history of developing whatever you are actually looking for. Software is the wild wild west. And there are many caveats because lots of people claiming to know what they are doing who know nothing. And even corporations who don't know what they are doing, but at good at marketing to get the contract.
- geomark 11y agoUS gov has been doing software projects for as long as there has been software. So it's not like there _should_ be new lessons. They could just look at the vast number of previous catastrophes, and the few that went reasonbly well. I've been involved in a few myself as an employee of a government contractor. The gov people overseeing the projects were always in over their heads. I think that's a common issue.
- threeseed 11y agoI wouldn't lay blame at the government over this. I have seen countless large implementations fail within enterprise companies i.e. massive overrun in budgets, not meeting requirements. And unfortunately waterfall is really the only approach for projects like this i.e. you are having to select different third party vendor products up front and integrate them with custom applications. You can't iterate in sprints when you don't control the software. And if only hiring competent people was so easy. Really talented IT professionals are very hard to find and keep. The Digital Service you mentioned really is a great step towards retaining talent and long overdue.
- devit 11y agoLooks like they should look at who stole the money. Even a very generous 100 engineer team paid at 300k/year for 3 years only costs 100 millions. If no one is looking into it, it's reasonable to assume that whoever is supposed to look into it is the one who stole the money.
- thaumasiotes 11y ago> If no one is looking into it, it's reasonable to assume that whoever is supposed to look into it is the one who stole the money. While this isn't exactly unreasonable, I don't see why it's such a slam dunk compared to "whoever is supposed to look into it is lazy" or "there is no one who's supposed to look into it".
- DanBC 11y agoBotched health IT projects aren't new The UK has had some horrifically overspent and underdelivering health IT projects. It's a shame that people don't appear to learn from other people's earlier mistakes. http://www.telegraph.co.uk/news/health/news/8780566/Disastrous-11.4bn-NHS-IT-programme-to-be-abandoned.html http://www.telegraph.co.uk/news/health/news/8780566/Disastro... £11.4bn !
- themartorana 11y agoYou presume the collectors of that $11B and those that benefit politically and financially indirectly (revolving doors) want to "learn."
- gaius 11y agoThis is why I laugh when people complain about the cost of a Trident replacement... You could pay for it out of the NHS IT budget and neither patients nor medical staff would even notice.
- smoyer 11y agoI'm a bit shocked that anyone could believe there was actually $600M of software to build. I know prices are inflated by the whole federal contracting guidelines but I'd say these contracts were at least two orders of magnitude greater than what they would have been in the private sector. And nothing against our friends to the north, but why wouldn't we bid more of this out to US companies? Perhaps those that satisfy the WO-MO requirements?
- wil421 11y agoIt boggles my mind that they spent so much. CGI Federal got almost 300 million. The company I work for has an IT budget not much smaller. We employ over 1,000 people around the world in IT alone. They must've had an army of people working on crap apparently. I'm sure they higher ups padded their wallets and walked away happy whether it worked or not. The first mistake was not finding a department with big budget experience.
- newjersey 11y agoPart of the issue is everyone who is present at every meeting gets billed for the time at the meeting for the project. Mikey Dickerson talked about how there were ~50 contractor firms and nobody seemed to be in charge. This is good for all contractors because if nobody's in charge then nobody gets in trouble. https://www.youtube.com/watch?v=7Vc8sxhy2I4 https://www.youtube.com/watch?v=7Vc8sxhy2I4
- giancarlostoro 11y agoI think that is the one thing about all of this that saddens me the most. There's definitely enough talent to make such a project function, and yet we totally botched that up completely. I've known teenagers in my younger days who've created much more complicated projects in shorter spans of time.
- dasil003 11y ago> I've known teenagers in my younger days who've created much more complicated projects in shorter spans of time. No you haven't, and the hyperbole doesn't help. It's quite possible the specification of all the legislation is actually logically unsolveable since it was created by thousands of different people over the course of decades with very little coordination. When the time comes to build a unified system it's no wonder the job is very very difficult.
- OliverJones 11y agoI find it interesting that this audit only covers administrative contract oversight. The failings uncovered seem to have been of the variety "Contractor A didn't submit monthly reports for February and March". I don't claim that contracts of that size should have poor administrative oversight, not at all. Even movies with big budgets mention the assistant producers and accountants in the credits. But what about technical oversight? Were there design-review meetings where somebody said "our system emits data in HL7" and somebody else said, "we haven't defined the XML schema for our data acceptance module yet." In a meeting like that somebody in tech oversight should have said "wait! is it HL7 or XML? Who is going to sort this out?" (I have no idea whether this actual issue came. Probably the stuff that actually came up, or should have, is far more intricate.) The purpose of administrative monthly reports' "issues" section is to track stuff like like this. That's why contracts require them. If the government can make the Apollo program work, the government can figure out technical oversight. It just takes the will to do so.
- viraptor 11y agoI see it as a high/low level oversight. If people don't know who controls the project and how to measure the progress on it, then who would even care about technical reports? That's like checking the details on a body with a head missing - maybe the guts are OK, maybe not. He's dead anyway.
- bingobob 11y agoits a joke software and governments here in Australia checkout out this mess. 1.5 billion on a Public transport metro card system https://en.wikipedia.org/wiki/Myki#Issues_and_criticisms https://en.wikipedia.org/wiki/Myki#Issues_and_criticisms 1.2 billion on a payroll systems WTF!! http://www.theaustralian.com.au/business/technology/queensland-government-to-sue-ibm-over-health-payroll-disaster/story-e6frgakx-1227140553687?nk=43303f5167de2715cd9df22efb653154-1442751558 http://www.theaustralian.com.au/business/technology/queensla...
- noir_lord 11y agoUK's Connecting for Health System > Originally expected to cost £2.3 billion (bn) over three years, in June 2006 the total cost was estimated by the National Audit Office to be £12.4bn over 10 years Which is roughly 27bn Australian Dollars, when it comes to this stuff our Government is in a class of it's own.
- pbhjpbhj 11y agoIIRC the system was then abandoned as unworkable?
- noir_lord 11y agoMost of it was, some bits where salvaged (I think, I followed it at the time) at huge cost and then they built a newer version which sorta works. In any base £20bn over 10 years seems like quite a lot on an initial budget of 2.4bn over 3.
- delecti 11y agoWow, even including the cost of outfitting all the readers, that metro card system has got to be at least two orders of magnitude higher than reasonable.
- gaius 11y ago1.2 billion on a payroll systems WTF! http://c2.com/cgi/wiki?WhyIsPayrollHard http://c2.com/cgi/wiki?WhyIsPayrollHard
- Fede_V 11y agoThe solution proposed to fix this mess is insane. A '96 hour training course' at the end of which you are able to handle 100+ million projects? That's 2 weeks of work + change. Does anyone think that they could end up qualified to handle contracts of that size after a generic 2 weeks course? Of the people that enroll in this program, what % fail? If it's serious, the % of people qualified should be at most 50% - yet it's obviously going to be just a rubberstamp where you put in the hours, then get the certification. As someone who firmly believes in the power of the state to do good, these things make me go insane. The solution to failures of this magnitude are not to pile on more useless regulations and certifications, or to put even more emphasis on useless credentialism. Healthcare.gov got fixed when they gave carte blanche to highly qualified engineers and managers that gave up very lucrative jobs to help fix things. I am convinced that lots more people would be willing to give up their cushy tech jobs for a job where they genuinely made a huge difference in the lives of people, provided you got rid of all this completely meaningless hoop jumping. Give clear lines of responsibility, take a serious look at your hiring process, and jesus, if your solution to a 600+ million dollar failure is to mandate a 2 week course... you really need to think things over.
- droopyEyelids 11y agoGlad I wasn't the only one who balked at the idea it would have gone well if everyone had more training. I think we really need intelligent, experienced people who have a proven record in large product management. However, they're rare and more expensive than the government will pay for. Compounding the failure is the fact that you can't get fired from government jobs for incompetence. These projects aren't going to get any better.
- Fede_V 11y agoI mostly agree, but I don't think that it's just a matter of money. Money is a way to motivate people - but for lots of people, the ability to work on a system that will cause a huge improvement in the lives of hundreds of millions of people is reward enough. However, nothing saps motivation more than pointless credentialism and bureaucracy.
- PretzelFisch 11y ago600m and the gov still has to take the contractor with the lowest bid still? Makes me think the requirements were so broken no one really wanted the work.
- mentos 11y agoCould this project have been completed by the likes of Google?
- deleted 11y ago[deleted]
- DanielBMarkham 11y agoThis will go into the big stack of incredibly large technology project failures, including the FBI case management system and the IRS modernization program. From a systems point of view, the problem here is the publicity. For each public disaster, there are many more that never get that much attention. Just got through reading a biography of John Boyd. Great narrative about how huge systems of people act politically to resist change and make decisions.
- nolepointer 11y agoI expect nothing less from the federal government.
- dangero 11y agoTangential question, how do software firms get contracts like this? Seems like a good business to be in.
- thrownaway2424 11y agoNavigating the byzantine contracting system is the only skill these companies have. They can't do the job, but they can fill out the forms, which is a rare skill.
- psykovsky 11y agoYou mean reading and following the instructions manual is hard work beyond the reach of a normal human being??
- thrownaway2424 11y agoIf you think it's easy to win government contracts then I invite you to do so. It can be lucrative.
- brandonb 11y agoHey HN! I worked on the "tech surge" to save healthcare.gov, and as many of you have noticed, this report gets everything backwards. A 96-hour training course would not have saved healthcare.gov, and you can't fix bureaucracy with more bureaucracy. If you want to understand what went wrong on healthcare.gov and how it was fixed, Mikey Dickerson's talk here is the closest thing we have to a postmortem: https://www.youtube.com/watch?v=0albm_hhQzM healthcare.gov has also given rise to new initiatives to fix the way the government writes software. Within the government, there is the US Digital Service, 18F, and the Presidential Innovation Fellows: https://medium.com/@USDigitalService/an-improbable-public-interest-start-up-6f9a54712411 18f.gsa.gov http://www.jasonshen.com/2015/when-did-you-do-your-tour-of-duty/ http://www.jasonshen.com/2014/everything-wanted-know-presidential-innovation-fellowship-afraid-ask/ There are also new types of government contractors, including Nava and Ad Hoc LLC: http://www.fastcodesign.com/welcome.html?destination=http://www.fastcodesign.com/3047856/innovation-by-design/meet-nava-a-startup-that-wants-to-fix-the-governments-crappy-design http://adhocteam.us/
- DanielBMarkham 11y agoPlease forgive the directness, but I need to ask you this because I have noticed a pattern. Whenever HealthCare.gov gets mentioned, there's always the most over-the-top enthusiasm for how it was such a great turnaround. I've worked with dozens of large projects -- including ones that were turned around -- and this is new to me. The only thing new I can see here is the political environment around the healthcare law. Hopefully I am wrong. So when the Republicans take control over the executive branch, this time or whenever in the future, are we going to keep hearing such great boosterism? The Republicans decide to automate voter identification and the project goes south, will we see such great turnaround stories? It would be very comforting to know that the new Digital Service and the influx of enthusiasm is not limited to one party or large government project. Some assurance would be most appreciated.
- gajomi 11y ago"So when the Republicans take control over the executive branch... are we going to keep hearing such great boosterism?" WHy are you asking him? I don't think anyone knows the answer to this. Obviously the major factor would be whether the next president (be they Republican or Democrat) decide to continue support for these kinds of teams, both in terms of funding and autonomy to do their job. This seems like something we should be asking the candidates and their parties about.
- sylvinus 11y agoYou can watch a very entertaining talk from one of the Google engineers who went to fix the website at the time: https://www.youtube.com/watch?v=GLQyj-kBRdo https://www.youtube.com/watch?v=GLQyj-kBRdo
- nodesocket 11y agoThe most ridiculous part, they didn't have a way to run distributed commands over ssh. When a filesystem mounted in read-only mode, somebody had to ssh into each of the 175 servers manually and run commands to fix it.
- psykovsky 11y agoWhat do you mean by "distributed commands over ssh"?
- garrettgrimsley 11y agoSay you want to run a command on a group of machines using SSH. Rather than manually running the command on each remote machine you can automate their execution over the group using what is known as distributed commands. http://www.tecmint.com/using-dsh-distributed-shell-to-run-linux-commands-across-multiple-machines/ http://www.tecmint.com/using-dsh-distributed-shell-to-run-li... http://www.ibm.com/developerworks/aix/library/au-satdistadmin/#N100E1 http://www.ibm.com/developerworks/aix/library/au-satdistadmi...
- spoiledtechie 11y agoDisclaimer: Highly political rant. You mean to tell me a company that got selected a no bid contract from their connections in the Obama administration couldn't do an adequate job? Crony politics at its best.
- pbreit 11y agoThis is why I wonder how much we need more and better coders versus better managers?
- ownagefool 11y agoYou need both. The biggest issue I take with good coders lead by non-technical managers is they invariably convince managment they need to rewrite everything in the flavour of the month or architect in requirements so they can use technologies they're interested in. Inevitably, these guys are now qualified with a whole new bunch of tools, so they move on and hopping about the industry writing a whole bunch of software that relies on a datastore that doesn't actually save data consistently, for example.