5 ms·

Poorly Managed HealthCare.gov Construction Cost $840M, Watchdog Finds

61 points by malchow 12y ago

malchow 12y ago
Here is the full testimony to be delivered tomorrow: http://docs.house.gov/meetings/IF/IF02/20140731/102587/HHRG-113-IF02-Wstate-WoodsW-20140731.pdf http://docs.house.gov/meetings/IF/IF02/20140731/102587/HHRG-...
Daishiman 12y ago
Is it me or is that not that much money for a system that serves a 350 million-person nation?
- wyager 12y ago
  It certainly doesn't serve all 350 million people. And we don't really know what kind of peak loads it can handle, so that might not be very impressive. Also, considering what it does (Which it might not even do well; time will tell.), it's not exactly computationally challenging.
  - tzs 12y ago
    Doesn't it have to interface to a whole bunch of other government and commercial sites, which have their own APIs and data formats? I believe I read somewhere (probably earlier HN discussion months ago) that dealing with all that made this a ridiculously more complex project than it might first appear based on what you actually see on the site.
    - wyager 12y ago
      That's probably true, and that's where I'd guess most of the cost came from. I'm just saying that if they only had to serve 350,000 people it probably would have been almost the same cost.
    - kkhire 12y ago
      Yea, I'm pretty positive you're both correct--it was a government related project that they needed utmost of security and secrecy while rapidly building it. All of which are very challenging obstacles for developers
      - wyager 12y ago
        >they needed utmost of security I can absolutely guarantee you that they do not have the utmost of security. For a project to be really secure, you need extremely talented and meticulous developers. Judging by the clusterfuck that was Healthcare.gov's original launch, that does not describe the people involved. For example, this sentence (lifted from Wikipedia) raises a million red flags in my mind re. data consistency checking and sanitation: "Even for those that did manage to enroll, insurance providers later reported some instances of applications submitted through the site with required information missing.". No project like that can feasibly have the "utmost of security".
        kkhire 12y ago
        haha needed and what they got are entirely two different things. It's unfortunate, but the problem was that because it was a government website, the developers were strictly not allowed to get help, i.e. have someone review it/ask a friend for help/post on stackoverflow/etc. this probably was the source of the initial failure.
      - spacehome 12y ago
        The final product they produce needs security to protect the end users' medical records, but why on Earth would you need secrecy for what is essentially a public works product?
    - dkhenry 12y ago
      So if your trying to point out that this is more then just a custom wordpress site then your correct. If however you perceive this to be some intrinsically complex task requiring extensive technical expertise you are wrong. A majority of that money is most likely overhead and even then the development is done via integrations with enterprise vendors so another large chunk of change goes to pay the IBM's and Oracle's of the world for their databases and webservers along with their "consultants" to get it all tied together. FWIW the current enrollment is 6mil so if you assume a very poor conversion rate of 1% then that is 600 million page views over the signup period and that comes down to about 38 views per second. At peak load that would only be like 1k concurrent views ( assuming a poisson distribution )
      - threeseed 12y ago
        > If however you perceive this to be some intrinsically complex task requiring extensive technical expertise you are wrong. Have you ever worked on these projects before ? It DOES require technical expertise. Working with legacy and enterprise systems requires knowledge that most developers think they could handle until they are actually in the weeds trying to understand some arcane domain model or file format. Combine that with the integration and scaling challenges and it is quite a technical problem. Is it expensive sure. But to dismiss these projects without any visibility or understanding of the challenge requires some arrogance.
        dkhenry 12y ago
        I used to be a government contractor so I am well aware of the challenges of integrating the various systems. The challenges are almost never technical I mean consider your argument here you essentially have a system connecting multiple databases that needs to retrieve and reformat the data then store it again. That process is not complex even at the scale they are doing it ( 8 Million registrants, even 350 Million wouldn't be a problem for most database systems ) Yes documentation is always lacking and there is some reverse engineering that needs to be done, and yes there will always be some stupid wrinkle in the process, but this is a well understood problem and claiming its a challenge due to "Legacy Systems" is just a way of justifying the over 100% overhead that the government runs.
  - threeseed 12y ago
    It may not be computationally challenging. But if you are integrating with dozens of legacy systems then getting that right is hardly trivial. Look at Netflix and Twitter and the work they did around resilience between their micro services. Getting that to work every time with all failure paths handled correctly is definitely architecturally challenging.
    - wyager 12y ago
      >Getting that to work every time with all failure paths handled correctly is definitely architecturally challenging. You're right. Good thing they completely failed to handle all failure paths correctly. This is extensively documented at http://en.wikipedia.org/wiki/HealthCare.gov#Launch_and_technical_problems http://en.wikipedia.org/wiki/HealthCare.gov#Launch_and_techn...
      - threeseed 12y ago
        I'm not saying that they did a perfect job of it. We all know they didn't. I am just disputing anyone who thinks projects like this aren't technically challenging.
- TallGuyShort 12y ago
  Can't get past the paywall on my phone, but 350 million people in the country, limited to working age adults, limited to those who didn't use state-level exchanges, limited to those who wanted to use the public option at all, and you're quickly at tens of millions, possibly less, spread over several months. And it sucked at serving that many people.
  - hippich 12y ago
    But these 10 millions who now got a chance to have insurance save dollars for the rest 340m. So indirectly it indeed served all 350m (or whatever total number of people living on USA territory)
    - TallGuyShort 12y ago
      That sense of the word "served" has nothing to do with the magnitude of the technical project.
    - jdjdjdidjd 12y ago
      How many dollars were saved exactly? Do you have a source? Anecdotally, I know plenty of people who saw an increase in their premiums.
      - hippich 12y ago
        I don't have numbers, nor anyone will have I bet, since that kind of change requires some time to really see benefits (or negatives) My opinion based on the fact that nothing is free, and people who currently don't have insurance, but still require medical attention, still cost the rest of us money. Also, since people without medical insurance tends to go to emergency room only in critical cases - problems they had could be much cheaper if they had insurance from the beginning. Anecdotally, I know such people :)
      - XorNot 12y ago
        Anecdotally, you met a lot of people who had health insurance they never actually used, and which wouldn't cover them for anything if they needed it. Those are the people who's plans were discontinued.
- tdicola 12y ago
  It's nothing compared to something like the B2 bomber which cost $40+ billion to develop and each one costs over $700 million. We built 20 of them. (they are still pretty awesome cold war relics though) edit: Or Nimitz class nuclear aircraft carriers for $4.5 billion each, with 10 of them built.
  - EpicEng 12y ago
    Comparing software development to hardware (+ software) development is not fair. I'm almost certain there was waste involved in that venture as well, but hardware is expensive.
- jdjdjdidjd 12y ago
  Yes, this explains why every Silicon Valley web company needs $840,000,000 in funding too before they can serve a small percentage of the US population. Oh wait.
  - threeseed 12y ago
    Do you really think this is in any way comparable to a web company ? The project was never about just the website. It is the huge amounts of behind the scenes integration with legacy systems. Some of which may require specialist development experience which never comes cheap.
    - jdjdjdidjd 12y ago
      I agree that this is not a typical web company project, but if you read the report, a lot of that money was for naught. http://blogs.marketwatch.com/health-exchange/2014/07/30/gao-to-point-finger-at-managers-for-obamacare-debacle-report-says/ http://blogs.marketwatch.com/health-exchange/2014/07/30/gao-... "Confusing and constantly changing marching orders from Centers for Medicare and Medicaid Services managers will get the blame for last year’s HealthCare.gov debacle under an independent government review, the Associated Press said Wednesday."
      - threeseed 12y ago
        >a lot of that money was for naught You've provided no evidence of this. Including that article. I am just disputing your assertion that this is comparable to your typical Silicon Valley web company who rarely have to deal with legacy systems. Not whether it was a well run project.
    - karmelapple 12y ago
      Is there a technical breakdown anywhere that talks about the specific technical challenges, such as which specific legacy systems were interfaced with? If there were real technical challenges, there was likely a risk register of some sort tracked by project management for this contractor, which would have listed any major technical issues like that.
- nostromo 12y ago
  The federal site only had 5.5 million people sign up for healthcare. That's $150 per registered user. http://www.businessinsider.com/how-many-people-signed-up-for-obamacare-2014-5 http://www.businessinsider.com/how-many-people-signed-up-for...
  - jjoonathan 12y ago
    So less than 2 weeks of health care coverage per user. While we're on the subject of waste, the USA's ongoing experiment with market-based health care costs ~20 weeks of coverage per user per year compared to its social counterparts. http://healthcarereform.procon.org/view.resource.php?resourceID=005522#III http://healthcarereform.procon.org/view.resource.php?resourc...
    - beat 12y ago
      Market-based isn't the problem. Switzerland, Germany, and Singapore all have mostly-private competition-driven systems (more private than the American model, even), and they are inexpensive and provide high-quality care, along with universal coverage. So private insurers aren't why the American system is so uniquely incompetent and inefficient. So what is the problem? The way I see it, there are two features that really distinguish the American system structurally. First, there's the state-by-state laws and regulations, resulting in effectively 51 different legal regimes. Second, American insurance is provided primarily through employers, rather than purchased individually. In fact, non-employer insurance is prohibitively expensive and often impossible to get. I think the employer role is a much more likely cause of problems than the state laws, which are mostly consistent. Worse, tying insurance to employment traps people in jobs they hate - a distinctly un-American outcome. So anyway, I used to think a more socialist system was the right answer too, until I looked more closely at the working systems in the world. It's not that one particular model works, because there are lots of models in use in other countries and they all work fine. It's that one model doesn't work - what we have in America.
      - waps 12y ago
        I was going to say something about how the US market is a lot bigger than the Swiss market, but I've had to reconsider. Switzerland is a country of 6 million people and is about the size of an "average" US state. Most US states are a little smaller in population (If Switzerland was a US state, it'd be the 13th largest), and a LOT larger in area. So for a "normal" state, Switzerland actually has a bigger healthcare market. California, about 5 times the size of the Swiss healthcare market, still manages to screw things up royally though. People are often blaming the US legal rules. In Switzerland, if you want to win a court case that a MD acted improperly (justifying damages), you have to get a jury of his peers (ie. MDs) to agree on that (which is not impossible, but ...). In the US, you have to get a jury of patients to agree on that. Needless to say, there are many, many more medical cases in the US, with spectacular damage payments (damage payments are also standardized in the EU. If an MD kills you through negligence or outright a mistake (not satisfying the level of a criminal act), his employer (even though they're mostly freelancers, this is an exception in the law) owes your family 300k euro (inflation adjusted since 19xx, so it's a bit more). In the US, it's not fixed, judges decide, but it seems that number runs into the tens of millions of dollars.
Acen 12y ago
Is there a way to bypass the pay wall to read this?
- nostromo 12y ago
  Click here: https://www.google.com/search?q=Poorly+Managed+HealthCare.gov+Construction+Cost+%24840+Million&tbm=nws https://www.google.com/search?q=Poorly+Managed+HealthCare.go... Then click the story.
  - click170 12y ago
    IMO you shouldn't have to do that, so I flagged it.
vnchr 12y ago
Consulting firms like Deloitte were taking payments of +$70 million[1] to do "analysis" that "informed development and decision making." The associated waste of resources is unfathomable in the startup world but seems to be common with federal contracting. 1. Tipsy Deloitte associate chatting at a Chicago bar
- doubt_me 12y ago
  Why the hell does everything the government do have to go through a contract and outsourced/ bid on? Why don't they contract someone to come up with a better solution?
  - adamnemecek 12y ago
    I've had someone explain it to me as follows. If shit hits the fan and you are the one responsible for hiring Joe Schmo, your job might be on the line. If you've hired someone like Deloitte and then do what they tell you and shit still hits the fan, you can always say 'I did what all what I could have done'.
    - rahimnathwani 12y ago
      aka 'Nobody got fired for hiring IBM'.
  - dragonwriter 12y ago
    > Why the hell does everything the government do have to go through a contract and outsourced/ bid on? Because: 1) There are lots of people ideologically motivated to minimize the "size of government" as measured by public payroll, and almost no one ideologically motivated in the opposite direction, and so contracting out helps to improve a metric that is important to a large number of stakeholders even if it is substantively meaningless, and 2) There's lots of money in firms that are specialized in taking government contracts, and they both lobby government and engage in public political advocacy effectively. And, no, those two factors are not independent -- #2 contributes directly to #1. (Also, outsourcing blame as adamnemecek points to in a sibling comment, can be a factor where the people who would potentially be at risk of blame actually have discretion of whether to do something internally or contract out, but that's often not the case.)
alien3d 12y ago
Wow.quite cheap...
brandonb 12y ago
I was part of the "tech surge" brought in to help patch up healthcare.gov in December. The GAO report gives a decent idea of what went wrong in 2010-2013 from a high-level contracting and policy perspective, but it doesn't really give much technical detail or describe what was broken about the organization, software architecture, and culture. A couple of my colleagues have given talks on healthcare.gov from the perspective of the engineers who worked around the clock to help get the site running again: http://youtu.be/0albm_hhQzM?t=3m40s http://youtu.be/0albm_hhQzM?t=3m40s http://www.youtube.com/watch?v=GLQyj-kBRdo http://www.youtube.com/watch?v=GLQyj-kBRdo A small team is also re-writing the site, starting with the most troublesome components for the next open enrollment period. Wired covered some of what is going on here: http://www.wired.com/2014/06/healthcare-gov-revamp/ http://www.wired.com/2014/06/healthcare-gov-revamp/ If any of you out there want to help, please email jobs@hcgov.us!
- butner 12y ago
  Fair perspective, but the government is still trying to solve a lot of the problems with closed-doors and the same contractors. There's a lot of startups that are focused on solving the problems of getting people health coverage. A bigger challenge is ensuring people get the right coverage for their situation and understand the plan and provider network they're buying. Disclaimer: I work at Stride Health, we help individuals buy plans in-context of their doctors, drugs, and health conditions they're treating. We're also hiring, sans the $840MM budget (https://stridehealth.com/jobs https://stridehealth.com/jobs)
Ecio78 12y ago
Each time I read about this I think about our dear Italian tourism portal www.italia.it that costed us more than 58mil€ in 2004-2008 and was a huge failure. According to wikipedia[1] we have then spent additional 10mil for redoing it, plus 20mil for managing it in these years. Interesting for the HN designers is probably the "nice" logo[2] that won a 100.000Euro logo competition in 2006 (it was created by Landor, a US or UK brand design company...) There were jokes about the green "t" that looked like a cucumber (and in Italy the cucumber - "il cetriolo" - has sometimes a double meaning).. [1] http://it.wikipedia.org/wiki/Italia.it http://it.wikipedia.org/wiki/Italia.it [2] http://www.marcosansalone.com/creativeantblog/?p=310 http://www.marcosansalone.com/creativeantblog/?p=310
sytelus 12y ago
This number may look big but remember average per head cost of a software engineer is about 230K/yr. Any software project that takes 3 years and requires ~1000 engineers would approximately cost similar amount. On the side, I'd first hand experience with healthcare.gov to help someone else. I can imagine myriads of integration points between insurance companies, keeping track of their plans, state level requirements, credit checks, other govt agencies like social security, customer service backend etc. From my experience in working with this kind of complexities I think 200-300 engineers may be more than enough for 2 year execution plan. So still it's about 3X-4X waste when government gets involved. PS: Per head cost needs to include base salary + bonuses + stock grants + hiring fees + employee events + office expenses + subsidized cafeteria + health benefits + 401K etc etc.
colmmacc 12y ago
Maybe this is facile, maybe it is illustrative, but I find it interesting to compare HealthCare.gov and Twitter. Twitter has raised over a billion in funding so far, has been going 8 years and has over 3,000 employees. Healthcare.gov is at about 3 years, $840M, and I can't find a number for how many people were involved, but it's likely also in the thousands. Healthcare.gov has delivered life-altering value to about 6 million people and cost about $150 per user (soon amortizable over at least two years). Twitter has about 200 million active users getting everything from minor titillation to revolutionary aid and has cost about $6 per user. Neither of those "prices" seems shocking to me, and I'd pay $6 to use twitter and $150 to use Healthcare.gov. If anything, healthcare.gov seems cheap in that context. In terms of scale of effort; if I took a cursory look at twitter's functionality, and even taking scale into account [1], I'd spitball it as taking maybe 3 or 4 small dev teams for the website and API, and another dev team per app; maybe 100 engineers total. I'm sure they have more than that though; because a cursory spitball guess is an insult and there's likely a lot of hidden complexity and business logic discovered only you if try to do something like twitter very well. I'm also sure there's a huge mass of employees dedicated to wet, human, problems too like sales acquisition and support and what direction the shading should be on the twitter eggs and why the logo should just be a bird. And so it likely is with Healthcare.gov, interfacing with hundreds of different healthcare providers, handling different administrations and so on; probably a real deep mess. I'm impressed they got it working at all; especially as when they set out it was never intended to be used much (as I understand it the idea was that the state exchanges would handle the larger burden). The frontend seems to have been shoddy, but how much of the expense was that? [1] I have built and operate services at the same scale as twitter.
- waps 12y ago
  This does not seem to be a fair assessment. The "life-altering value" is not provided by the cost of the IT project, but by the total cost of healthcare in the US. That's 3.8 trillion dollars, or 11900 USD per person per year (first figure divided by 300 million). Wikipedia lists the per capita expenditure as 8500 USD per person per year for 2011, not sure why there is such a large discrepancy. Total amount of people on healthcare in the US is listed as 11.7 million jobs.
known 12y ago
Mediawiki is better for storing/retrieving UNSTRUCTURED data
Daishiman 12y ago
It's amazing the amount of downvotes on posts that comment just how tremendously difficult it is to plan, design, code, test, and support a website that has to serve literally hundreds of use cases and is meant to last multiple decades and integrate with thousands of other government systems that have been standing since the 60s and all have their own idiosyncracies in protocols, documentations, and bureaucracy. It looks like most people in HN have never actually worked on large systems and have no idea just how costs increase exponentially, how internal politics and policies affect those projects, and how infernally hard it is to think in terms of systems that have to be used by more than a handful of use cases. There's not doubt that government projects have a huge amount of internal overhead, but if you compare them to the stuff that goes on in multinationals that have working systems in the 60s, it's really nothing out of the ordinary. I know a guy who worked in a project related to one of the largest insurance companies in the US. It's a 60 year-old system running on IBM System Zs, AS/400s, Unix and Windows systems, with varying degrees of hardware and software obsolescence, running custom COBOL builds and compilers, integrating with thousands of external APIs, on half a dozen data centers with completely different specs. There was a literal library of documentation spanning millions of pages of requirements to describe how policies flowed through the systems, how batch processes should run, and what tests should be used to make sure a lady who took out an insurance policy on her TV in the 70s can still cash out on it now and withing 20 years in the future. All this does not even include the network topologies spanning TCP/IP, custom frame relay protocols, proprietary NICs, SANs for storage and backup, tape libraries, electrical layouts for data centers, etc. There's more than a few reasons why large software projects fail; I have not even begun to include the intricacies of the aforementioned industry and have not even counted the regular issues of any project.
- lifeformed 12y ago
  Sure, it's quite an engineering challenge, but... $840M? That's a staggering figure.
  - Daishiman 12y ago
    Have you seen the costs of similarly large software projects? It's actually a figure well within those margins. For the same insurance company I quoted, the people in question estimated that modernizing their system would cost $500 million USD in 1997 dollars and take 6 years without considering continuous upgrades of code and infrastructure. And it's likely that this company's infrastructure was probably much cleaner and uniform that goverment agencies'.
rakoo 12y ago
So, where's the github repo ?

61 points by malchow 12y ago

Here is the full testimony to be delivered tomorrow: http://docs.house.gov/meetings/IF/IF02/20140731/102587/HHRG-113-IF02-Wstate-WoodsW-20140731.pdf http://docs.house.gov/meetings/IF/IF02/20140731/102587/HHRG-...

Daishiman 12y ago

Is it me or is that not that much money for a system that serves a 350 million-person nation?

wyager 12y ago
It certainly doesn't serve all 350 million people. And we don't really know what kind of peak loads it can handle, so that might not be very impressive. Also, considering what it does (Which it might not even do well; time will tell.), it's not exactly computationally challenging.
- tzs 12y ago
  Doesn't it have to interface to a whole bunch of other government and commercial sites, which have their own APIs and data formats? I believe I read somewhere (probably earlier HN discussion months ago) that dealing with all that made this a ridiculously more complex project than it might first appear based on what you actually see on the site.
  - wyager 12y ago
    That's probably true, and that's where I'd guess most of the cost came from. I'm just saying that if they only had to serve 350,000 people it probably would have been almost the same cost.
  - kkhire 12y ago
    Yea, I'm pretty positive you're both correct--it was a government related project that they needed utmost of security and secrecy while rapidly building it. All of which are very challenging obstacles for developers
    - wyager 12y ago
      >they needed utmost of security I can absolutely guarantee you that they do not have the utmost of security. For a project to be really secure, you need extremely talented and meticulous developers. Judging by the clusterfuck that was Healthcare.gov's original launch, that does not describe the people involved. For example, this sentence (lifted from Wikipedia) raises a million red flags in my mind re. data consistency checking and sanitation: "Even for those that did manage to enroll, insurance providers later reported some instances of applications submitted through the site with required information missing.". No project like that can feasibly have the "utmost of security".
      - kkhire 12y ago
        haha needed and what they got are entirely two different things. It's unfortunate, but the problem was that because it was a government website, the developers were strictly not allowed to get help, i.e. have someone review it/ask a friend for help/post on stackoverflow/etc. this probably was the source of the initial failure.
    - spacehome 12y ago
      The final product they produce needs security to protect the end users' medical records, but why on Earth would you need secrecy for what is essentially a public works product?
  - dkhenry 12y ago
    So if your trying to point out that this is more then just a custom wordpress site then your correct. If however you perceive this to be some intrinsically complex task requiring extensive technical expertise you are wrong. A majority of that money is most likely overhead and even then the development is done via integrations with enterprise vendors so another large chunk of change goes to pay the IBM's and Oracle's of the world for their databases and webservers along with their "consultants" to get it all tied together. FWIW the current enrollment is 6mil so if you assume a very poor conversion rate of 1% then that is 600 million page views over the signup period and that comes down to about 38 views per second. At peak load that would only be like 1k concurrent views ( assuming a poisson distribution )
    - threeseed 12y ago
      > If however you perceive this to be some intrinsically complex task requiring extensive technical expertise you are wrong. Have you ever worked on these projects before ? It DOES require technical expertise. Working with legacy and enterprise systems requires knowledge that most developers think they could handle until they are actually in the weeds trying to understand some arcane domain model or file format. Combine that with the integration and scaling challenges and it is quite a technical problem. Is it expensive sure. But to dismiss these projects without any visibility or understanding of the challenge requires some arrogance.
      - dkhenry 12y ago
        I used to be a government contractor so I am well aware of the challenges of integrating the various systems. The challenges are almost never technical I mean consider your argument here you essentially have a system connecting multiple databases that needs to retrieve and reformat the data then store it again. That process is not complex even at the scale they are doing it ( 8 Million registrants, even 350 Million wouldn't be a problem for most database systems ) Yes documentation is always lacking and there is some reverse engineering that needs to be done, and yes there will always be some stupid wrinkle in the process, but this is a well understood problem and claiming its a challenge due to "Legacy Systems" is just a way of justifying the over 100% overhead that the government runs.
- threeseed 12y ago
  It may not be computationally challenging. But if you are integrating with dozens of legacy systems then getting that right is hardly trivial. Look at Netflix and Twitter and the work they did around resilience between their micro services. Getting that to work every time with all failure paths handled correctly is definitely architecturally challenging.
  - wyager 12y ago
    >Getting that to work every time with all failure paths handled correctly is definitely architecturally challenging. You're right. Good thing they completely failed to handle all failure paths correctly. This is extensively documented at http://en.wikipedia.org/wiki/HealthCare.gov#Launch_and_technical_problems http://en.wikipedia.org/wiki/HealthCare.gov#Launch_and_techn...
    - threeseed 12y ago
      I'm not saying that they did a perfect job of it. We all know they didn't. I am just disputing anyone who thinks projects like this aren't technically challenging.
TallGuyShort 12y ago
Can't get past the paywall on my phone, but 350 million people in the country, limited to working age adults, limited to those who didn't use state-level exchanges, limited to those who wanted to use the public option at all, and you're quickly at tens of millions, possibly less, spread over several months. And it sucked at serving that many people.
- hippich 12y ago
  But these 10 millions who now got a chance to have insurance save dollars for the rest 340m. So indirectly it indeed served all 350m (or whatever total number of people living on USA territory)
  - TallGuyShort 12y ago
    That sense of the word "served" has nothing to do with the magnitude of the technical project.
  - jdjdjdidjd 12y ago
    How many dollars were saved exactly? Do you have a source? Anecdotally, I know plenty of people who saw an increase in their premiums.
    - hippich 12y ago
      I don't have numbers, nor anyone will have I bet, since that kind of change requires some time to really see benefits (or negatives) My opinion based on the fact that nothing is free, and people who currently don't have insurance, but still require medical attention, still cost the rest of us money. Also, since people without medical insurance tends to go to emergency room only in critical cases - problems they had could be much cheaper if they had insurance from the beginning. Anecdotally, I know such people :)
    - XorNot 12y ago
      Anecdotally, you met a lot of people who had health insurance they never actually used, and which wouldn't cover them for anything if they needed it. Those are the people who's plans were discontinued.
tdicola 12y ago
It's nothing compared to something like the B2 bomber which cost $40+ billion to develop and each one costs over $700 million. We built 20 of them. (they are still pretty awesome cold war relics though) edit: Or Nimitz class nuclear aircraft carriers for $4.5 billion each, with 10 of them built.
- EpicEng 12y ago
  Comparing software development to hardware (+ software) development is not fair. I'm almost certain there was waste involved in that venture as well, but hardware is expensive.
jdjdjdidjd 12y ago
Yes, this explains why every Silicon Valley web company needs $840,000,000 in funding too before they can serve a small percentage of the US population. Oh wait.
- threeseed 12y ago
  Do you really think this is in any way comparable to a web company ? The project was never about just the website. It is the huge amounts of behind the scenes integration with legacy systems. Some of which may require specialist development experience which never comes cheap.
  - jdjdjdidjd 12y ago
    I agree that this is not a typical web company project, but if you read the report, a lot of that money was for naught. http://blogs.marketwatch.com/health-exchange/2014/07/30/gao-to-point-finger-at-managers-for-obamacare-debacle-report-says/ http://blogs.marketwatch.com/health-exchange/2014/07/30/gao-... "Confusing and constantly changing marching orders from Centers for Medicare and Medicaid Services managers will get the blame for last year’s HealthCare.gov debacle under an independent government review, the Associated Press said Wednesday."
    - threeseed 12y ago
      >a lot of that money was for naught You've provided no evidence of this. Including that article. I am just disputing your assertion that this is comparable to your typical Silicon Valley web company who rarely have to deal with legacy systems. Not whether it was a well run project.
  - karmelapple 12y ago
    Is there a technical breakdown anywhere that talks about the specific technical challenges, such as which specific legacy systems were interfaced with? If there were real technical challenges, there was likely a risk register of some sort tracked by project management for this contractor, which would have listed any major technical issues like that.
nostromo 12y ago
The federal site only had 5.5 million people sign up for healthcare. That's $150 per registered user. http://www.businessinsider.com/how-many-people-signed-up-for-obamacare-2014-5 http://www.businessinsider.com/how-many-people-signed-up-for...
- jjoonathan 12y ago
  So less than 2 weeks of health care coverage per user. While we're on the subject of waste, the USA's ongoing experiment with market-based health care costs ~20 weeks of coverage per user per year compared to its social counterparts. http://healthcarereform.procon.org/view.resource.php?resourceID=005522#III http://healthcarereform.procon.org/view.resource.php?resourc...
  - beat 12y ago
    Market-based isn't the problem. Switzerland, Germany, and Singapore all have mostly-private competition-driven systems (more private than the American model, even), and they are inexpensive and provide high-quality care, along with universal coverage. So private insurers aren't why the American system is so uniquely incompetent and inefficient. So what is the problem? The way I see it, there are two features that really distinguish the American system structurally. First, there's the state-by-state laws and regulations, resulting in effectively 51 different legal regimes. Second, American insurance is provided primarily through employers, rather than purchased individually. In fact, non-employer insurance is prohibitively expensive and often impossible to get. I think the employer role is a much more likely cause of problems than the state laws, which are mostly consistent. Worse, tying insurance to employment traps people in jobs they hate - a distinctly un-American outcome. So anyway, I used to think a more socialist system was the right answer too, until I looked more closely at the working systems in the world. It's not that one particular model works, because there are lots of models in use in other countries and they all work fine. It's that one model doesn't work - what we have in America.
    - waps 12y ago
      I was going to say something about how the US market is a lot bigger than the Swiss market, but I've had to reconsider. Switzerland is a country of 6 million people and is about the size of an "average" US state. Most US states are a little smaller in population (If Switzerland was a US state, it'd be the 13th largest), and a LOT larger in area. So for a "normal" state, Switzerland actually has a bigger healthcare market. California, about 5 times the size of the Swiss healthcare market, still manages to screw things up royally though. People are often blaming the US legal rules. In Switzerland, if you want to win a court case that a MD acted improperly (justifying damages), you have to get a jury of his peers (ie. MDs) to agree on that (which is not impossible, but ...). In the US, you have to get a jury of patients to agree on that. Needless to say, there are many, many more medical cases in the US, with spectacular damage payments (damage payments are also standardized in the EU. If an MD kills you through negligence or outright a mistake (not satisfying the level of a criminal act), his employer (even though they're mostly freelancers, this is an exception in the law) owes your family 300k euro (inflation adjusted since 19xx, so it's a bit more). In the US, it's not fixed, judges decide, but it seems that number runs into the tens of millions of dollars.

Acen 12y ago

Is there a way to bypass the pay wall to read this?

nostromo 12y ago
Click here: https://www.google.com/search?q=Poorly+Managed+HealthCare.gov+Construction+Cost+%24840+Million&tbm=nws https://www.google.com/search?q=Poorly+Managed+HealthCare.go... Then click the story.
- click170 12y ago
  IMO you shouldn't have to do that, so I flagged it.

vnchr 12y ago

Consulting firms like Deloitte were taking payments of +$70 million[1] to do "analysis" that "informed development and decision making." The associated waste of resources is unfathomable in the startup world but seems to be common with federal contracting. 1. Tipsy Deloitte associate chatting at a Chicago bar

doubt_me 12y ago
Why the hell does everything the government do have to go through a contract and outsourced/ bid on? Why don't they contract someone to come up with a better solution?
- adamnemecek 12y ago
  I've had someone explain it to me as follows. If shit hits the fan and you are the one responsible for hiring Joe Schmo, your job might be on the line. If you've hired someone like Deloitte and then do what they tell you and shit still hits the fan, you can always say 'I did what all what I could have done'.
  - rahimnathwani 12y ago
    aka 'Nobody got fired for hiring IBM'.
- dragonwriter 12y ago
  > Why the hell does everything the government do have to go through a contract and outsourced/ bid on? Because: 1) There are lots of people ideologically motivated to minimize the "size of government" as measured by public payroll, and almost no one ideologically motivated in the opposite direction, and so contracting out helps to improve a metric that is important to a large number of stakeholders even if it is substantively meaningless, and 2) There's lots of money in firms that are specialized in taking government contracts, and they both lobby government and engage in public political advocacy effectively. And, no, those two factors are not independent -- #2 contributes directly to #1. (Also, outsourcing blame as adamnemecek points to in a sibling comment, can be a factor where the people who would potentially be at risk of blame actually have discretion of whether to do something internally or contract out, but that's often not the case.)

alien3d 12y ago

Wow.quite cheap...

brandonb 12y ago

I was part of the "tech surge" brought in to help patch up healthcare.gov in December. The GAO report gives a decent idea of what went wrong in 2010-2013 from a high-level contracting and policy perspective, but it doesn't really give much technical detail or describe what was broken about the organization, software architecture, and culture. A couple of my colleagues have given talks on healthcare.gov from the perspective of the engineers who worked around the clock to help get the site running again: http://youtu.be/0albm_hhQzM?t=3m40s http://youtu.be/0albm_hhQzM?t=3m40s http://www.youtube.com/watch?v=GLQyj-kBRdo http://www.youtube.com/watch?v=GLQyj-kBRdo A small team is also re-writing the site, starting with the most troublesome components for the next open enrollment period. Wired covered some of what is going on here: http://www.wired.com/2014/06/healthcare-gov-revamp/ http://www.wired.com/2014/06/healthcare-gov-revamp/ If any of you out there want to help, please email jobs@hcgov.us!

butner 12y ago
Fair perspective, but the government is still trying to solve a lot of the problems with closed-doors and the same contractors. There's a lot of startups that are focused on solving the problems of getting people health coverage. A bigger challenge is ensuring people get the right coverage for their situation and understand the plan and provider network they're buying. Disclaimer: I work at Stride Health, we help individuals buy plans in-context of their doctors, drugs, and health conditions they're treating. We're also hiring, sans the $840MM budget (https://stridehealth.com/jobs https://stridehealth.com/jobs)

Ecio78 12y ago

Each time I read about this I think about our dear Italian tourism portal www.italia.it that costed us more than 58mil€ in 2004-2008 and was a huge failure. According to wikipedia[1] we have then spent additional 10mil for redoing it, plus 20mil for managing it in these years. Interesting for the HN designers is probably the "nice" logo[2] that won a 100.000Euro logo competition in 2006 (it was created by Landor, a US or UK brand design company...) There were jokes about the green "t" that looked like a cucumber (and in Italy the cucumber - "il cetriolo" - has sometimes a double meaning).. [1] http://it.wikipedia.org/wiki/Italia.it http://it.wikipedia.org/wiki/Italia.it [2] http://www.marcosansalone.com/creativeantblog/?p=310 http://www.marcosansalone.com/creativeantblog/?p=310

sytelus 12y ago

This number may look big but remember average per head cost of a software engineer is about 230K/yr. Any software project that takes 3 years and requires ~1000 engineers would approximately cost similar amount. On the side, I'd first hand experience with healthcare.gov to help someone else. I can imagine myriads of integration points between insurance companies, keeping track of their plans, state level requirements, credit checks, other govt agencies like social security, customer service backend etc. From my experience in working with this kind of complexities I think 200-300 engineers may be more than enough for 2 year execution plan. So still it's about 3X-4X waste when government gets involved. PS: Per head cost needs to include base salary + bonuses + stock grants + hiring fees + employee events + office expenses + subsidized cafeteria + health benefits + 401K etc etc.

colmmacc 12y ago

Maybe this is facile, maybe it is illustrative, but I find it interesting to compare HealthCare.gov and Twitter. Twitter has raised over a billion in funding so far, has been going 8 years and has over 3,000 employees. Healthcare.gov is at about 3 years, $840M, and I can't find a number for how many people were involved, but it's likely also in the thousands. Healthcare.gov has delivered life-altering value to about 6 million people and cost about $150 per user (soon amortizable over at least two years). Twitter has about 200 million active users getting everything from minor titillation to revolutionary aid and has cost about $6 per user. Neither of those "prices" seems shocking to me, and I'd pay $6 to use twitter and $150 to use Healthcare.gov. If anything, healthcare.gov seems cheap in that context. In terms of scale of effort; if I took a cursory look at twitter's functionality, and even taking scale into account [1], I'd spitball it as taking maybe 3 or 4 small dev teams for the website and API, and another dev team per app; maybe 100 engineers total. I'm sure they have more than that though; because a cursory spitball guess is an insult and there's likely a lot of hidden complexity and business logic discovered only you if try to do something like twitter very well. I'm also sure there's a huge mass of employees dedicated to wet, human, problems too like sales acquisition and support and what direction the shading should be on the twitter eggs and why the logo should just be a bird. And so it likely is with Healthcare.gov, interfacing with hundreds of different healthcare providers, handling different administrations and so on; probably a real deep mess. I'm impressed they got it working at all; especially as when they set out it was never intended to be used much (as I understand it the idea was that the state exchanges would handle the larger burden). The frontend seems to have been shoddy, but how much of the expense was that? [1] I have built and operate services at the same scale as twitter.

waps 12y ago
This does not seem to be a fair assessment. The "life-altering value" is not provided by the cost of the IT project, but by the total cost of healthcare in the US. That's 3.8 trillion dollars, or 11900 USD per person per year (first figure divided by 300 million). Wikipedia lists the per capita expenditure as 8500 USD per person per year for 2011, not sure why there is such a large discrepancy. Total amount of people on healthcare in the US is listed as 11.7 million jobs.

known 12y ago

Mediawiki is better for storing/retrieving UNSTRUCTURED data

Daishiman 12y ago

It's amazing the amount of downvotes on posts that comment just how tremendously difficult it is to plan, design, code, test, and support a website that has to serve literally hundreds of use cases and is meant to last multiple decades and integrate with thousands of other government systems that have been standing since the 60s and all have their own idiosyncracies in protocols, documentations, and bureaucracy. It looks like most people in HN have never actually worked on large systems and have no idea just how costs increase exponentially, how internal politics and policies affect those projects, and how infernally hard it is to think in terms of systems that have to be used by more than a handful of use cases. There's not doubt that government projects have a huge amount of internal overhead, but if you compare them to the stuff that goes on in multinationals that have working systems in the 60s, it's really nothing out of the ordinary. I know a guy who worked in a project related to one of the largest insurance companies in the US. It's a 60 year-old system running on IBM System Zs, AS/400s, Unix and Windows systems, with varying degrees of hardware and software obsolescence, running custom COBOL builds and compilers, integrating with thousands of external APIs, on half a dozen data centers with completely different specs. There was a literal library of documentation spanning millions of pages of requirements to describe how policies flowed through the systems, how batch processes should run, and what tests should be used to make sure a lady who took out an insurance policy on her TV in the 70s can still cash out on it now and withing 20 years in the future. All this does not even include the network topologies spanning TCP/IP, custom frame relay protocols, proprietary NICs, SANs for storage and backup, tape libraries, electrical layouts for data centers, etc. There's more than a few reasons why large software projects fail; I have not even begun to include the intricacies of the aforementioned industry and have not even counted the regular issues of any project.

lifeformed 12y ago
Sure, it's quite an engineering challenge, but... $840M? That's a staggering figure.
- Daishiman 12y ago
  Have you seen the costs of similarly large software projects? It's actually a figure well within those margins. For the same insurance company I quoted, the people in question estimated that modernizing their system would cost $500 million USD in 1997 dollars and take 6 years without considering continuous upgrades of code and infrastructure. And it's likely that this company's infrastructure was probably much cleaner and uniform that goverment agencies'.

rakoo 12y ago

So, where's the github repo ?