5 ms·

Is it me or is that not that much money for a system that serves a 350 million-person nation?

wyager 12y ago
It certainly doesn't serve all 350 million people. And we don't really know what kind of peak loads it can handle, so that might not be very impressive. Also, considering what it does (Which it might not even do well; time will tell.), it's not exactly computationally challenging.
- tzs 12y ago
  Doesn't it have to interface to a whole bunch of other government and commercial sites, which have their own APIs and data formats? I believe I read somewhere (probably earlier HN discussion months ago) that dealing with all that made this a ridiculously more complex project than it might first appear based on what you actually see on the site.
  - wyager 12y ago
    That's probably true, and that's where I'd guess most of the cost came from. I'm just saying that if they only had to serve 350,000 people it probably would have been almost the same cost.
  - kkhire 12y ago
    Yea, I'm pretty positive you're both correct--it was a government related project that they needed utmost of security and secrecy while rapidly building it. All of which are very challenging obstacles for developers
    - wyager 12y ago
      >they needed utmost of security I can absolutely guarantee you that they do not have the utmost of security. For a project to be really secure, you need extremely talented and meticulous developers. Judging by the clusterfuck that was Healthcare.gov's original launch, that does not describe the people involved. For example, this sentence (lifted from Wikipedia) raises a million red flags in my mind re. data consistency checking and sanitation: "Even for those that did manage to enroll, insurance providers later reported some instances of applications submitted through the site with required information missing.". No project like that can feasibly have the "utmost of security".
      - kkhire 12y ago
        haha needed and what they got are entirely two different things. It's unfortunate, but the problem was that because it was a government website, the developers were strictly not allowed to get help, i.e. have someone review it/ask a friend for help/post on stackoverflow/etc. this probably was the source of the initial failure.
    - spacehome 12y ago
      The final product they produce needs security to protect the end users' medical records, but why on Earth would you need secrecy for what is essentially a public works product?
  - dkhenry 12y ago
    So if your trying to point out that this is more then just a custom wordpress site then your correct. If however you perceive this to be some intrinsically complex task requiring extensive technical expertise you are wrong. A majority of that money is most likely overhead and even then the development is done via integrations with enterprise vendors so another large chunk of change goes to pay the IBM's and Oracle's of the world for their databases and webservers along with their "consultants" to get it all tied together. FWIW the current enrollment is 6mil so if you assume a very poor conversion rate of 1% then that is 600 million page views over the signup period and that comes down to about 38 views per second. At peak load that would only be like 1k concurrent views ( assuming a poisson distribution )
    - threeseed 12y ago
      > If however you perceive this to be some intrinsically complex task requiring extensive technical expertise you are wrong. Have you ever worked on these projects before ? It DOES require technical expertise. Working with legacy and enterprise systems requires knowledge that most developers think they could handle until they are actually in the weeds trying to understand some arcane domain model or file format. Combine that with the integration and scaling challenges and it is quite a technical problem. Is it expensive sure. But to dismiss these projects without any visibility or understanding of the challenge requires some arrogance.
      - dkhenry 12y ago
        I used to be a government contractor so I am well aware of the challenges of integrating the various systems. The challenges are almost never technical I mean consider your argument here you essentially have a system connecting multiple databases that needs to retrieve and reformat the data then store it again. That process is not complex even at the scale they are doing it ( 8 Million registrants, even 350 Million wouldn't be a problem for most database systems ) Yes documentation is always lacking and there is some reverse engineering that needs to be done, and yes there will always be some stupid wrinkle in the process, but this is a well understood problem and claiming its a challenge due to "Legacy Systems" is just a way of justifying the over 100% overhead that the government runs.
- threeseed 12y ago
  It may not be computationally challenging. But if you are integrating with dozens of legacy systems then getting that right is hardly trivial. Look at Netflix and Twitter and the work they did around resilience between their micro services. Getting that to work every time with all failure paths handled correctly is definitely architecturally challenging.
  - wyager 12y ago
    >Getting that to work every time with all failure paths handled correctly is definitely architecturally challenging. You're right. Good thing they completely failed to handle all failure paths correctly. This is extensively documented at http://en.wikipedia.org/wiki/HealthCare.gov#Launch_and_technical_problems http://en.wikipedia.org/wiki/HealthCare.gov#Launch_and_techn...
    - threeseed 12y ago
      I'm not saying that they did a perfect job of it. We all know they didn't. I am just disputing anyone who thinks projects like this aren't technically challenging.
TallGuyShort 12y ago
Can't get past the paywall on my phone, but 350 million people in the country, limited to working age adults, limited to those who didn't use state-level exchanges, limited to those who wanted to use the public option at all, and you're quickly at tens of millions, possibly less, spread over several months. And it sucked at serving that many people.
- hippich 12y ago
  But these 10 millions who now got a chance to have insurance save dollars for the rest 340m. So indirectly it indeed served all 350m (or whatever total number of people living on USA territory)
  - TallGuyShort 12y ago
    That sense of the word "served" has nothing to do with the magnitude of the technical project.
  - jdjdjdidjd 12y ago
    How many dollars were saved exactly? Do you have a source? Anecdotally, I know plenty of people who saw an increase in their premiums.
    - hippich 12y ago
      I don't have numbers, nor anyone will have I bet, since that kind of change requires some time to really see benefits (or negatives) My opinion based on the fact that nothing is free, and people who currently don't have insurance, but still require medical attention, still cost the rest of us money. Also, since people without medical insurance tends to go to emergency room only in critical cases - problems they had could be much cheaper if they had insurance from the beginning. Anecdotally, I know such people :)
    - XorNot 12y ago
      Anecdotally, you met a lot of people who had health insurance they never actually used, and which wouldn't cover them for anything if they needed it. Those are the people who's plans were discontinued.
tdicola 12y ago
It's nothing compared to something like the B2 bomber which cost $40+ billion to develop and each one costs over $700 million. We built 20 of them. (they are still pretty awesome cold war relics though) edit: Or Nimitz class nuclear aircraft carriers for $4.5 billion each, with 10 of them built.
- EpicEng 12y ago
  Comparing software development to hardware (+ software) development is not fair. I'm almost certain there was waste involved in that venture as well, but hardware is expensive.
jdjdjdidjd 12y ago
Yes, this explains why every Silicon Valley web company needs $840,000,000 in funding too before they can serve a small percentage of the US population. Oh wait.
- threeseed 12y ago
  Do you really think this is in any way comparable to a web company ? The project was never about just the website. It is the huge amounts of behind the scenes integration with legacy systems. Some of which may require specialist development experience which never comes cheap.
  - jdjdjdidjd 12y ago
    I agree that this is not a typical web company project, but if you read the report, a lot of that money was for naught. http://blogs.marketwatch.com/health-exchange/2014/07/30/gao-to-point-finger-at-managers-for-obamacare-debacle-report-says/ http://blogs.marketwatch.com/health-exchange/2014/07/30/gao-... "Confusing and constantly changing marching orders from Centers for Medicare and Medicaid Services managers will get the blame for last year’s HealthCare.gov debacle under an independent government review, the Associated Press said Wednesday."
    - threeseed 12y ago
      >a lot of that money was for naught You've provided no evidence of this. Including that article. I am just disputing your assertion that this is comparable to your typical Silicon Valley web company who rarely have to deal with legacy systems. Not whether it was a well run project.
  - karmelapple 12y ago
    Is there a technical breakdown anywhere that talks about the specific technical challenges, such as which specific legacy systems were interfaced with? If there were real technical challenges, there was likely a risk register of some sort tracked by project management for this contractor, which would have listed any major technical issues like that.
nostromo 12y ago
The federal site only had 5.5 million people sign up for healthcare. That's $150 per registered user. http://www.businessinsider.com/how-many-people-signed-up-for-obamacare-2014-5 http://www.businessinsider.com/how-many-people-signed-up-for...
- jjoonathan 12y ago
  So less than 2 weeks of health care coverage per user. While we're on the subject of waste, the USA's ongoing experiment with market-based health care costs ~20 weeks of coverage per user per year compared to its social counterparts. http://healthcarereform.procon.org/view.resource.php?resourceID=005522#III http://healthcarereform.procon.org/view.resource.php?resourc...
  - beat 12y ago
    Market-based isn't the problem. Switzerland, Germany, and Singapore all have mostly-private competition-driven systems (more private than the American model, even), and they are inexpensive and provide high-quality care, along with universal coverage. So private insurers aren't why the American system is so uniquely incompetent and inefficient. So what is the problem? The way I see it, there are two features that really distinguish the American system structurally. First, there's the state-by-state laws and regulations, resulting in effectively 51 different legal regimes. Second, American insurance is provided primarily through employers, rather than purchased individually. In fact, non-employer insurance is prohibitively expensive and often impossible to get. I think the employer role is a much more likely cause of problems than the state laws, which are mostly consistent. Worse, tying insurance to employment traps people in jobs they hate - a distinctly un-American outcome. So anyway, I used to think a more socialist system was the right answer too, until I looked more closely at the working systems in the world. It's not that one particular model works, because there are lots of models in use in other countries and they all work fine. It's that one model doesn't work - what we have in America.
    - waps 12y ago
      I was going to say something about how the US market is a lot bigger than the Swiss market, but I've had to reconsider. Switzerland is a country of 6 million people and is about the size of an "average" US state. Most US states are a little smaller in population (If Switzerland was a US state, it'd be the 13th largest), and a LOT larger in area. So for a "normal" state, Switzerland actually has a bigger healthcare market. California, about 5 times the size of the Swiss healthcare market, still manages to screw things up royally though. People are often blaming the US legal rules. In Switzerland, if you want to win a court case that a MD acted improperly (justifying damages), you have to get a jury of his peers (ie. MDs) to agree on that (which is not impossible, but ...). In the US, you have to get a jury of patients to agree on that. Needless to say, there are many, many more medical cases in the US, with spectacular damage payments (damage payments are also standardized in the EU. If an MD kills you through negligence or outright a mistake (not satisfying the level of a criminal act), his employer (even though they're mostly freelancers, this is an exception in the law) owes your family 300k euro (inflation adjusted since 19xx, so it's a bit more). In the US, it's not fixed, judges decide, but it seems that number runs into the tens of millions of dollars.

It certainly doesn't serve all 350 million people. And we don't really know what kind of peak loads it can handle, so that might not be very impressive. Also, considering what it does (Which it might not even do well; time will tell.), it's not exactly computationally challenging.

tzs 12y ago
Doesn't it have to interface to a whole bunch of other government and commercial sites, which have their own APIs and data formats? I believe I read somewhere (probably earlier HN discussion months ago) that dealing with all that made this a ridiculously more complex project than it might first appear based on what you actually see on the site.
- wyager 12y ago
  That's probably true, and that's where I'd guess most of the cost came from. I'm just saying that if they only had to serve 350,000 people it probably would have been almost the same cost.
- kkhire 12y ago
  Yea, I'm pretty positive you're both correct--it was a government related project that they needed utmost of security and secrecy while rapidly building it. All of which are very challenging obstacles for developers
  - wyager 12y ago
    >they needed utmost of security I can absolutely guarantee you that they do not have the utmost of security. For a project to be really secure, you need extremely talented and meticulous developers. Judging by the clusterfuck that was Healthcare.gov's original launch, that does not describe the people involved. For example, this sentence (lifted from Wikipedia) raises a million red flags in my mind re. data consistency checking and sanitation: "Even for those that did manage to enroll, insurance providers later reported some instances of applications submitted through the site with required information missing.". No project like that can feasibly have the "utmost of security".
    - kkhire 12y ago
      haha needed and what they got are entirely two different things. It's unfortunate, but the problem was that because it was a government website, the developers were strictly not allowed to get help, i.e. have someone review it/ask a friend for help/post on stackoverflow/etc. this probably was the source of the initial failure.
  - spacehome 12y ago
    The final product they produce needs security to protect the end users' medical records, but why on Earth would you need secrecy for what is essentially a public works product?
- dkhenry 12y ago
  So if your trying to point out that this is more then just a custom wordpress site then your correct. If however you perceive this to be some intrinsically complex task requiring extensive technical expertise you are wrong. A majority of that money is most likely overhead and even then the development is done via integrations with enterprise vendors so another large chunk of change goes to pay the IBM's and Oracle's of the world for their databases and webservers along with their "consultants" to get it all tied together. FWIW the current enrollment is 6mil so if you assume a very poor conversion rate of 1% then that is 600 million page views over the signup period and that comes down to about 38 views per second. At peak load that would only be like 1k concurrent views ( assuming a poisson distribution )
  - threeseed 12y ago
    > If however you perceive this to be some intrinsically complex task requiring extensive technical expertise you are wrong. Have you ever worked on these projects before ? It DOES require technical expertise. Working with legacy and enterprise systems requires knowledge that most developers think they could handle until they are actually in the weeds trying to understand some arcane domain model or file format. Combine that with the integration and scaling challenges and it is quite a technical problem. Is it expensive sure. But to dismiss these projects without any visibility or understanding of the challenge requires some arrogance.
    - dkhenry 12y ago
      I used to be a government contractor so I am well aware of the challenges of integrating the various systems. The challenges are almost never technical I mean consider your argument here you essentially have a system connecting multiple databases that needs to retrieve and reformat the data then store it again. That process is not complex even at the scale they are doing it ( 8 Million registrants, even 350 Million wouldn't be a problem for most database systems ) Yes documentation is always lacking and there is some reverse engineering that needs to be done, and yes there will always be some stupid wrinkle in the process, but this is a well understood problem and claiming its a challenge due to "Legacy Systems" is just a way of justifying the over 100% overhead that the government runs.
threeseed 12y ago
It may not be computationally challenging. But if you are integrating with dozens of legacy systems then getting that right is hardly trivial. Look at Netflix and Twitter and the work they did around resilience between their micro services. Getting that to work every time with all failure paths handled correctly is definitely architecturally challenging.
- wyager 12y ago
  >Getting that to work every time with all failure paths handled correctly is definitely architecturally challenging. You're right. Good thing they completely failed to handle all failure paths correctly. This is extensively documented at http://en.wikipedia.org/wiki/HealthCare.gov#Launch_and_technical_problems http://en.wikipedia.org/wiki/HealthCare.gov#Launch_and_techn...
  - threeseed 12y ago
    I'm not saying that they did a perfect job of it. We all know they didn't. I am just disputing anyone who thinks projects like this aren't technically challenging.

TallGuyShort 12y ago

Can't get past the paywall on my phone, but 350 million people in the country, limited to working age adults, limited to those who didn't use state-level exchanges, limited to those who wanted to use the public option at all, and you're quickly at tens of millions, possibly less, spread over several months. And it sucked at serving that many people.

hippich 12y ago
But these 10 millions who now got a chance to have insurance save dollars for the rest 340m. So indirectly it indeed served all 350m (or whatever total number of people living on USA territory)
- TallGuyShort 12y ago
  That sense of the word "served" has nothing to do with the magnitude of the technical project.
- jdjdjdidjd 12y ago
  How many dollars were saved exactly? Do you have a source? Anecdotally, I know plenty of people who saw an increase in their premiums.
  - hippich 12y ago
    I don't have numbers, nor anyone will have I bet, since that kind of change requires some time to really see benefits (or negatives) My opinion based on the fact that nothing is free, and people who currently don't have insurance, but still require medical attention, still cost the rest of us money. Also, since people without medical insurance tends to go to emergency room only in critical cases - problems they had could be much cheaper if they had insurance from the beginning. Anecdotally, I know such people :)
  - XorNot 12y ago
    Anecdotally, you met a lot of people who had health insurance they never actually used, and which wouldn't cover them for anything if they needed it. Those are the people who's plans were discontinued.

tdicola 12y ago

It's nothing compared to something like the B2 bomber which cost $40+ billion to develop and each one costs over $700 million. We built 20 of them. (they are still pretty awesome cold war relics though) edit: Or Nimitz class nuclear aircraft carriers for $4.5 billion each, with 10 of them built.

EpicEng 12y ago
Comparing software development to hardware (+ software) development is not fair. I'm almost certain there was waste involved in that venture as well, but hardware is expensive.

jdjdjdidjd 12y ago

Yes, this explains why every Silicon Valley web company needs $840,000,000 in funding too before they can serve a small percentage of the US population. Oh wait.

threeseed 12y ago
Do you really think this is in any way comparable to a web company ? The project was never about just the website. It is the huge amounts of behind the scenes integration with legacy systems. Some of which may require specialist development experience which never comes cheap.
- jdjdjdidjd 12y ago
  I agree that this is not a typical web company project, but if you read the report, a lot of that money was for naught. http://blogs.marketwatch.com/health-exchange/2014/07/30/gao-to-point-finger-at-managers-for-obamacare-debacle-report-says/ http://blogs.marketwatch.com/health-exchange/2014/07/30/gao-... "Confusing and constantly changing marching orders from Centers for Medicare and Medicaid Services managers will get the blame for last year’s HealthCare.gov debacle under an independent government review, the Associated Press said Wednesday."
  - threeseed 12y ago
    >a lot of that money was for naught You've provided no evidence of this. Including that article. I am just disputing your assertion that this is comparable to your typical Silicon Valley web company who rarely have to deal with legacy systems. Not whether it was a well run project.
- karmelapple 12y ago
  Is there a technical breakdown anywhere that talks about the specific technical challenges, such as which specific legacy systems were interfaced with? If there were real technical challenges, there was likely a risk register of some sort tracked by project management for this contractor, which would have listed any major technical issues like that.

nostromo 12y ago

The federal site only had 5.5 million people sign up for healthcare. That's $150 per registered user. http://www.businessinsider.com/how-many-people-signed-up-for-obamacare-2014-5 http://www.businessinsider.com/how-many-people-signed-up-for...

jjoonathan 12y ago
So less than 2 weeks of health care coverage per user. While we're on the subject of waste, the USA's ongoing experiment with market-based health care costs ~20 weeks of coverage per user per year compared to its social counterparts. http://healthcarereform.procon.org/view.resource.php?resourceID=005522#III http://healthcarereform.procon.org/view.resource.php?resourc...
- beat 12y ago
  Market-based isn't the problem. Switzerland, Germany, and Singapore all have mostly-private competition-driven systems (more private than the American model, even), and they are inexpensive and provide high-quality care, along with universal coverage. So private insurers aren't why the American system is so uniquely incompetent and inefficient. So what is the problem? The way I see it, there are two features that really distinguish the American system structurally. First, there's the state-by-state laws and regulations, resulting in effectively 51 different legal regimes. Second, American insurance is provided primarily through employers, rather than purchased individually. In fact, non-employer insurance is prohibitively expensive and often impossible to get. I think the employer role is a much more likely cause of problems than the state laws, which are mostly consistent. Worse, tying insurance to employment traps people in jobs they hate - a distinctly un-American outcome. So anyway, I used to think a more socialist system was the right answer too, until I looked more closely at the working systems in the world. It's not that one particular model works, because there are lots of models in use in other countries and they all work fine. It's that one model doesn't work - what we have in America.
  - waps 12y ago
    I was going to say something about how the US market is a lot bigger than the Swiss market, but I've had to reconsider. Switzerland is a country of 6 million people and is about the size of an "average" US state. Most US states are a little smaller in population (If Switzerland was a US state, it'd be the 13th largest), and a LOT larger in area. So for a "normal" state, Switzerland actually has a bigger healthcare market. California, about 5 times the size of the Swiss healthcare market, still manages to screw things up royally though. People are often blaming the US legal rules. In Switzerland, if you want to win a court case that a MD acted improperly (justifying damages), you have to get a jury of his peers (ie. MDs) to agree on that (which is not impossible, but ...). In the US, you have to get a jury of patients to agree on that. Needless to say, there are many, many more medical cases in the US, with spectacular damage payments (damage payments are also standardized in the EU. If an MD kills you through negligence or outright a mistake (not satisfying the level of a criminal act), his employer (even though they're mostly freelancers, this is an exception in the law) owes your family 300k euro (inflation adjusted since 19xx, so it's a bit more). In the US, it's not fixed, judges decide, but it seems that number runs into the tens of millions of dollars.