18 ms·

Inside the race to rescue a health site, and Obama

53 points by kanamekun 13y ago

sylvinus 13y ago
Amazingly, the website appears to be down again! https://www.healthcare.gov/ https://www.healthcare.gov/ Looks like they had planned that downtime, but I wonder how messed up this whole project must be that they can't even manage to have a static maintenance page up!
- quaunaut 13y ago
  It's up.
- dkhenry 13y ago
  It up for me. I was just able to register an account. After looking at all the steps to do that 1. This is already too complicated. ( why a username when you know I am going to need to use my SSN ) 2. This is not a complicated web app. I am now fully convinced that this could have been done with better results by anyone other then the government.
  - ceejayoz 13y ago
    > why a username when you know I am going to need to use my SSN The fewer times you have to enter an SSN, the better. Especially as username-type data is usually shown in text on the screen, might need to be read out over the phone, is vulnerable to phishing, etc.
  - markdown 13y ago
    > I am now fully convinced that this could have been done with better results by anyone other then the government. But it wasn't built by the government. Do you think Obama asked for a username field? It was built by a private contractor ie. your usual capitalist, profit-oriented enterprise IT company.
    - growupkids 13y ago
      You've never built a system for the government I take it? You don't build these things in a vacuum, they approve and co-design the system, drive the technology decisions, require you to use "approved" technologies, ban the use of icky technologies because they aren't on the interoperability list (which locks you into a few vendors, the usual suspects, Microsoft, oracle, etc.), slap on all kinds of additional requirements, force you to use hardware from a cut rate vendor so they can make their 8A quota, and so on. It's nothing like a private company building a system. It's more like being a peasant to a lord. Sure you can do what you want, as long it's exactly what they want to you, when, where and how they want you to do it.
    - nickff 13y ago
      Using this logic would allow the legislators and bureaucrats to escape all culpability by contracting out implementation of all policy. Unrealistic project goals, incompetent oversight, a complete lack of testing, and general unwillingness to adjust to failures are all factors which contribute to project failure.
- oulu2006 13y ago
  Looks like it's down again.
knowitall 13y ago
I'm not from the US. I wonder, isn't this overly dramatic? So it takes a while to fix the web site. People have presumably been waiting for health insurance for decades, what is one more month? Also, really, people get worked up about government processes that are difficult to use?
- ceejayoz 13y ago
  > People have presumably been waiting for health insurance for decades, what is one more month? If you're going to have to go outside the exchange for coverage, the open enrollment period is probably closing very soon. My work ones have traditionally needed a decision made by mid November. By now, I'd have had to tell my employer yes/no on his healthcare - regardless of whether I'd been able to log in to shop around.
  - SilasX 13y ago
    Why did they even set up the exchanges that way? "Well, employers traditionally have retarded deadlines and signup windows, so when we fix healthcare, let's make sure to carry that part over!"
    - nickff 13y ago
      These exchange requirements are set up to avoid "free-riders", a problem caused by the mandatory disregard of pre-existing conditions. The logic is quite interesting, as the regulations create problems which must be solved by further intervention; this causes ever-increasing levels of interference in the market.
      - SilasX 13y ago
        No, that's unrelated and handled by the mandate. Note that the exchanges still have to take newcomers in the future; making people do it during arbitrary windows does nothing about the free rider problem.
        13y ago
dpapathanasiou 13y ago
Does anyone know what the author meant by this? "Software that assigned identities to enrollees and ensured that they saw only their own personal data, known internally as the EIdM, was being quickly overwhelmed. Customers could not log in to create accounts." "Experts disagree on what went wrong. But several said that errors in the software code written to stitch the Oracle product into the online system and improperly configured hardware trapped users in endless technological loops. It would take eight days to resolve just that one bottleneck."
- adventured 13y ago
  Consultant cover-your-ass speak for we set everything up really poorly and had to do it all over again.
  - growupkids 13y ago
    Or possibly it's they-paid-us-for-x-and-told-their-bosses-to-expect-y situation.
    - nickff 13y ago
      All branches of government contract out a wide variety of services, the only things they all have in common are cost overruns, delays, and under-performance. It is astonishing that all these different contractors are always to blame, and the government has done nothing wrong. Fool me once, blame the contractor; fool me twice, blame the management.
      - hga 13y ago
        And in this case, the government's Centers for Medicare and Medicaid Services (CMS, a part of HHS) took upon itself the role of integrator, including integration testing. For which they manifestly didn't have what it took, right down to not assigning any one person the full time responsibility for getting it done. Towards the end of October they got "fired" from this role and replaced with QSSI ... who are responsible for the "data hub" and this Oracle EIDM system (which is part of the data hub? Development has been so politicized that they refused to publish architectural diagrams, not even supplying them to the state exchanges).
eliteraspberrie 13y ago
As a Canadian I wonder, with all the American technology giants, why was this project outsourced to Canada? If the idea was that outsourcing would lower cost, why were the contracts no-bid?
- bane 13y ago
  CGI Federal is a big-ish Fed contractor in the U.S. and, though a subsidiary of a Canadian firm, for various legal reasons is owned and operated as an American firm. It performs quite a bit of tech work for the U.S. government, work won in generally open competition. Because the U.S. Fed space is such a big market, it's not unusual to have locally owned subsidiaries in the mix e.g. BAE, Quinetiq, etc. all have similar groups. My understanding is that CGI was already handling various health care related work and was selected to do the work without a proper competitive bidding process. But having seen these processes, they definitely wouldn't have resolved the kinds of widescale technical issues.
  - growupkids 13y ago
    And the federal government is a remarkably ignorant customer. They don't know what they are asking for, meddle in the details adding in things they think they need, and don't really know what to pay but often think lower=better. Top that off with insane complex procurement rules that require the work be divided up to meet social goals: x% to small biz, y% to 8a, z% to hub zone and so on and you have a guaranteed mess. Even if there was a highly qualified company to do the work, the FAR requires it be divided up. Top that off with a silly idea that a company needs to have past performance doing something that's never been done before, and well you end up with selection committees picking the "safe" (I.e. Big firm) choice. And you're guaranteed to get a firm that doesn't know how to innovate, they just imitate and throw money, hardware and sub contractors at the problem.
  - hga 13y ago
    Addressing you and growupkids, CGI Federal is responsible for the Medicare.gov site, which I can attest to being a bit clunky but otherwise working well for claims and Part D selection and application (the prescription drug benefit program). They were on a list of already vetted vendors, plus if "a proper competitive bidding process" had been done it could easily be hung up in the courts by losers lawsuits. As you note, and as I've been noting, given the structure and processes of the project, especially it being run by political and bureaucratic types from the White House on down to CMS (the integrator prior to getting replaced in late October), there's no way they and the other contractors could have possibly succeeded, especially with change orders continuing though the week before launch. Heck, the integration tests CMS ran in the last week or two before the launch told them it couldn't possibly work, but it was launched anyway....
rathbun 13y ago
Data In. Data Out. I have seen very little of exactly which databases are being integrated. I saw a reference a few days ago that there were many arguments about using SSN in the system. Does that mean that some of systems being pinged for data don't use it? Very hard with SSN, I can't imagine trying to wing a match with similar name, similar address, similar birthday.
United857 13y ago
Anyone know what tech stack they're using? Linux or some other *nix or Microsoft?
- deleted 13y ago
  [deleted]
- genericresponse 13y ago
  Most likely multiple vendors. It probably depends on the component and contractor.
- toomuchtodo 13y ago
  http://medcitynews.com/wp-content/uploads/healthcaredotgovinfographiceduardogarcia-588x968.jpg http://medcitynews.com/wp-content/uploads/healthcaredotgovin... Mirror: http://web.archive.org/web/20131201071252/http://medcitynews.com/wp-content/uploads/healthcaredotgovinfographiceduardogarcia-588x968.jpg http://web.archive.org/web/20131201071252/http://medcitynews...
rdhyee 13y ago
The folks at MarkLogic must be unhappy with being singled out in the following paragraph: Some of the companies building the system opposed an early decision by the Medicare agency to use database software from a company called MarkLogic, which handles data differently from systems by companies like IBM and Oracle. Some suggest that its unfamiliar nature slowed their work. By mid-November, more than six weeks after the rollout, the MarkLogic database -- essentially the website's virtual filing cabinet and index -- continued to perform below expectations, according to one person who works in the command center. The company's response is recorded as: In interviews, MarkLogic’s executives faulted inadequate computing power and instability at the site’s data center, as well as the failure to properly integrate their product, problems repeatedly cited by other website vendors. Any insight on what is meant by how MarkLogic "handles data differently"?
- arosenbaum 13y ago
  Not really a slam at all. Customers choose MarkLogic to replace Oracle and DB2 specifically because it "handles data differently". "Schemaless" aka "Schema on read" is the biggest difference between Only "SQL" and "Not Only SQL" systems. Scale-out on commodity hw vs. scale-up on shared storage is another. Customers often choose MarkLogic instead of other NoSQL systems because of it's ACID capabilities, ability to run multi-statement transactions between multiple databases (including Oracle and MarkLogic in the same transaction) and a multitude of enterprise capabilities from government grade security to point-in-time restore. Read more: http://developer.marklogic.com/pubs/architecture/inside-marklogic-server-r7.pdf http://developer.marklogic.com/pubs/architecture/inside-mark...
  - realmandan 13y ago
    Totally, not a slam at all. As long as you don't think a company name and "perform below expectations" sitting next to each other in a NYT long-form piece is a BAD thing...
    - arosenbaum 13y ago
      I'll repeat what our CEO said in the NYTimes article "He said MarkLogic is performing up to standard, but “the network and the storage systems are not properly sized and not properly run.” Put any database on top of the resources that were provided and I think performance "below expectations" would have been likely.

53 points by kanamekun 13y ago

Amazingly, the website appears to be down again! https://www.healthcare.gov/ https://www.healthcare.gov/ Looks like they had planned that downtime, but I wonder how messed up this whole project must be that they can't even manage to have a static maintenance page up!

quaunaut 13y ago
It's up.
dkhenry 13y ago
It up for me. I was just able to register an account. After looking at all the steps to do that 1. This is already too complicated. ( why a username when you know I am going to need to use my SSN ) 2. This is not a complicated web app. I am now fully convinced that this could have been done with better results by anyone other then the government.
- ceejayoz 13y ago
  > why a username when you know I am going to need to use my SSN The fewer times you have to enter an SSN, the better. Especially as username-type data is usually shown in text on the screen, might need to be read out over the phone, is vulnerable to phishing, etc.
- markdown 13y ago
  > I am now fully convinced that this could have been done with better results by anyone other then the government. But it wasn't built by the government. Do you think Obama asked for a username field? It was built by a private contractor ie. your usual capitalist, profit-oriented enterprise IT company.
  - growupkids 13y ago
    You've never built a system for the government I take it? You don't build these things in a vacuum, they approve and co-design the system, drive the technology decisions, require you to use "approved" technologies, ban the use of icky technologies because they aren't on the interoperability list (which locks you into a few vendors, the usual suspects, Microsoft, oracle, etc.), slap on all kinds of additional requirements, force you to use hardware from a cut rate vendor so they can make their 8A quota, and so on. It's nothing like a private company building a system. It's more like being a peasant to a lord. Sure you can do what you want, as long it's exactly what they want to you, when, where and how they want you to do it.
  - nickff 13y ago
    Using this logic would allow the legislators and bureaucrats to escape all culpability by contracting out implementation of all policy. Unrealistic project goals, incompetent oversight, a complete lack of testing, and general unwillingness to adjust to failures are all factors which contribute to project failure.
oulu2006 13y ago
Looks like it's down again.

knowitall 13y ago

I'm not from the US. I wonder, isn't this overly dramatic? So it takes a while to fix the web site. People have presumably been waiting for health insurance for decades, what is one more month? Also, really, people get worked up about government processes that are difficult to use?

ceejayoz 13y ago
> People have presumably been waiting for health insurance for decades, what is one more month? If you're going to have to go outside the exchange for coverage, the open enrollment period is probably closing very soon. My work ones have traditionally needed a decision made by mid November. By now, I'd have had to tell my employer yes/no on his healthcare - regardless of whether I'd been able to log in to shop around.
- SilasX 13y ago
  Why did they even set up the exchanges that way? "Well, employers traditionally have retarded deadlines and signup windows, so when we fix healthcare, let's make sure to carry that part over!"
  - nickff 13y ago
    These exchange requirements are set up to avoid "free-riders", a problem caused by the mandatory disregard of pre-existing conditions. The logic is quite interesting, as the regulations create problems which must be solved by further intervention; this causes ever-increasing levels of interference in the market.
    - SilasX 13y ago
      No, that's unrelated and handled by the mandate. Note that the exchanges still have to take newcomers in the future; making people do it during arbitrary windows does nothing about the free rider problem.
      - 13y ago

dpapathanasiou 13y ago

Does anyone know what the author meant by this? "Software that assigned identities to enrollees and ensured that they saw only their own personal data, known internally as the EIdM, was being quickly overwhelmed. Customers could not log in to create accounts." "Experts disagree on what went wrong. But several said that errors in the software code written to stitch the Oracle product into the online system and improperly configured hardware trapped users in endless technological loops. It would take eight days to resolve just that one bottleneck."

adventured 13y ago
Consultant cover-your-ass speak for we set everything up really poorly and had to do it all over again.
- growupkids 13y ago
  Or possibly it's they-paid-us-for-x-and-told-their-bosses-to-expect-y situation.
  - nickff 13y ago
    All branches of government contract out a wide variety of services, the only things they all have in common are cost overruns, delays, and under-performance. It is astonishing that all these different contractors are always to blame, and the government has done nothing wrong. Fool me once, blame the contractor; fool me twice, blame the management.
    - hga 13y ago
      And in this case, the government's Centers for Medicare and Medicaid Services (CMS, a part of HHS) took upon itself the role of integrator, including integration testing. For which they manifestly didn't have what it took, right down to not assigning any one person the full time responsibility for getting it done. Towards the end of October they got "fired" from this role and replaced with QSSI ... who are responsible for the "data hub" and this Oracle EIDM system (which is part of the data hub? Development has been so politicized that they refused to publish architectural diagrams, not even supplying them to the state exchanges).

eliteraspberrie 13y ago

As a Canadian I wonder, with all the American technology giants, why was this project outsourced to Canada? If the idea was that outsourcing would lower cost, why were the contracts no-bid?

bane 13y ago
CGI Federal is a big-ish Fed contractor in the U.S. and, though a subsidiary of a Canadian firm, for various legal reasons is owned and operated as an American firm. It performs quite a bit of tech work for the U.S. government, work won in generally open competition. Because the U.S. Fed space is such a big market, it's not unusual to have locally owned subsidiaries in the mix e.g. BAE, Quinetiq, etc. all have similar groups. My understanding is that CGI was already handling various health care related work and was selected to do the work without a proper competitive bidding process. But having seen these processes, they definitely wouldn't have resolved the kinds of widescale technical issues.
- growupkids 13y ago
  And the federal government is a remarkably ignorant customer. They don't know what they are asking for, meddle in the details adding in things they think they need, and don't really know what to pay but often think lower=better. Top that off with insane complex procurement rules that require the work be divided up to meet social goals: x% to small biz, y% to 8a, z% to hub zone and so on and you have a guaranteed mess. Even if there was a highly qualified company to do the work, the FAR requires it be divided up. Top that off with a silly idea that a company needs to have past performance doing something that's never been done before, and well you end up with selection committees picking the "safe" (I.e. Big firm) choice. And you're guaranteed to get a firm that doesn't know how to innovate, they just imitate and throw money, hardware and sub contractors at the problem.
- hga 13y ago
  Addressing you and growupkids, CGI Federal is responsible for the Medicare.gov site, which I can attest to being a bit clunky but otherwise working well for claims and Part D selection and application (the prescription drug benefit program). They were on a list of already vetted vendors, plus if "a proper competitive bidding process" had been done it could easily be hung up in the courts by losers lawsuits. As you note, and as I've been noting, given the structure and processes of the project, especially it being run by political and bureaucratic types from the White House on down to CMS (the integrator prior to getting replaced in late October), there's no way they and the other contractors could have possibly succeeded, especially with change orders continuing though the week before launch. Heck, the integration tests CMS ran in the last week or two before the launch told them it couldn't possibly work, but it was launched anyway....

rathbun 13y ago

Data In. Data Out. I have seen very little of exactly which databases are being integrated. I saw a reference a few days ago that there were many arguments about using SSN in the system. Does that mean that some of systems being pinged for data don't use it? Very hard with SSN, I can't imagine trying to wing a match with similar name, similar address, similar birthday.

United857 13y ago

Anyone know what tech stack they're using? Linux or some other *nix or Microsoft?

deleted 13y ago
[deleted]
genericresponse 13y ago
Most likely multiple vendors. It probably depends on the component and contractor.
toomuchtodo 13y ago
http://medcitynews.com/wp-content/uploads/healthcaredotgovinfographiceduardogarcia-588x968.jpg http://medcitynews.com/wp-content/uploads/healthcaredotgovin... Mirror: http://web.archive.org/web/20131201071252/http://medcitynews.com/wp-content/uploads/healthcaredotgovinfographiceduardogarcia-588x968.jpg http://web.archive.org/web/20131201071252/http://medcitynews...

rdhyee 13y ago

The folks at MarkLogic must be unhappy with being singled out in the following paragraph: Some of the companies building the system opposed an early decision by the Medicare agency to use database software from a company called MarkLogic, which handles data differently from systems by companies like IBM and Oracle. Some suggest that its unfamiliar nature slowed their work. By mid-November, more than six weeks after the rollout, the MarkLogic database -- essentially the website's virtual filing cabinet and index -- continued to perform below expectations, according to one person who works in the command center. The company's response is recorded as: In interviews, MarkLogic’s executives faulted inadequate computing power and instability at the site’s data center, as well as the failure to properly integrate their product, problems repeatedly cited by other website vendors. Any insight on what is meant by how MarkLogic "handles data differently"?

arosenbaum 13y ago
Not really a slam at all. Customers choose MarkLogic to replace Oracle and DB2 specifically because it "handles data differently". "Schemaless" aka "Schema on read" is the biggest difference between Only "SQL" and "Not Only SQL" systems. Scale-out on commodity hw vs. scale-up on shared storage is another. Customers often choose MarkLogic instead of other NoSQL systems because of it's ACID capabilities, ability to run multi-statement transactions between multiple databases (including Oracle and MarkLogic in the same transaction) and a multitude of enterprise capabilities from government grade security to point-in-time restore. Read more: http://developer.marklogic.com/pubs/architecture/inside-marklogic-server-r7.pdf http://developer.marklogic.com/pubs/architecture/inside-mark...
- realmandan 13y ago
  Totally, not a slam at all. As long as you don't think a company name and "perform below expectations" sitting next to each other in a NYT long-form piece is a BAD thing...
  - arosenbaum 13y ago
    I'll repeat what our CEO said in the NYTimes article "He said MarkLogic is performing up to standard, but “the network and the storage systems are not properly sized and not properly run.” Put any database on top of the resources that were provided and I think performance "below expectations" would have been likely.