10 ms·

HealthCare.gov’s glitches prompt Obama to call in expert troubleshooters

77 points by cgtyoder 13y ago

yeukhon 13y ago
What kind of computer programmers did the government hire to oversee the progress? I am going to question this person's ability. While I am just a senior in college, I probably can hammer a lot more practical issues before its launch. I am sure half of the HNers can do that too. sigh
- yeukhon 13y ago
  Thanks for downvoting me, government employees.
  - tmorton 13y ago
    If you're lucky, you'll learn this lesson early in your career: when you think "oh, that's easy, those people must be idiots" it's a sign that you don't understand the problem. People are not idiots. Large problems are never easy. Politics, management, and salesmanship are genuine parts of the problem, not useless distractions from technical stuff. I'm not saying that improvement is impossible, I'm saying that improvement is hard. The healthcare.gov system could have gone live, and been functional, on October 1 - if you could hire better technical people. Can you hire a top-notch team of techies in this environment? Ever tried it? It could be working if the political appointees had nailed down the requirements earlier. Have you ever tried getting hundreds of legally-mandated decisions pushed through the federal bureaucracy? Many of those decisions were pushed back for political reasons, which sucks - but could you made the unpopular decisions early, then won enough elections to keep Republicans from dismantling the entire thing? The system could be working if the law made more sense in a dozen different ways, but all of those compromises were necessary to get the law passed in the first place. Have you ever tried to write legislation, and get it through congress? Don't get me wrong - the healthcare.gov system is a clusterfuck, and there are easily identifiable reasons for that - in retrospect. And hopefully we can learn from the failures and move forward. But laying it all at the feet of some nebulous government employee is completely unhelpful.
pmorici 13y ago
What is the point of healthcare.gov anyways? You can go and buy an ACA compliant plan on a comparison shop website like ehealthinsurance.com that works pretty well why did the government need to create it's own?
- amalag 13y ago
  A very good point. The federal government was supposed to have states create their own exchanges, but yes, this it seems like the commercial space could handle some of this.
  - peterstjohn 13y ago
    Many of the states refused to create their own exchanges, thus the Federal site needed to be more extensive than originally planned (not that's an excuse for the screw-ups at all). States that have their own exchanges seem to be doing better, from anecdotal evidence supplied so far…
    - amalag 13y ago
      It is an interesting question though of why we need a federal site in the first place in order to shop for a commercial product. We are required to have car insurance but we don't need the state government running a car insurance shopping site. I am not for/against the Affordable Care Act, but I think it is a legitimate question.
      - mgkimsal 13y ago
        requirements for car insurance are set by each state, IIRC. and not everyone has a car - it's not mandatory. Everyone has a ... 'health'.
      - amalag 13y ago
        Well the answer is that government sites will determine subsidies. That includes verifying income and checking with numerous federal agencies. http://www.washingtonpost.com/blogs/the-switch/wp/2013/10/09/heres-everything-you-need-to-know-about-obamacares-error-plagued-web-sites/ http://www.washingtonpost.com/blogs/the-switch/wp/2013/10/09...
DigitalSea 13y ago
Sounds like a standard run-of-the-mill Government contract job to me. It's a known fact getting a Government contract is like winning the lottery for a lot of businesses. Ramp up your usual price by 1000%, take twice as long as you should and when SHTH proclaim the problem is a lot more complex and bigger than you thought and all is forgiven. Having said that, not all contractors deliberately waste money and drag out projects, but it is a known aspect of contract work. Whether or not that is the reason in this instance has not been proven and probably never will be. The Obama Care fiasco continues, almost like it was deliberately on purpose in a way.
byoogle 13y ago
Obama didn't read The Mythical Man-Month.
- mangala 13y ago
  (Chuckle) Well seeing as how half the tech managers I talk to at private companies haven't ever heard of it, I think he should be given a little bit of a break.
- venomsnake 13y ago
  He obviously had - his campaign was praised about the high quality of IT personal and solutions that were developed. But after all getting reelected is more important than making sure your signature piece of legislation is actually working at launch.
ams6110 13y ago
So, the fiasco continues. A cynical person might think first contractor failed on purpose, so that the second bunch of "experts" could be brought in to fix it. One system for the price of two (or more, "expert troublshooters" don't come cheap). I'd really like to see a transparent accounting of who is getting paid, and how much, to implement this so-called website.
- KevinEldon 13y ago
  I don't think the cynical view makes a lot of sense. The original contractor is likely billing for hours to fix the site now. Bringing in additional experts doesn't benefit them. A failure of this scale and visibility isn't going to help their business reputation and if they lose the maintenance contract on the site they're likely to be out for millions. I can see bidding low with the expectation of making a killing on change orders (I think the contractor succeeded at that), but outright failure to deliver doesn't make a lot of sense. I think like most IT projects you need to look at the management team in charge and the decisions they made to understand if this failure could have been identified and avoided.
  - mgkimsal 13y ago
    > A failure of this scale and visibility isn't going to help their business reputation Do big failures really hurt people or companies all that much? At some point, you're dealing in a realm in which relatively few people even play (or perhaps even want to play) and you may be the only game in town anyway. Bankers/CEOs who lost millions were still given large bonuses with the reasoning that "we wouldn't be able to find anyone else to do their job". Well... I could certainly lose millions faster than many of them, and I'd do it for half their bonus, thereby saving the companies money :)
    - yen223 13y ago
      That's something I used to wonder too. I used to work for a company that had a number of highly-visible failures, the kind that gets reported in the news. They somehow still managed to score multi-million-dollar contracts. I suspect having a great sales team is more important that having a great engineering team.
DanielBMarkham 13y ago
IIRC, the site was handed out to 55 (!) contractors to create? Maybe #56 is the charm. I take no pleasure in watching this train wreck. It could, however, be a teaching moment for the country on what's wrong with Federal IT. Probably won't be, but it could be. So the story is worth following.
- forkandwait 13y ago
  I think 3 really good programmers, 2 sys admins, 1 non-clueless PM, and 1 person to arrange their travel would have gotten this done faster.
  - brazzy 13y ago
    No - they'd by now have read through about 20% of the regulations the system needs to fulfill.
smoyer 13y ago
I live about three hours away from D.C. and the suburbs where much of the government contracting takes place, and I'm going to make an offer that doesn't throw more of the tax-payer's money into the fire. I will, at my own expense drive to D.C. and spend two days of my vacation evaluating the architecture of the failing system, and spot check the source code quality. Additionally, I'll add the project to my Sonar source code quality analyzer so that there's a focus on what code isn't up to industry standards and which sections should be addressed first. It's still astounding to me that we spent $700M on such a pathetic LITTLE system ... there simply isn't that much code to write. If there are others who'd like to show the government how it should be done, I'd be happy to start an open-source project to rewrite the whole project, as well as a CI/CT/CD/CM system to make sure that the code is maintainable, that changes are delivered in a reasonable amount of time and that servers are constructed and clustered in a consistent way. This really isn't rocket science (and it's a good thing ... there would have been deaths). EDIT: If someone in the government a) reads my offer and b) is willing to let "the industry" voluntarily help, you can contact me via the e-mail address on my profile page.
- noonespecial 13y ago
  They can't afford you for free. It would cost them so much more to vet you than using one of their usual "pre-approved" government contractors, it wouldn't even be worth their considering. Even if that could be done, no one would be willing to take responsibility for giving you access. The system is designed (or has evolved) specifically to diffuse this sort of responsibility. It would be impossible to focus it sharply enough to allow you to do this. The code is broken, but not the broken part of this system.
  - smoyer 13y ago
    I made the offer with the assumption that there were three or four reasons it wouldn't be taken ... you've described one of them. The others are: b) The selected contractor won the work as payback for some political benefit, and the person who made sure they won can't allow them to be perceived as incompetent. c) They don't believe it's really broken so much as it just needs to be "tuned" (almost "the myth of sunk costs"). d) No one in the government cares ... so they're not reading our thoughts and comments on the system.
vaadu 13y ago
Why are there no consequences for this epic failure? Someone should go to jail for lying about the costs and the schedule. Many more should get fired and no longer permitted to work on government projects.
- CamperBob2 13y ago
  Why are there no consequences for this epic failure? Because government. They get paid the same, by you and me, whether it works or not.
- sseveran 13y ago
  This probably doesn't really qualify as a failure yet in government contracting. People haven't been fired for much more twisted projects, like the F35.
spoiledtechie 13y ago
What I am more worried about than any other HealthCare.Gov glitch, is if this website was so badly designed, that the data it holds, the functions it does would also be extremely badly designed and secured. Yet NO ONE is talking about this angle. Thats what keeps me up at night.
napoleond 13y ago
Does anyone have more information about what exactly is going wrong with this site? Apparently Development Seed (the company behind http://mapbox.com http://mapbox.com) was (one of?) the contractors behind the project[0] and they've always seemed (at least judging from their product offerings) quite competent. 0. http://developmentseed.org/blog/2013/06/25/healthcare-launches-in-the-open/ http://developmentseed.org/blog/2013/06/25/healthcare-launch...
- apendleton 13y ago
  Dev Seed did the user-information/knowledge-base component, which has worked fine. They didn't work on the healthcare exchange, which is what has been broken.
bbit 13y ago
Obama is a total idiot.
chaseideas 13y ago
What did anyone really expect? Sounds like a typical government contract. I can only imagine how these conversations go... "We need more QA! STAT! It's going to take at least $50 million to test this system... it's awfully broken!" chuckle face/palm That said, I think there should be a lot more concern about the security of this site if there's been such a mishandling of the basic functions and operational flow. Seems only reasonable to suspect this will be leading to a horrible leak of many people's private records at some near point in the future.
- sseveran 13y ago
  This one doesn't even seem that bad yet. I mean its not like the new bay bridge which was a decade late.
microcolonel 13y ago
What a pile of BS. They're so moronic that they built the thing such that registration of the userbase would take more than a hundred years. Stop coddling the state, they're not going to do it right, they don't welcome the debate, they don't care, they won't present a balanced budget, the debt default is constitutionally prohibited, the war on drugs is a war on people, their wars are so obviously wrong that there are more soldier suicides than deaths in combat, the military burns more oil than the entire population of india, socialism doesn't suddenly work when the current regime tries it.... Oh never mind, go ahead and keep wasting your time listening to sociopaths and participating in silly, futile politics; I'll just make a greasemonkey script to filter out all stories with "Obama" in them, and be on my merry way.
ck2 13y ago
Shouldn't they have been called in a year ago? Six months ago? A month ago? $95 Million apparently buys everything but new management and proper coders.
jowiar 13y ago
The magic quote: One major problem slowing repairs, people close to the program say, is that the Centers for Medicare and Medicaid Services, the federal agency in charge of the exchange, is responsible for making sure that the separately designed databases and pieces of software from 55 contractors work together. We're talking about integrating a slew of both legacy (think decades old) and new (think untested) systems, many of which don't have half-decent APIs to work with (I'd be willing to bet there's at least one "database" in there somewhere that consisted of an Excel workbook on a shared drive, perhaps with an "API" of a paper form and a fax number), and many of which almost certainly weren't built to scale to consumer web traffic. Steve Yegge's platforms rant comes to mind here, and from talking with some of my federal employee friends about the challenges they face, get to root of the problem. Building protocol-centric services from the beginning goes a long way towards fostering interoperability. More broadly (and apologies for the non-HN bent to this), the project serves as an example for the utter shitshow that is the United States healthcare system. The exchange is intended to serve as an abstraction for the process of figuring out what public and private options are available to you. The motivation for building the exchange was "Hey! Figuring out what options exist for healthcare is complicated!". Of course, building such a system doesn't make the complexity go away - it just makes it someone else's job. Perhaps that this determination is so complicated that it requires a ridiculous system is an indication that we're "doing it wrong".
- joemaller1 13y ago
  I've seen this "ancient systems" idea go by a few times. Honest question: If these platforms are so old, they likely aren't terribly complex, from the perspective of what's now computationally cheap. So, how hard would it be to drop a computer on their local network and just dumb-clone the whole thing?
JumpCrisscross 13y ago
Politics were allowed to trump project management. “One highly unusual decision, reached early in the project, proved critical: the Medicare and Medicaid agency assumed the role of project quarterback, responsible for making sure each separately designed database and piece of software worked with the others, instead of assigning that task to a lead contractor. Some people intimately involved in the project seriously doubted that the agency had the in-house capability to handle such a mammoth technical task of software engineering while simultaneously supervising 55 contractors. An internal government progress report in September 2011 identified a lack of employees ‘to manage the multiple activities and contractors happening concurrently’ as a ‘major risk’ to the whole project” [1]. It is speculated that "facing intense opposition from congressional Republicans, the administration was in a bunker mentality as it built the enrollment system, one former administration official said. Officials feared that if they called on outsiders to help with the technical details of how to run a commerce website, those companies could be subpoenaed by Hill Republicans, the former aide said” [2]. [1] http://www.nytimes.com/2013/10/13/us/politics/from-the-start-signs-of-trouble-at-health-portal.html?hpw&pagewanted=all&_r=0 http://www.nytimes.com/2013/10/13/us/politics/from-the-start... [2] http://www.politico.com/story/2013/10/obamacare-aca-website-broken-98412_Page2.html#ixzz2iL0ClGKC http://www.politico.com/story/2013/10/obamacare-aca-website-...
- nhebb 13y ago
  [2] doesn't make sense to me. > "Officials feared that if they called on outsiders to help with the technical details" Aren't the 55 contractors outsiders? This sounds like typical DC finger pointing, blame game, and the-buck-doesn't-stop-here tactics.
- jbooth 13y ago
  It's washington, politics are part of project management. A 'bunker mentality' WRT subpoenas is totally justified on this project, given how Republicans have acted. You want to pay people by the hour to give testimony while their job goes undone? As for having medicare/medicaid quarterback it.. honestly, that makes sense to me too. It's healthcare in the government, it involves deep integration with medicare/medicaid, if they're not capable of managing those contractors then they need to learn how. I'm not defending the execution in general, but those 2 specific decisions seem ok to me. There weren't a lot of political problems building the site, so those decisions paid off -- they just needed to execute better on the technical stuff.
devx 13y ago
Now what? Another $600 million down the drain to fix the problems of the first $600 million?
dodyg 13y ago
This type of integration project is always challenging regardless whether it is government/private sector. Then add tons of required-by-law requirements and you are talking about massive tasks to get things right. Their main problem is the hard deadline for October 1st launch. Things should have been rolled out state by state - but I am not sure whether this is in the law or just a political decision.
nbevans 13y ago
They could learn a lot from how UK.gov (http://digital.cabinetoffice.gov.uk/ http://digital.cabinetoffice.gov.uk/) operates these days. It's a true agile shop, that's been ridiculously productive and forward-thinking for at least the last 5 years.
pyalot2 13y ago
The solution was built in 2/3rds by Accidenture, who got awared a $400m contract (out of the total of $600m awarded) with the bulk of the remainder going to IBM and the 50 other contractors. If I know one thing, and I say this from firsthand experience, it's that Accidenture is absolutely the worst choice for any kind of contract. Accidenture isn't an "outsourcing" company. They're an oversized roleplaying project, where the overlords (controllers) have the power and whim to kill projects and just pay the termination penalty, and where everybody strives to become a partner because it's up or out. Any problem inside Accidenture is seen as a nail, and every solution is a hammer, meaning a combination of ISS, MSSQL, Sharepoint, Windows Server and copious application of ridiculously overpriced "consultants".
mortdeus 13y ago
All I know is that they need to start making decisions like minimizing their css and js scripts and fetching jquery from Google before I start taking them serious when they say the best engineers in the fed are working on it. <script src='//assets.healthcare.gov/global/js/lib/jquery-1.8.2.js'></script> https://www.healthcare.gov/css/all.css https://www.healthcare.gov/css/all.css https://www.healthcare.gov/js/all.js https://www.healthcare.gov/js/all.js
bnolsen 13y ago
So in order to reduce health care costs the government declares itself as all knowing in the medical industry, inserts itself into the system with all the beauracracy. Sounds to me like adding more middle men just raises costs and also happens to kick open the door for typical government graft, corruption and embezzlement. Better yet the government sets itself up to decide who gets treatment, and data mining by the NSA and IRS can help identify polical opponents or non supporters to determine on the fly available coverages and rates. Perfect for any tyranical statist type. Does anyone really believe the whole website thingy wasn't full of graft, corruption, kickbacks and paybacks? Come on, seriously.

77 points by cgtyoder 13y ago

What kind of computer programmers did the government hire to oversee the progress? I am going to question this person's ability. While I am just a senior in college, I probably can hammer a lot more practical issues before its launch. I am sure half of the HNers can do that too. sigh

yeukhon 13y ago
Thanks for downvoting me, government employees.
- tmorton 13y ago
  If you're lucky, you'll learn this lesson early in your career: when you think "oh, that's easy, those people must be idiots" it's a sign that you don't understand the problem. People are not idiots. Large problems are never easy. Politics, management, and salesmanship are genuine parts of the problem, not useless distractions from technical stuff. I'm not saying that improvement is impossible, I'm saying that improvement is hard. The healthcare.gov system could have gone live, and been functional, on October 1 - if you could hire better technical people. Can you hire a top-notch team of techies in this environment? Ever tried it? It could be working if the political appointees had nailed down the requirements earlier. Have you ever tried getting hundreds of legally-mandated decisions pushed through the federal bureaucracy? Many of those decisions were pushed back for political reasons, which sucks - but could you made the unpopular decisions early, then won enough elections to keep Republicans from dismantling the entire thing? The system could be working if the law made more sense in a dozen different ways, but all of those compromises were necessary to get the law passed in the first place. Have you ever tried to write legislation, and get it through congress? Don't get me wrong - the healthcare.gov system is a clusterfuck, and there are easily identifiable reasons for that - in retrospect. And hopefully we can learn from the failures and move forward. But laying it all at the feet of some nebulous government employee is completely unhelpful.

pmorici 13y ago

What is the point of healthcare.gov anyways? You can go and buy an ACA compliant plan on a comparison shop website like ehealthinsurance.com that works pretty well why did the government need to create it's own?

amalag 13y ago
A very good point. The federal government was supposed to have states create their own exchanges, but yes, this it seems like the commercial space could handle some of this.
- peterstjohn 13y ago
  Many of the states refused to create their own exchanges, thus the Federal site needed to be more extensive than originally planned (not that's an excuse for the screw-ups at all). States that have their own exchanges seem to be doing better, from anecdotal evidence supplied so far…
  - amalag 13y ago
    It is an interesting question though of why we need a federal site in the first place in order to shop for a commercial product. We are required to have car insurance but we don't need the state government running a car insurance shopping site. I am not for/against the Affordable Care Act, but I think it is a legitimate question.
    - mgkimsal 13y ago
      requirements for car insurance are set by each state, IIRC. and not everyone has a car - it's not mandatory. Everyone has a ... 'health'.
    - amalag 13y ago
      Well the answer is that government sites will determine subsidies. That includes verifying income and checking with numerous federal agencies. http://www.washingtonpost.com/blogs/the-switch/wp/2013/10/09/heres-everything-you-need-to-know-about-obamacares-error-plagued-web-sites/ http://www.washingtonpost.com/blogs/the-switch/wp/2013/10/09...

DigitalSea 13y ago

Sounds like a standard run-of-the-mill Government contract job to me. It's a known fact getting a Government contract is like winning the lottery for a lot of businesses. Ramp up your usual price by 1000%, take twice as long as you should and when SHTH proclaim the problem is a lot more complex and bigger than you thought and all is forgiven. Having said that, not all contractors deliberately waste money and drag out projects, but it is a known aspect of contract work. Whether or not that is the reason in this instance has not been proven and probably never will be. The Obama Care fiasco continues, almost like it was deliberately on purpose in a way.

byoogle 13y ago

Obama didn't read The Mythical Man-Month.

mangala 13y ago
(Chuckle) Well seeing as how half the tech managers I talk to at private companies haven't ever heard of it, I think he should be given a little bit of a break.
venomsnake 13y ago
He obviously had - his campaign was praised about the high quality of IT personal and solutions that were developed. But after all getting reelected is more important than making sure your signature piece of legislation is actually working at launch.

ams6110 13y ago

So, the fiasco continues. A cynical person might think first contractor failed on purpose, so that the second bunch of "experts" could be brought in to fix it. One system for the price of two (or more, "expert troublshooters" don't come cheap). I'd really like to see a transparent accounting of who is getting paid, and how much, to implement this so-called website.

KevinEldon 13y ago
I don't think the cynical view makes a lot of sense. The original contractor is likely billing for hours to fix the site now. Bringing in additional experts doesn't benefit them. A failure of this scale and visibility isn't going to help their business reputation and if they lose the maintenance contract on the site they're likely to be out for millions. I can see bidding low with the expectation of making a killing on change orders (I think the contractor succeeded at that), but outright failure to deliver doesn't make a lot of sense. I think like most IT projects you need to look at the management team in charge and the decisions they made to understand if this failure could have been identified and avoided.
- mgkimsal 13y ago
  > A failure of this scale and visibility isn't going to help their business reputation Do big failures really hurt people or companies all that much? At some point, you're dealing in a realm in which relatively few people even play (or perhaps even want to play) and you may be the only game in town anyway. Bankers/CEOs who lost millions were still given large bonuses with the reasoning that "we wouldn't be able to find anyone else to do their job". Well... I could certainly lose millions faster than many of them, and I'd do it for half their bonus, thereby saving the companies money :)
  - yen223 13y ago
    That's something I used to wonder too. I used to work for a company that had a number of highly-visible failures, the kind that gets reported in the news. They somehow still managed to score multi-million-dollar contracts. I suspect having a great sales team is more important that having a great engineering team.

DanielBMarkham 13y ago

IIRC, the site was handed out to 55 (!) contractors to create? Maybe #56 is the charm. I take no pleasure in watching this train wreck. It could, however, be a teaching moment for the country on what's wrong with Federal IT. Probably won't be, but it could be. So the story is worth following.

forkandwait 13y ago
I think 3 really good programmers, 2 sys admins, 1 non-clueless PM, and 1 person to arrange their travel would have gotten this done faster.
- brazzy 13y ago
  No - they'd by now have read through about 20% of the regulations the system needs to fulfill.

smoyer 13y ago

I live about three hours away from D.C. and the suburbs where much of the government contracting takes place, and I'm going to make an offer that doesn't throw more of the tax-payer's money into the fire. I will, at my own expense drive to D.C. and spend two days of my vacation evaluating the architecture of the failing system, and spot check the source code quality. Additionally, I'll add the project to my Sonar source code quality analyzer so that there's a focus on what code isn't up to industry standards and which sections should be addressed first. It's still astounding to me that we spent $700M on such a pathetic LITTLE system ... there simply isn't that much code to write. If there are others who'd like to show the government how it should be done, I'd be happy to start an open-source project to rewrite the whole project, as well as a CI/CT/CD/CM system to make sure that the code is maintainable, that changes are delivered in a reasonable amount of time and that servers are constructed and clustered in a consistent way. This really isn't rocket science (and it's a good thing ... there would have been deaths). EDIT: If someone in the government a) reads my offer and b) is willing to let "the industry" voluntarily help, you can contact me via the e-mail address on my profile page.

noonespecial 13y ago
They can't afford you for free. It would cost them so much more to vet you than using one of their usual "pre-approved" government contractors, it wouldn't even be worth their considering. Even if that could be done, no one would be willing to take responsibility for giving you access. The system is designed (or has evolved) specifically to diffuse this sort of responsibility. It would be impossible to focus it sharply enough to allow you to do this. The code is broken, but not the broken part of this system.
- smoyer 13y ago
  I made the offer with the assumption that there were three or four reasons it wouldn't be taken ... you've described one of them. The others are: b) The selected contractor won the work as payback for some political benefit, and the person who made sure they won can't allow them to be perceived as incompetent. c) They don't believe it's really broken so much as it just needs to be "tuned" (almost "the myth of sunk costs"). d) No one in the government cares ... so they're not reading our thoughts and comments on the system.

vaadu 13y ago

Why are there no consequences for this epic failure? Someone should go to jail for lying about the costs and the schedule. Many more should get fired and no longer permitted to work on government projects.

CamperBob2 13y ago
Why are there no consequences for this epic failure? Because government. They get paid the same, by you and me, whether it works or not.
sseveran 13y ago
This probably doesn't really qualify as a failure yet in government contracting. People haven't been fired for much more twisted projects, like the F35.

spoiledtechie 13y ago

What I am more worried about than any other HealthCare.Gov glitch, is if this website was so badly designed, that the data it holds, the functions it does would also be extremely badly designed and secured. Yet NO ONE is talking about this angle. Thats what keeps me up at night.

napoleond 13y ago

Does anyone have more information about what exactly is going wrong with this site? Apparently Development Seed (the company behind http://mapbox.com http://mapbox.com) was (one of?) the contractors behind the project[0] and they've always seemed (at least judging from their product offerings) quite competent. 0. http://developmentseed.org/blog/2013/06/25/healthcare-launches-in-the-open/ http://developmentseed.org/blog/2013/06/25/healthcare-launch...

apendleton 13y ago
Dev Seed did the user-information/knowledge-base component, which has worked fine. They didn't work on the healthcare exchange, which is what has been broken.

bbit 13y ago

Obama is a total idiot.

chaseideas 13y ago

What did anyone really expect? Sounds like a typical government contract. I can only imagine how these conversations go... "We need more QA! STAT! It's going to take at least $50 million to test this system... it's awfully broken!" chuckle face/palm That said, I think there should be a lot more concern about the security of this site if there's been such a mishandling of the basic functions and operational flow. Seems only reasonable to suspect this will be leading to a horrible leak of many people's private records at some near point in the future.

sseveran 13y ago
This one doesn't even seem that bad yet. I mean its not like the new bay bridge which was a decade late.

microcolonel 13y ago

What a pile of BS. They're so moronic that they built the thing such that registration of the userbase would take more than a hundred years. Stop coddling the state, they're not going to do it right, they don't welcome the debate, they don't care, they won't present a balanced budget, the debt default is constitutionally prohibited, the war on drugs is a war on people, their wars are so obviously wrong that there are more soldier suicides than deaths in combat, the military burns more oil than the entire population of india, socialism doesn't suddenly work when the current regime tries it.... Oh never mind, go ahead and keep wasting your time listening to sociopaths and participating in silly, futile politics; I'll just make a greasemonkey script to filter out all stories with "Obama" in them, and be on my merry way.

ck2 13y ago

Shouldn't they have been called in a year ago? Six months ago? A month ago? $95 Million apparently buys everything but new management and proper coders.

jowiar 13y ago

The magic quote: One major problem slowing repairs, people close to the program say, is that the Centers for Medicare and Medicaid Services, the federal agency in charge of the exchange, is responsible for making sure that the separately designed databases and pieces of software from 55 contractors work together. We're talking about integrating a slew of both legacy (think decades old) and new (think untested) systems, many of which don't have half-decent APIs to work with (I'd be willing to bet there's at least one "database" in there somewhere that consisted of an Excel workbook on a shared drive, perhaps with an "API" of a paper form and a fax number), and many of which almost certainly weren't built to scale to consumer web traffic. Steve Yegge's platforms rant comes to mind here, and from talking with some of my federal employee friends about the challenges they face, get to root of the problem. Building protocol-centric services from the beginning goes a long way towards fostering interoperability. More broadly (and apologies for the non-HN bent to this), the project serves as an example for the utter shitshow that is the United States healthcare system. The exchange is intended to serve as an abstraction for the process of figuring out what public and private options are available to you. The motivation for building the exchange was "Hey! Figuring out what options exist for healthcare is complicated!". Of course, building such a system doesn't make the complexity go away - it just makes it someone else's job. Perhaps that this determination is so complicated that it requires a ridiculous system is an indication that we're "doing it wrong".

joemaller1 13y ago
I've seen this "ancient systems" idea go by a few times. Honest question: If these platforms are so old, they likely aren't terribly complex, from the perspective of what's now computationally cheap. So, how hard would it be to drop a computer on their local network and just dumb-clone the whole thing?

JumpCrisscross 13y ago

Politics were allowed to trump project management. “One highly unusual decision, reached early in the project, proved critical: the Medicare and Medicaid agency assumed the role of project quarterback, responsible for making sure each separately designed database and piece of software worked with the others, instead of assigning that task to a lead contractor. Some people intimately involved in the project seriously doubted that the agency had the in-house capability to handle such a mammoth technical task of software engineering while simultaneously supervising 55 contractors. An internal government progress report in September 2011 identified a lack of employees ‘to manage the multiple activities and contractors happening concurrently’ as a ‘major risk’ to the whole project” [1]. It is speculated that "facing intense opposition from congressional Republicans, the administration was in a bunker mentality as it built the enrollment system, one former administration official said. Officials feared that if they called on outsiders to help with the technical details of how to run a commerce website, those companies could be subpoenaed by Hill Republicans, the former aide said” [2]. [1] http://www.nytimes.com/2013/10/13/us/politics/from-the-start-signs-of-trouble-at-health-portal.html?hpw&pagewanted=all&_r=0 http://www.nytimes.com/2013/10/13/us/politics/from-the-start... [2] http://www.politico.com/story/2013/10/obamacare-aca-website-broken-98412_Page2.html#ixzz2iL0ClGKC http://www.politico.com/story/2013/10/obamacare-aca-website-...

nhebb 13y ago
[2] doesn't make sense to me. > "Officials feared that if they called on outsiders to help with the technical details" Aren't the 55 contractors outsiders? This sounds like typical DC finger pointing, blame game, and the-buck-doesn't-stop-here tactics.
jbooth 13y ago
It's washington, politics are part of project management. A 'bunker mentality' WRT subpoenas is totally justified on this project, given how Republicans have acted. You want to pay people by the hour to give testimony while their job goes undone? As for having medicare/medicaid quarterback it.. honestly, that makes sense to me too. It's healthcare in the government, it involves deep integration with medicare/medicaid, if they're not capable of managing those contractors then they need to learn how. I'm not defending the execution in general, but those 2 specific decisions seem ok to me. There weren't a lot of political problems building the site, so those decisions paid off -- they just needed to execute better on the technical stuff.

devx 13y ago

Now what? Another $600 million down the drain to fix the problems of the first $600 million?

dodyg 13y ago

This type of integration project is always challenging regardless whether it is government/private sector. Then add tons of required-by-law requirements and you are talking about massive tasks to get things right. Their main problem is the hard deadline for October 1st launch. Things should have been rolled out state by state - but I am not sure whether this is in the law or just a political decision.

nbevans 13y ago

They could learn a lot from how UK.gov (http://digital.cabinetoffice.gov.uk/ http://digital.cabinetoffice.gov.uk/) operates these days. It's a true agile shop, that's been ridiculously productive and forward-thinking for at least the last 5 years.

pyalot2 13y ago

The solution was built in 2/3rds by Accidenture, who got awared a $400m contract (out of the total of $600m awarded) with the bulk of the remainder going to IBM and the 50 other contractors. If I know one thing, and I say this from firsthand experience, it's that Accidenture is absolutely the worst choice for any kind of contract. Accidenture isn't an "outsourcing" company. They're an oversized roleplaying project, where the overlords (controllers) have the power and whim to kill projects and just pay the termination penalty, and where everybody strives to become a partner because it's up or out. Any problem inside Accidenture is seen as a nail, and every solution is a hammer, meaning a combination of ISS, MSSQL, Sharepoint, Windows Server and copious application of ridiculously overpriced "consultants".

mortdeus 13y ago

All I know is that they need to start making decisions like minimizing their css and js scripts and fetching jquery from Google before I start taking them serious when they say the best engineers in the fed are working on it. <script src='//assets.healthcare.gov/global/js/lib/jquery-1.8.2.js'></script> https://www.healthcare.gov/css/all.css https://www.healthcare.gov/css/all.css https://www.healthcare.gov/js/all.js https://www.healthcare.gov/js/all.js

bnolsen 13y ago

So in order to reduce health care costs the government declares itself as all knowing in the medical industry, inserts itself into the system with all the beauracracy. Sounds to me like adding more middle men just raises costs and also happens to kick open the door for typical government graft, corruption and embezzlement. Better yet the government sets itself up to decide who gets treatment, and data mining by the NSA and IRS can help identify polical opponents or non supporters to determine on the fly available coverages and rates. Perfect for any tyranical statist type. Does anyone really believe the whole website thingy wasn't full of graft, corruption, kickbacks and paybacks? Come on, seriously.