6 ms·
Troubled Obamacare website wasn't tested until a week before launch
- atwebb 13y agoShocked! I am shocked to hear that a major tech project with a hard deadline was pushed through at the last minute. Kidding aside, a selfish part of me likes things like this because it lifts the veil and shows me that plenty of other people screw up. It's easy to dismiss it as a botch or idiot contractors but I'm guessing there's a lot of smart people, who worked long hours, and it's just hard to do right in these circumstances.
- jreed91 13y agoSurprise, surprise the government is still using waterfall.
- marcc 13y agoWell, read the article. The claim is that requirements were still changing 4 days before the launch. That's not waterfall, that's just no process at all.
- RougeFemme 13y agoI wonder if the requirements had already been finalized and it's just the "final" signed off documentation that was delivered so late. . .yes, still broken/no process.
- RougeFemme 13y agoI find it hard to believe that CMMS actually performed system integration, such as it was. I find it easy to believe that CMMS hired a contractor to perform system integration. And just as the individual contractor employees saw issues, there were probably government employees within CMMS who also saw issues. . .and just enough senior executives - government and contractor - to ignore the "naysayers". . .and insist on having their fingers deeply in the pie of such a high-visibility project. And, of course system integration was not the only project management/technical issue with this project.
- hga 13y agoI don't see why you find it so hard to believe. Hubris is a pretty common human flaw, perhaps it was felt this was too important to leave to the professionals. In this article, we have two sets of sources using two different methods telling us this, and one of them is able to go on record, true name and all. It's also entirely consistent with the observed results (of course, so is having an outside integrator).
- RougeFemme 13y agoClimb down. . .I'm not accusing anyone of lying or fabricating. And yes, I'm well aware of the prevalence of hubris. I find it hard to believe because, having worked on both sides of the fence, I've never known or heard of any agency of any size, even DoD, doing the integration themselves. They almost always have contractors doing it, though they may manage the contractors and call themselves integrators. I believe they may have referred to CMMS as integrators without giving any thought to the contractor who actually performed the integration - at the contracted direction of CMMS.
- hga 13y agoErrr, I didn't think you were, and am sorry if I gave that impression. But I've stumbled across even more confirmation, first from a New York Times article on the 12th that I'm amazed we missed (http://www.nytimes.com/2013/10/13/us/politics/from-the-start-signs-of-trouble-at-health-portal.html?_r=2&pagewanted=all& http://www.nytimes.com/2013/10/13/us/politics/from-the-start...), which says: "One highly unusual decision, reached early in the project, proved critical: the Medicare and Medicaid agency assumed the role of project quarterback, responsible for making sure each separately designed database and piece of software worked with the others, instead of assigning that task to a lead contractor." And then a National Review Online (flagship publication of the Right, going back to 1955) article by a non-specialist who got information from 5 CMMS officials who also aren't software engineering types (but of course all concerned are learning a lot now, as I'm sure most of us did once upon a time). In http://nationalreview.com/corner/361577/assessing-exchanges-yuval-levin http://nationalreview.com/corner/361577/assessing-exchanges-... he says they confirm most everything in the NYT article but the also disputed here by eyewitnesses claim that coding didn't start until February. For integration: "The people I spoke with did all confirm the importance of one other detail in the Times story: that CMS did not hire a general contractor to manage the exchange project but handled that overall technical management task itself. None of the people I spoke with wanted to get into how this decision was made or at what level, but all of them agreed that it was a very bad idea and was at the core of the disaster they have so far experienced." The above also might have input from the 3 insurance industry people he talked to. And it's refreshing to see a lot of people put politics somewhat to the side to figure out just what went wrong in project management and software engineering land. The whole nation is getting an education in this, or as the NRO author put it: "But [the CMMS officials] are policy and management people, not information-technology experts. This latter point turns out to be quite important. The reaction of these individuals to what has happened in the last two weeks is the reaction of people who are coming to realize that their expectations and understanding of web development were mistaken. They believed (as I did too, I admit) that whatever technical problems the exchange sites encountered at first could be cleared up quickly and simply once things got going—that the contractors developing the websites could just respond to problems on the fly, as they became apparent. It is now increasingly obvious to them that this is simply not how things work, that building a website like this is a matter of exceedingly complex programming and not “design,” and that the problems that plague the federal exchanges (and some state exchanges) are much more severe and fundamental than anything they imagined possible. That doesn’t mean they can’t be fixed, of course, and perhaps even fixed relatively quickly, but it means that at the very least the opening weeks (and quite possibly months) of the Obamacare exchanges will be very different from what either the administration or its critics expected." Well, its policy type critics. A lot of us expected the common big project screwups, which are hardly confined to the government.
- deleted 13y ago[deleted]
- JPKab 13y agoI've commented on HN recently on how ABSOLUTELY EYE-GOUGE INDUCING AWFUL it is to do any kind of software work for the Federal Government. Due to a broken hiring process and union-like job security, it is filled with incompetent morons constantly scheming to find any desperate way they can to demonstrate value. (I of course am talking about the GS-12 and up levels) The men and women who work hard and care are like the proverbial Atlas: they do the work of 15, with the other 14 people in their group using their exurban DC commute to excuse their 7AM - 3PM schedule, of which the first 3 hours of their day is spent surfing the net doing jack shit. And then these fucking assclowns get promoted. These are the people who do things like make major interface and usability changes to a massive healthcare website and just magically expect the contractor (CGI in this case) to be able to do it. They don't give a shit about wasting taxpayer money, because it has no effect on their career at all. Failed projects don't get you demoted or fired, schedule slips are so normal that nobody even bats an eyelash.... This causes any sane person who points out the cost in time and money of bullshit bikeshed type changes to be rolled over by the horde of overpaid lemmings who are trying to yell as loud as they can before they head out at 3pm for their kids volleyball game in Manassas. If this looks like a rant, it is. I hate the GS system. It gives government a bad name, and these overpaid assclowns do nothing but fail, supplying ammo to every Rush Limbaugh in the world advocating for elimination of government. Please, please, if somebody with power is reading this: fix the government hiring system. Fire the lazy assholes who work 5 hour days and pull six figures. Fire the people who think not knowing how to use a computer beyond Outlook doesn't interfere with their ability to perform as CIO of an agency. Fire people who are in charge of projects that go over budget and late. Fucking fire people. Next week I'll be going in and meeting a GS-14 (just promoted for 13) who does NOTHING at work but talk about his motorcycle and martial arts. When tasks are handed to him, everyone else groans, knowing that it will simply not be done. Instead they go behind him and perform the tasks and hope he doesn't find out, cause if he does he will pull rank and humiliate them..... Venting over.
- deleted 13y ago[deleted]
- hga 13y ago"every Rush Limbaugh in the world advocating for elimination of government" As far as I know a strawman, unless you can come up with some telling quotes. You really think he's an anarchist??? Now, a return to the Founder's vision of a limited government, the pre-Hoover, not counting e.g. Woodrow Wilson during the war era and other Progressive excesses, that's another thing altogether. Which your vent strongly supports. Basically, seriously reduce the size, scope, and most especially control of the Federal government, and return to the spoils system. No matter how bad it was and would be, it's still accountable in a way we can see the Civil Service isn't. And if the Federal government isn't trying to control just about everything, the need for a huge, "professional", "apolitical" (like the National Park Service, anyone???) "Civil" (again, the NPS is our current poster boy) Service is much less compelling. If you can't admit to yourself that the Rush Limbaughs of the country have some good points, you are part of the problem, and will remain so.
- lvs 13y agoI have no particular opinion about this. However, in the context of an article that relies heavily on unnamed sources, it's worth pointing out that the Washington Examiner is a conservative tabloid founded in 2005 by Philip Anschutz.[1] [1] http://www.politico.com/news/stories/1009/28355.html http://www.politico.com/news/stories/1009/28355.html
- hga 13y ago"the Washington Examiner is a conservative tabloid founded in 2005 by Philip Anschutz" That's a bullshit argument ... especially since we aren't likely to get any good dirt from the cheer-leading part of the MSM (almost all of it minus Fox, which at last count Jay Carney is refusing to call on, rather telling, that, and The Wall Street Journal editorial page). A good argument is from a right winger like me, who from experience says that aside from the columns from major figures like Michael Barone, Washington Examiner is a very low quality rag. To the point that if I see an item linked to on e.g. Drudge, if I notice it's to them, I don't follow it. Just about any major entity on the right is better, save perhaps the Red State web site. On the other hand, this actually isn't the typical report that entirely depends on unnamed sources. It has several from the inside, speaking with specificity, plus one who's able to go on record who used different methods comes to the same basic conclusion about who's the integrator.
- hga 13y agoHere's https://news.ycombinator.com/item?id=6572467 https://news.ycombinator.com/item?id=6572467 a much better/more reliable account from the Right, echoing the CMMS did the integration meme (which apparently first came from the New York Times), and at the end of this long piece the author looks at possible consequences. Particularly the "danger of a rapid adverse selection spiral", that is, the people who will put up with a very difficult site are the ones who really need healthcare, whereas the "The healthy young man who sees an ad for his state exchange during a baseball game and loads up the site to get coverage—the dream consumer so essential to the design of the exchange system—will not keep trying 25 times over a week if the site is not working." Also how fixing the upfront/front end problems will exacerbate what look to be some bad back end problems, particularly the calculation of subsidies, which all the state exchanges depend upon. Ah, and according to the 3 insurance types he talked to, they "believe that only Nevada, Colorado, Washington state, and Kentucky have what could reasonably be described as working systems at this point.", although in general they're in better shape than the federal one.
- puppetmaster3 13y ago$500mm web site. 1/2 of a billion. for a website.
- alphakappa 13y agoPlease stop repeating this incorrect number unless you have a source for it.
- wikiburner 13y agoHere are the most current numbers - $196 Million as of 5/15/2013, and it appears to be well on its way to $292 Million: http://www.nbcnews.com/video/nightly-news/53309356/#53309356 http://www.nbcnews.com/video/nightly-news/53309356/#53309356
- flavor8 13y ago$94 Million. Still bad, but get it right.
- wikiburner 13y agoNope, $196 Million as of 5/15/2013, and it appears to be well on its way to $292 Million: http://www.nbcnews.com/video/nightly-news/53309356/#53309356 http://www.nbcnews.com/video/nightly-news/53309356/#53309356
- flavor8 13y agoMy bad, I hadn't heard that update.
- dingaling 13y ago> for a website. No, for a complex web of third-party integrations underlying byzantine business logic. With a web site as the end-user interface.
- hga 13y ago
- jackmaney 13y agoWait...it was tested?
- alphakappa 13y agoWhy would this be surprising. It's a project with huge scope, with dependencies that they can't fully control (IRS, Social Security etc), with likely changing requirements, and a tight deadline. It's not like they could have released in beta and iterated, or waited until it was perfect. It's the nature of the beast - if the major annoyances don't get fixed in a reasonable [1] amount of time, then it's worth writing reams over. 1. Undefined.
- nhebb 13y agoThe IRS interface is working fine [1]: Sarah Hall Ingram, the director of the IRS's Affordable Care Act office, told a House panel Wednesday that the IRS hasn't experienced similar problems. "The portion of the responsibilities the IRS is in charge of is going fine," Hall Ingram said. [1] http://thehill.com/blogs/healthwatch/health-reform-implementation/327483-irss-obamacare-implementation-going-fine-official-says http://thehill.com/blogs/healthwatch/health-reform-implement...
- snowwrestler 13y agoAccording to the IRS.
- hga 13y agoWell, they could be telling the truth. E.g. "(If and) When we get a query, we respond to it in our N second allowed window" (as I recall the total time allowed is 7 seconds). I'm pretty sure some decent fraction of the entities Healthcare.gov queries behind the scenes are doing their job. Well, most of the time. One thing that occurred to me is that if certain actions require a lot of backend queries to different sources, multiplying the probabilities each could outright fail or blow the window could result in an overall flaky system. Which doesn't directly explain why the offline part of the registration system is failing so often.
- lancewiggs 13y agoThe fundamental issue here, like many projects of this size, is that it was trying to deliver a solution to a ridiculously complicated process. Moreover the way of procuring that solution is also fundamentally broken. Simplifying the underlying process would take a level of politics that is sadly well beyond even one party, let alone two.
- wikiburner 13y agoThere was a $600+ Million price for the website that was previously mentioned in HN comments on this topic, and others were claiming that this number was debunked, and the actual price was $30 Million or so. NBC News has now released a report on what the actual numbers are. It's $196 Million as of 5/15/2013, and the price appears to be on its way to $292 Million: http://www.nbcnews.com/video/nightly-news/53309356/ http://www.nbcnews.com/video/nightly-news/53309356/ Anyway, I apologize if it looks like I'm spamming the thread with it (I posted at the bottom of the thread as a reply). That's not my intent, but I think it's important information.