20 ms·

Healthcare.gov failed despite agile practices

58 points by refactormonkey 13y ago

Gorimek 13y ago
The evidence presented in the article is quite weak: "I have seen some of the developer documentation, and it clearly discusses sprints, user stories and incremental testing — all of which are hallmarks of an agile process." For a project with reportedly 50+ subsystems, where the main failures seem to have been integration, that doesn't tell us much either about this project or agile practices.
seivan 13y ago
Or 'agile' lead by a non-engineer. Getting tired of 'agile' or 'XP' getting the blame when in reality they are half-assed versions of some scrum masters cook up.
iyulaev 13y ago
Why don't we take the brilliant hackers and programmers at the NSA and task them to fix and re-build the ACA exchange system? Building a database to track the communications of ever single person in the US and ever foreigner they talk to? Piece of cake. Building a database to track the health insurance policies for a subset of Americans? Over budget, behind schedule, still doesn't really work yet. Priorities.
- caryhartline 13y ago
  Those are two things that don't have anything to do with each other. You seem to think that every single web service and database works the exact same way.
- bmelton 13y ago
  Having actually worked with NSA programmers, I can tell you that while they tend to be absolutely brilliant, that doesn't necessarily make them any better at making things "easy to use", "intuitive" or even necessarily "good". When it comes to algorithm design, reverse engineering, or crypto, I know exactly who to call when I need help. When it comes to a layout that works, or in many cases, even just coding up some fairly simple HTML with CSS, my NSA buddies aren't in the top 50 of people I'd call. Yes, they're brilliant. The best of them though, produce websites (if they're capable of coding on the web) that look more like stallman.org than anything like healthcare.org.
  - udit99 13y ago
    I see your point, but the primary challenge of delivering large, complex software projects in a reasonable timeframe applies to both endeavours.
- monokrome 13y ago
  They're too busy stealing your emails, man. They don't have time to spend helping anyone.
- andrewfong 13y ago
  Not that I'd hire the NSA folks per se, but I'd imagine that the NSA's programmers are full time employees. One thing that might dramatically improve government IT in general is to shift more of it in-house rather than relying on contractors.
- jacalata 13y ago
  What makes you think that was on time or in budget?
  - hga 13y ago
    We don't hear much about it, but the NSA has had some stunning IT failures. Development (heck, they can't open their Utah site because of severe, ruin $100K of hardware each time, power glitches), something major above their collection system(s) failed hard once for more than a few days ... and since they're the No Such Agency, we only hear about some of the very worst ones, I'm sure. Flip side is, doing this in the black budget does allow them to avoid some of the usual government contracting insanities. While that's no gaurentee of success, e.g. look at the NRO's recent expensive stumbles, from the dozen years I spent Inside the Beltway I've gotten a strong impression the intelligence community does a generally much better job than run of the mill open Federal IT.
mhurron 13y ago
Agile is not some magic bullet that guarantees success. To say that something succeeded or failed because of some particular method of project management is is simply absurd.
Maarten88 13y ago
No process can compensate for an unsufficiently qualified/talented team.
jroseattle 13y ago
Healthcare.gov was not built in an agile manner. Sure, some items that are labeled as artifacts of agile-based software development, i.e. user stories, may have been utilized, but that doesn't make for an agile process. Heck, even an agile process doesn't make one agile. To say it failed "despite agile practices" misses the mark. It's like blaming a roof for leaking despite the use of hammers. Process is a tool, and if used properly, it may be a solution. But pixie dust, it is not.
- sanskritabelt 13y ago
  That's like saying 'oh its not capitalism it's crony capitalism' or similar excuses.
stirno 13y ago
Reacting mostly to the link-baity title.. Corporations act like using the word 'Agile' instantly brings the benefits of flexible planning and proper management to a team. Most people never really interact with a proper agile process - its kind of a unicorn in the development world. The failure is not necessarily the process but the managers not understanding that it isn't a silver bullet. It is often used as an excuse to delay requirements gathering or understanding of requested features from stakeholders. This leads to churn and failure.
koudi 13y ago
Nothing new, really. Methods and tools can help you, but do not guarantee success. Saying that some project failed because they used/didn't use something dosn't make sense to me. In the end, it's all about developers and overall project management.
mkramlich 13y ago
it's almost certainly yet another case of "dumb people in, dumb results out" dumb + agile = agile dumb
YZF 13y ago
Let's face it- Agile has become a codeword for let's micromanage software developers while increasing the pressure on them to deliver. My theory is that Ken Schwaber and Jeff Sutherland came up with Scrum in order to limit management's ability to screw up software development in corporate settings. That was mixed with some observations about the need for iteration, not doing upfront work when it's wasteful (and it's not always wasteful) and visibility that came from the Agile manifesto. However, one lesson I'm constantly re-learning is never underestimate people's ability to screw things up and I think Ken and Jeff underestimated. The whole agile thing has been subverted and if indeed Healthcare.gov was done using "Agile" practices then this is another bit of evidence. We, as software engineers, need to move away from these and figure out some way to qualify what makes good software engineering. You don't see non-technical hospital managers going around telling doctors where to place an incision and you don't see non-technical aerospace management telling engineers what bolt to use in a rocket engine. But for some reason "programming" has been demoted to something that just being "agile" is enough to do. Badly. In his YouTube Scrum talk Ken Schwaber says something about if you have a poor team with bad engineering practices than Scrum will generate crap for you sprint after sprint. Somehow no one listens to that part.
- jarrett 13y ago
  > figure out some way to qualify what makes good software engineering I doubt we'll ever be able to distill it to a formula. People ran successful software projects decades ago, long before any methodologies were formally described. And they have failing projects now, even when they use a supposedly good formal methodology. Instead of another alternative to waterfall, scrum, or agile, I'd propose the following ingredients for success: Organized people, a good working relationship with plenty of trust, a clear understanding of roles and responsibilities, realistic expectations, and colleagues who honor their agreements with each other. To name a few. All of this is peopleware, not process. Unfortunately, there's no silver bullet to get this stuff right.
  - fragsworth 13y ago
    Agreed. These development processes and methodologies are pretty strange, actually. The confidence expressed by their proponents in the face of failure gives it a very cultish vibe, and I get the sense that they're just making things up for the sake of having "an answer". The best answer is probably to get people who are experienced in similar projects to manage your project, and has little to do with the process they choose to use.
djvu9 13y ago
Many software development failures are due to lack of understanding and controlling the complexity given cost, people and time constraints. Processes helps the controlling part but sometimes gives a wrong illusion that things are under control and thus hurts the understanding part: people stop bothering to think. Building large software is really delicate work. You need to have a correct decomposition (thus to reduce the level of complexity) and have a good control of flexibility and complexity trade off on the component level. It is somehow like solving a formula with many parameters. You can not just look at one or two of them.
pasbesoin 13y ago
I'll have to read the article, but at an initial reaction, I think this project was not suffering for a lack of "agile" practice. The didn't need a bunch of "nimble", independent teams. They needed some early, stable requirements. Some early, hard deadlines that they had to hit. And a focus on making the fucking back end work, first. The last thing this project needed was some sort of last minute "agile". Of course, now they'll attempt some variation of it, out of necessity.
MortenK 13y ago
Project went bad? It's just a matter of fixing the process. That's a reasoning that's simple, easy to understand and entirely wrong. Contrary to what the evangelists' tell us, process has small (I'd almost say minimal) influence on software development failure or success. The responsibility is on the lead project manager. He's not been good enough. A failure on a project of this size doesn't just happen overnight. If you just look, you can see it coming from miles away. The solution will always be highly uncomfortable though. You'd have to stand up to powerful people who hold authority over you, fire sub-contractors or employees, significantly reduce scope, move deadlines. Big, high impact actions that won't be popular, hell they might even get you fired. Not a job for the faint of heart for sure. It's depressingly typical of big corporations and organizations, to allow people not capable enough to be put in charge of such large projects. And it's even more depressing knowing tax dollars are being wasted in such huge amounts on relatively (big emphasis) simple software.
gemma 13y ago
Wow, massively flawed premise here. 1. Using user stories to define your requirements does not make your process agile, just like writing a traditional requirements doc does not make your process waterfall. It's not about the tools. If you're not capturing the requirements you need to capture, your process is just broken. 2. Design documents do not magically make your project successful. Agile doesn't like design documents because it typically uses more lightweight artifacts to capture design decisions--but it still captures them! Again, if those decisions are not captured at all, your process is broken. 3. Wat? Asynchronous vs. synchronous is a project design decision; how does this demonstrate the failure of agile methods? Multiple sources ([1], [2], etc.) have reported that the requirements for the system were not known until Spring of this year, and that they were in flux until weeks before the release. That's not a problem that would be fixed with use cases, UML and design documents. [1] http://www.nytimes.com/2013/10/13/us/politics/from-the-start-signs-of-trouble-at-health-portal.html?pagewanted=all http://www.nytimes.com/2013/10/13/us/politics/from-the-start... [2] http://www.huffingtonpost.com/2013/10/22/obamacare-website-programmers_n_4141411.html http://www.huffingtonpost.com/2013/10/22/obamacare-website-p...
coldcode 13y ago
Failure is such a imprecise word. Assuming it works eventually is that failure or success? Success or failure, assuming you can even define them precisely, and blaming the result on some vaguely defined reason is pointless. You may as well blame healthcare.gov's problems on bad pizza.
natrius 13y ago
Isn't a system specified by legislation fundamentally at odds with an agile process? Neither the requirements nor the dates could change without an act of Congress (or in some cases, the President).
- ericcumbee 13y ago
  Most of the Legislation on finer points of implementation are simply "In a manner that the Secretary shall designate."
  - mgkimsal 13y ago
    Doesn't necessarily help on timing though, if the legislation has specific deadlines built in.
fragsworth 13y ago
No, it failed because it was managed by technically inexperienced and/or incompetent people. These people know who they are (the contractors), but they won't admit it. Talk about process all you want, but everyone is just waving their hands around spouting buzz words. A technically experienced (in large scale systems) management team can make sure the project is done right regardless of the process they use.
- hga 13y ago
  The "technically inexperienced and/or incompetent" managers by now have been well established, CMS on up to the White House, and CMS at least has officially been removed from it's role, replaced by QSSI. What I wonder is how many political types are still telling the new fix-it czar, Jeff Zients, what he can and can't do. I.e. right now all features that can possibly be punting into the future should be. Although he said his #1 priority was to stop sending garbage to insurers, which isn't particularly political at this point, one would hope.
droopybuns 13y ago
I would love to know what version control software was implemented throughout the project. I suspect that the answer is none. But if I'm wrong, it'd be interesting to know what tools they tried to use and where there processes worked, if they ever worked at all.
- jroseattle 13y ago
  Why would you suspect that version control wasn't in use? Is there an aspect of this project that would be resolved with a specific version control system? These aren't rhetorical questions, I'm genuinely curious why you're asking.
  - GhotiFish 13y ago
    A guess at his line of thinking: People, who didn't understand the development processes programmers use, caught a wiff of agile. They mandated agile, but only established the methodology, with no technical direction to back it up. So in the arena of VCS the novice teams didn't use anything (they didn't know better), teams with some experienced devs were ignored, or used their own VCS. a symptom of a more systemic problem.
    - jroseattle 13y ago
      Wow, that's alarming. We need a real post-mortem on this process. The congressional inquisitions aren't getting to the root of the problem. We just need the individual who managed this project to explain, from square one, how the project flow went. (Believe it or not, I intend no sarcasm with this post.)
      - GhotiFish 13y ago
        >(Believe it or not, I intend no sarcasm with this post.) that didn't help T.T
        deleted 13y ago
        [deleted]
hobs 13y ago
Ignoring the agile point, the whole idea of requesting data from ye olde bespoke applications and data storage systems in a synchronous manner for (even just) a few thousand users seems like it is destined to fail.
ajarmst 13y ago
What? Agile's not magic?
poissonpie 13y ago
A bad workman blames his/her tools. Bad project manager blames his/her process.
- hga 13y ago
  "Bad project manager blames his/her process." In my experience he blames one or more of his subordinates. "His process" implies ownership of something responsible for the failure. Shifting blame is pretty much by definition a core competency for bad project managers, it's that, get better or get fired.
5vforest 13y ago
Seems like as good of a time as any to bring up Zed Shaw's "Programming, Motherfucker": http://programming-motherfucker.com/ http://programming-motherfucker.com/ We are a community of motherfucking programmers who have been humiliated by software development methodologies for years. We are tired of XP, Scrum, Kanban, Waterfall, Software Craftsmanship (aka XP-Lite) and anything else getting in the way of...Programming, Motherfucker. We are tired of being told we're autistic idiots who need to be manipulated to work in a Forced Pair Programming chain gang without any time to be creative because none of the 10 managers on the project can do... Programming, Motherfucker. We must destroy these methodologies that get in the way of...Programming, Motherfucker.
- trentmb 13y ago
  He's very annoying to read. EDIT: To clarify, his style is incredibly annoying.
  - djillionsmix 13y ago
    Annoying to people who don't like that he's correct
    - deleted 13y ago
      [deleted]
    - tcdent 13y ago
      I agree with his thinking, but his presentation is just too in-your-face. It can be hard to find a balance when communicating something you're passionate about.
    - rbanffy 13y ago
      Actually, no. He isn't right. Any significant programming project involves more than one person. Being arrogant is probably the most serious flaw in a programmer. You will be wrong. You will under or overestimate something important. You do have blind spots. Most important, you are not as smart as you think you are. If you are not careful, your mistakes will ruin your project. They'll delay you or create liabilities you'll have to deal with. Methodologies exist to help you manage yourself, not for your managers to better manage you (although they can be misused for that). You don't have to feel "humiliated" by them (neither methodologies nor managers). You don't have to be insecure.
duhballs 13y ago
guess which two parts actually work?
dep_b 13y ago
As a user I would like to be able to use the Healthcare.gov with at least 50.000 people or more at the same time Here's your story I think it is the same deadly cocktail of empty slogans in politics that meet the naked truth of booleans in software that wreaks havoc among government IT programs all around the globe. We should ban the use of booleans in any governement project from now on. Politics can't handle booleans.
robomartin 13y ago
I am trying to not get sucked into political discussions on HN because, well, they are bad for you. There are a wide range of people on HN with different ideas. From the demographics derived from occasional polls quite a few of you are younger and less experienced in life and business. That's not a put-down, just a fact. There's also a layer of HN participant with obvious college-originated indoctrination that is hard to break through until something pulls them out of the cave [0]. And, of course, in general terms, arguing anything with a programmer can be incredibly frustrating --just ask my wife-- because the good ones instantly focus on corner cases, relevant or not, and will argue irrelevant minutiae to death. So I won't critique ACA/Obamacae, the website or the development process. That's pointless. There are people who, just as in the case of various religions, will stick to their belief system no matter what anyone says. It is only when something external and utterly impossible to argue against presents itself that some of these people are willing to reason. So, none of that talk. It's pointless. In thinking about this whole healthcare/website issue while kayaking (great way to unplug and just think) I came to the conclusion that what we are seeing unfold in front of us is of incredible value. In many ways, up until this ACA fiasco became front page news everywhere most US citizens dealt with government incompetence by simply ignoring it. It's one of those out-of-sight-out-of-mind deals. And, while most probably knew that ugly things have been happening behind closed doors and underneath each and every bill passed (regardless of party affiliation) most voters had better things to do than to stop their busy lives to go kayaking and critically think about these matters. Having solitude to just think is a rare commodity these days. What the ACA has done in a major way is open the doors so we can all see the sausage being made. Plain and simple. We can see government incompetence in action. You don't have to be a programmer to recoil a the idea that a bad website that does not work cost taxpayers $600+ million dollars. And, you don't need a degree in computer science to probably guess that the final cost might reach double or triple that. And you don't need to even look at the code or know how it was written to start to understand that it is probably a complete mess and that three hundred million people are going to be forced to provide their most intimate data to this ill-conceived and ill-executed monster. Let's not even discuss the tens of millions who are predicted to suffer significant collateral damage due to the the letter of the law and the intended or unintended consequences. I don't care who you are. Write a law consisting of 30,000 pages and an ever-expanding set of rules. Don't take years to debate, analyze, simulate and evolve it. Then go and try to implement it. Well, there's an absolute certainty of having huge problems. Again, there's no partisan bend here. I am not arguing for or against the contents of the ACA. I am simply stating that something so massive that was created as the ACA ultimately was is almost guaranteed to be a minefield (or have one hidden inside somewhere). Imagine writing 30,000 pages of code in your favorite language and not testing it until you release it to your customers. I know, I know, this isn't a perfect analogy. I am just using it to stress the point that something that large, without suitable testing and "quality control" is bound to be a mess. Yet government operates this way. And we are the ones who get to eat what they cook-up. They certainly don't. Virtually nobody in the very top layers of this country's government has to live by a lot of the laws they pass. The Supreme Court certainly does not. Neither does congress or the office of the President. That should tell you something. They like to talk about the 1%, yet nobody talks about the 0.003%: The top 1,000 people in government who, to some extent, get to live above the law. The easiest way to have people not focus on you is to create focus elsewhere. The ACA, from my admittedly Libertarian, perspective is great in that here we have something that irrefutably exposes how bad it is to put almost anything in the hands of government. This is not what you probably heard in school. This is not what you probably heard your professors proselytize to the captive audience of a classroom. To those who have never lived outside the security of government jobs or jobs where real-world consequences are, well, not real, their view of reality is vastly distorted from "real" reality. And that's what they spew out to our youngsters. And, because they are young and impressionable, they take it in and believe it. Very soon they are repeating the mantra without every having set foot in the real world. Take this as an opportunity to watch and learn what reality looks like. If you are a reasonably accomplished coder you know, without a shred of a doubt, that $600 million dollars to implement the ACA portal --brilliantly or dismally-- is nothing else than obscene, if not criminal. A number in the range of one to, at worst, ten million (still sounds ridiculous) is what I hope most of you would think, no, would know, this should cost to do it right and to do it better. Leave the details of the 30,000 pages and all of the collateral damage they will surely cause out of the equation. Just look at the sausage being made and form an opinion. Now realize this: Everything. Absolutely everything in government. Is done. JUST. LIKE. THIS. You simply don't get to see it every day. Nobody reports on it. And it becomes water under the bridge. Now you can see it. Learn from it. Does this seem sensible to you? [0] http://en.wikipedia.org/wiki/Allegory_of_the_Cave http://en.wikipedia.org/wiki/Allegory_of_the_Cave
LekkoscPiwa 13y ago
I work at "agile" project for a federal government agency. These people need to stop using words meaning of which is a mystery to them. For them Agile means we have a daily meeting with developers now to get status updates and throw even more stuff on them. Right now working on "sprint zero" which exists to 'determine deliveries'. I mean they still operate it as they have always done. They just stole another buzz word that sounds cool. Yes Mr. President we do agile. And because Mr. President doesn't know what agile is, the agency doesn't know what it is, so we can continue business as usual, just make sure that our contractors speak sprints instead of iterations. Stupid.
tunesmith 13y ago
This seems to be a flawed analysis based off of a flawed analysis... First, agile principles don't imply that an agile approach will always succeed where a waterfall approach will fail. It merely asserts that agile increases (not guarantees) the likelihood of success. Second, the article's three sources didn't "proclaim that HealthCare.gov would not have failed if it had just used “modern” software practices, called agile development". The closest that comes is the third source that says, "None of these missteps would have occurred if the contractors had taken a gradual, agile approach." But in general, the articles were arguing that an agile approach would have improved matters, not guaranteed success. Third, agile is a spectrum. People attempt agile within larger waterfall restrictions all the time. It doesn't work very well, but it still might work better than being 100% waterfall. But it doesn't change the fact that this was a project with guaranteed high traffic on launch day and a hard deadline, and a prohibition to deliver a smaller MVP early. That right there is incompatible with the purest forms of agile.
- ternaryoperator 13y ago
  >"It merely asserts that agile increases (not guarantees) the likelihood of success." That's not my understanding. I understand Agile to assert that it's better able to accommodate changes to the project during development.
laughfactory 13y ago
It seems to me that the best virtue of the Agile methodology is simply in advocating for rapid iteration as input from users is received and getting to market sooner so input can begin to be gathered. Neither of these principles where obviously utilized in the development of Healthcare.gov. Anything beyond those two core principles is easily and frequently subverted by business interests outside of those actually doing the development.
rbanffy 13y ago
I almost convulsed when I read "“modern” software practices, called agile development, just like commercial companies do". No company in the club of companies governments hire does it. There is no agile way to cut through all procurement rules, all documentation requirements and all miscellaneous bureaucratic processes (required in order to prevent government funds misappropriation) present in any big project, much less one so publicly examined.
babesh 13y ago
Hire the best people you can. Support them. Get out of the way.
vannevar 13y ago
The author suffers from the common misconception that agile is a process for designing software. It is not. It is a process for implementing a design, while maintaining flexibility to feed implementation experience back up into the design process. It's this feedback that the author's process lacks. FTA: "This [process] will take a few months, but when it is done we will know exactly what we have to build." Most of the time, experience has shown this not to be true; only when you've gotten into the implementation do you know what you 'have to' build (within some reasonable timeframe). That's because large software projects are typically nonlinear complex systems, where small implementation details can sometimes have large design consequences. There is no prohibition in agile from using UML, and the process is certainly not hostile to design. It may be that some people believe that using agile implementation processes relieves them of the responsibility to design their software, and that it's OK for user stories to be generated off the top of their heads without analysis. Those people are mistaken. Nowhere does agile promise to design your system for you.
gesman 13y ago
The most successful, most praised by clients, most robust, most stable and most financially profitable applications I took part at were written by professionals using "waterfall" methods. This were absolutely not a test-driven development but rather strategies i personally developed that resulted in the most stable and bug free deployments. These were security applications, real time data backup and mirroring services. Agile in the wrong hands (or heads) doesn't work and in many cases is just a buzzword.
- hga 13y ago
  Indeed. Especially if you really understand what you're trying to accomplish, waterfall can work just fine. Come to think of it, the most successful "greensfield" project I ever headed was essentially waterfall, in large part because I'd been designing it in my head for several years before, had implemented variations on many components in those years, and had figured out why everyone else who'd tried it had failed to a greater or lessor extent.
pkolaczk 13y ago
Having observed a few big projects failed like that in my country, I can't deny the feeling that "throw big money onto company/institute(s) X to deliver a very big system that will replace the old system on day Y" is an awesome recipe for a spectacular failure, regardless of the software development process and skills of the programmers. Eventually the initial failure would be converted into something usable by spending 10x more money and 10x more time. Contrary, a minority of national-level projects that succeeded here were always almost things that were initially planned to be very small, experimental, with no hard deadline and implemented without a big-bang in media. The biggest distinction is that the small systems tend to be put in production only partially but much earlier, often as an opt-in, working in parallel to the old often paperwork based systems. So there was no huge switch to something new and the risk was pretty small. This is what agile means to me.
memracom 13y ago
So yet another magic bullet doesn't really work. Maybe one day we will learn that it is not the process that matters but the people. Good people will work with a good process that looks a lot like agile, but if you cram that same process down the throats of a random group of developers then you are going to get random results. The details of the methodology don't matter. If you look at what http://semat.org/ http://semat.org/ is proposing for the standard engineering methodology for developing software you will find that it IS agile, but it is not specified down to the nth detail and more importantly, it is extensible and adjustable so that any significantly sized project can build their own compliant methodology and process within the SEMAT framework. That part is so often missing, where the developers themselves discuss how to go about building the software, and management listens to them and defers to the developers because they are software engineering professionals.

The evidence presented in the article is quite weak: "I have seen some of the developer documentation, and it clearly discusses sprints, user stories and incremental testing — all of which are hallmarks of an agile process." For a project with reportedly 50+ subsystems, where the main failures seem to have been integration, that doesn't tell us much either about this project or agile practices.

seivan 13y ago

Or 'agile' lead by a non-engineer. Getting tired of 'agile' or 'XP' getting the blame when in reality they are half-assed versions of some scrum masters cook up.

iyulaev 13y ago

Why don't we take the brilliant hackers and programmers at the NSA and task them to fix and re-build the ACA exchange system? Building a database to track the communications of ever single person in the US and ever foreigner they talk to? Piece of cake. Building a database to track the health insurance policies for a subset of Americans? Over budget, behind schedule, still doesn't really work yet. Priorities.

caryhartline 13y ago
Those are two things that don't have anything to do with each other. You seem to think that every single web service and database works the exact same way.
bmelton 13y ago
Having actually worked with NSA programmers, I can tell you that while they tend to be absolutely brilliant, that doesn't necessarily make them any better at making things "easy to use", "intuitive" or even necessarily "good". When it comes to algorithm design, reverse engineering, or crypto, I know exactly who to call when I need help. When it comes to a layout that works, or in many cases, even just coding up some fairly simple HTML with CSS, my NSA buddies aren't in the top 50 of people I'd call. Yes, they're brilliant. The best of them though, produce websites (if they're capable of coding on the web) that look more like stallman.org than anything like healthcare.org.
- udit99 13y ago
  I see your point, but the primary challenge of delivering large, complex software projects in a reasonable timeframe applies to both endeavours.
monokrome 13y ago
They're too busy stealing your emails, man. They don't have time to spend helping anyone.
andrewfong 13y ago
Not that I'd hire the NSA folks per se, but I'd imagine that the NSA's programmers are full time employees. One thing that might dramatically improve government IT in general is to shift more of it in-house rather than relying on contractors.
jacalata 13y ago
What makes you think that was on time or in budget?
- hga 13y ago
  We don't hear much about it, but the NSA has had some stunning IT failures. Development (heck, they can't open their Utah site because of severe, ruin $100K of hardware each time, power glitches), something major above their collection system(s) failed hard once for more than a few days ... and since they're the No Such Agency, we only hear about some of the very worst ones, I'm sure. Flip side is, doing this in the black budget does allow them to avoid some of the usual government contracting insanities. While that's no gaurentee of success, e.g. look at the NRO's recent expensive stumbles, from the dozen years I spent Inside the Beltway I've gotten a strong impression the intelligence community does a generally much better job than run of the mill open Federal IT.

mhurron 13y ago

Agile is not some magic bullet that guarantees success. To say that something succeeded or failed because of some particular method of project management is is simply absurd.

Maarten88 13y ago

No process can compensate for an unsufficiently qualified/talented team.

jroseattle 13y ago

Healthcare.gov was not built in an agile manner. Sure, some items that are labeled as artifacts of agile-based software development, i.e. user stories, may have been utilized, but that doesn't make for an agile process. Heck, even an agile process doesn't make one agile. To say it failed "despite agile practices" misses the mark. It's like blaming a roof for leaking despite the use of hammers. Process is a tool, and if used properly, it may be a solution. But pixie dust, it is not.

sanskritabelt 13y ago
That's like saying 'oh its not capitalism it's crony capitalism' or similar excuses.

stirno 13y ago

Reacting mostly to the link-baity title.. Corporations act like using the word 'Agile' instantly brings the benefits of flexible planning and proper management to a team. Most people never really interact with a proper agile process - its kind of a unicorn in the development world. The failure is not necessarily the process but the managers not understanding that it isn't a silver bullet. It is often used as an excuse to delay requirements gathering or understanding of requested features from stakeholders. This leads to churn and failure.

koudi 13y ago

Nothing new, really. Methods and tools can help you, but do not guarantee success. Saying that some project failed because they used/didn't use something dosn't make sense to me. In the end, it's all about developers and overall project management.

mkramlich 13y ago

it's almost certainly yet another case of "dumb people in, dumb results out" dumb + agile = agile dumb

YZF 13y ago

Let's face it- Agile has become a codeword for let's micromanage software developers while increasing the pressure on them to deliver. My theory is that Ken Schwaber and Jeff Sutherland came up with Scrum in order to limit management's ability to screw up software development in corporate settings. That was mixed with some observations about the need for iteration, not doing upfront work when it's wasteful (and it's not always wasteful) and visibility that came from the Agile manifesto. However, one lesson I'm constantly re-learning is never underestimate people's ability to screw things up and I think Ken and Jeff underestimated. The whole agile thing has been subverted and if indeed Healthcare.gov was done using "Agile" practices then this is another bit of evidence. We, as software engineers, need to move away from these and figure out some way to qualify what makes good software engineering. You don't see non-technical hospital managers going around telling doctors where to place an incision and you don't see non-technical aerospace management telling engineers what bolt to use in a rocket engine. But for some reason "programming" has been demoted to something that just being "agile" is enough to do. Badly. In his YouTube Scrum talk Ken Schwaber says something about if you have a poor team with bad engineering practices than Scrum will generate crap for you sprint after sprint. Somehow no one listens to that part.

jarrett 13y ago
> figure out some way to qualify what makes good software engineering I doubt we'll ever be able to distill it to a formula. People ran successful software projects decades ago, long before any methodologies were formally described. And they have failing projects now, even when they use a supposedly good formal methodology. Instead of another alternative to waterfall, scrum, or agile, I'd propose the following ingredients for success: Organized people, a good working relationship with plenty of trust, a clear understanding of roles and responsibilities, realistic expectations, and colleagues who honor their agreements with each other. To name a few. All of this is peopleware, not process. Unfortunately, there's no silver bullet to get this stuff right.
- fragsworth 13y ago
  Agreed. These development processes and methodologies are pretty strange, actually. The confidence expressed by their proponents in the face of failure gives it a very cultish vibe, and I get the sense that they're just making things up for the sake of having "an answer". The best answer is probably to get people who are experienced in similar projects to manage your project, and has little to do with the process they choose to use.

djvu9 13y ago

Many software development failures are due to lack of understanding and controlling the complexity given cost, people and time constraints. Processes helps the controlling part but sometimes gives a wrong illusion that things are under control and thus hurts the understanding part: people stop bothering to think. Building large software is really delicate work. You need to have a correct decomposition (thus to reduce the level of complexity) and have a good control of flexibility and complexity trade off on the component level. It is somehow like solving a formula with many parameters. You can not just look at one or two of them.

pasbesoin 13y ago

I'll have to read the article, but at an initial reaction, I think this project was not suffering for a lack of "agile" practice. The didn't need a bunch of "nimble", independent teams. They needed some early, stable requirements. Some early, hard deadlines that they had to hit. And a focus on making the fucking back end work, first. The last thing this project needed was some sort of last minute "agile". Of course, now they'll attempt some variation of it, out of necessity.

MortenK 13y ago

Project went bad? It's just a matter of fixing the process. That's a reasoning that's simple, easy to understand and entirely wrong. Contrary to what the evangelists' tell us, process has small (I'd almost say minimal) influence on software development failure or success. The responsibility is on the lead project manager. He's not been good enough. A failure on a project of this size doesn't just happen overnight. If you just look, you can see it coming from miles away. The solution will always be highly uncomfortable though. You'd have to stand up to powerful people who hold authority over you, fire sub-contractors or employees, significantly reduce scope, move deadlines. Big, high impact actions that won't be popular, hell they might even get you fired. Not a job for the faint of heart for sure. It's depressingly typical of big corporations and organizations, to allow people not capable enough to be put in charge of such large projects. And it's even more depressing knowing tax dollars are being wasted in such huge amounts on relatively (big emphasis) simple software.

gemma 13y ago

Wow, massively flawed premise here. 1. Using user stories to define your requirements does not make your process agile, just like writing a traditional requirements doc does not make your process waterfall. It's not about the tools. If you're not capturing the requirements you need to capture, your process is just broken. 2. Design documents do not magically make your project successful. Agile doesn't like design documents because it typically uses more lightweight artifacts to capture design decisions--but it still captures them! Again, if those decisions are not captured at all, your process is broken. 3. Wat? Asynchronous vs. synchronous is a project design decision; how does this demonstrate the failure of agile methods? Multiple sources ([1], [2], etc.) have reported that the requirements for the system were not known until Spring of this year, and that they were in flux until weeks before the release. That's not a problem that would be fixed with use cases, UML and design documents. [1] http://www.nytimes.com/2013/10/13/us/politics/from-the-start-signs-of-trouble-at-health-portal.html?pagewanted=all http://www.nytimes.com/2013/10/13/us/politics/from-the-start... [2] http://www.huffingtonpost.com/2013/10/22/obamacare-website-programmers_n_4141411.html http://www.huffingtonpost.com/2013/10/22/obamacare-website-p...

coldcode 13y ago

Failure is such a imprecise word. Assuming it works eventually is that failure or success? Success or failure, assuming you can even define them precisely, and blaming the result on some vaguely defined reason is pointless. You may as well blame healthcare.gov's problems on bad pizza.

natrius 13y ago

Isn't a system specified by legislation fundamentally at odds with an agile process? Neither the requirements nor the dates could change without an act of Congress (or in some cases, the President).

ericcumbee 13y ago
Most of the Legislation on finer points of implementation are simply "In a manner that the Secretary shall designate."
- mgkimsal 13y ago
  Doesn't necessarily help on timing though, if the legislation has specific deadlines built in.

fragsworth 13y ago

No, it failed because it was managed by technically inexperienced and/or incompetent people. These people know who they are (the contractors), but they won't admit it. Talk about process all you want, but everyone is just waving their hands around spouting buzz words. A technically experienced (in large scale systems) management team can make sure the project is done right regardless of the process they use.

hga 13y ago
The "technically inexperienced and/or incompetent" managers by now have been well established, CMS on up to the White House, and CMS at least has officially been removed from it's role, replaced by QSSI. What I wonder is how many political types are still telling the new fix-it czar, Jeff Zients, what he can and can't do. I.e. right now all features that can possibly be punting into the future should be. Although he said his #1 priority was to stop sending garbage to insurers, which isn't particularly political at this point, one would hope.

droopybuns 13y ago

I would love to know what version control software was implemented throughout the project. I suspect that the answer is none. But if I'm wrong, it'd be interesting to know what tools they tried to use and where there processes worked, if they ever worked at all.

jroseattle 13y ago
Why would you suspect that version control wasn't in use? Is there an aspect of this project that would be resolved with a specific version control system? These aren't rhetorical questions, I'm genuinely curious why you're asking.
- GhotiFish 13y ago
  A guess at his line of thinking: People, who didn't understand the development processes programmers use, caught a wiff of agile. They mandated agile, but only established the methodology, with no technical direction to back it up. So in the arena of VCS the novice teams didn't use anything (they didn't know better), teams with some experienced devs were ignored, or used their own VCS. a symptom of a more systemic problem.
  - jroseattle 13y ago
    Wow, that's alarming. We need a real post-mortem on this process. The congressional inquisitions aren't getting to the root of the problem. We just need the individual who managed this project to explain, from square one, how the project flow went. (Believe it or not, I intend no sarcasm with this post.)
    - GhotiFish 13y ago
      >(Believe it or not, I intend no sarcasm with this post.) that didn't help T.T
      - deleted 13y ago
        [deleted]

hobs 13y ago

Ignoring the agile point, the whole idea of requesting data from ye olde bespoke applications and data storage systems in a synchronous manner for (even just) a few thousand users seems like it is destined to fail.

ajarmst 13y ago

What? Agile's not magic?

poissonpie 13y ago

A bad workman blames his/her tools. Bad project manager blames his/her process.

hga 13y ago
"Bad project manager blames his/her process." In my experience he blames one or more of his subordinates. "His process" implies ownership of something responsible for the failure. Shifting blame is pretty much by definition a core competency for bad project managers, it's that, get better or get fired.

5vforest 13y ago

Seems like as good of a time as any to bring up Zed Shaw's "Programming, Motherfucker": http://programming-motherfucker.com/ http://programming-motherfucker.com/ We are a community of motherfucking programmers who have been humiliated by software development methodologies for years. We are tired of XP, Scrum, Kanban, Waterfall, Software Craftsmanship (aka XP-Lite) and anything else getting in the way of...Programming, Motherfucker. We are tired of being told we're autistic idiots who need to be manipulated to work in a Forced Pair Programming chain gang without any time to be creative because none of the 10 managers on the project can do... Programming, Motherfucker. We must destroy these methodologies that get in the way of...Programming, Motherfucker.

trentmb 13y ago
He's very annoying to read. EDIT: To clarify, his style is incredibly annoying.
- djillionsmix 13y ago
  Annoying to people who don't like that he's correct
  - deleted 13y ago
    [deleted]
  - tcdent 13y ago
    I agree with his thinking, but his presentation is just too in-your-face. It can be hard to find a balance when communicating something you're passionate about.
  - rbanffy 13y ago
    Actually, no. He isn't right. Any significant programming project involves more than one person. Being arrogant is probably the most serious flaw in a programmer. You will be wrong. You will under or overestimate something important. You do have blind spots. Most important, you are not as smart as you think you are. If you are not careful, your mistakes will ruin your project. They'll delay you or create liabilities you'll have to deal with. Methodologies exist to help you manage yourself, not for your managers to better manage you (although they can be misused for that). You don't have to feel "humiliated" by them (neither methodologies nor managers). You don't have to be insecure.

duhballs 13y ago

guess which two parts actually work?

dep_b 13y ago

As a user I would like to be able to use the Healthcare.gov with at least 50.000 people or more at the same time Here's your story I think it is the same deadly cocktail of empty slogans in politics that meet the naked truth of booleans in software that wreaks havoc among government IT programs all around the globe. We should ban the use of booleans in any governement project from now on. Politics can't handle booleans.

robomartin 13y ago

I am trying to not get sucked into political discussions on HN because, well, they are bad for you. There are a wide range of people on HN with different ideas. From the demographics derived from occasional polls quite a few of you are younger and less experienced in life and business. That's not a put-down, just a fact. There's also a layer of HN participant with obvious college-originated indoctrination that is hard to break through until something pulls them out of the cave [0]. And, of course, in general terms, arguing anything with a programmer can be incredibly frustrating --just ask my wife-- because the good ones instantly focus on corner cases, relevant or not, and will argue irrelevant minutiae to death. So I won't critique ACA/Obamacae, the website or the development process. That's pointless. There are people who, just as in the case of various religions, will stick to their belief system no matter what anyone says. It is only when something external and utterly impossible to argue against presents itself that some of these people are willing to reason. So, none of that talk. It's pointless. In thinking about this whole healthcare/website issue while kayaking (great way to unplug and just think) I came to the conclusion that what we are seeing unfold in front of us is of incredible value. In many ways, up until this ACA fiasco became front page news everywhere most US citizens dealt with government incompetence by simply ignoring it. It's one of those out-of-sight-out-of-mind deals. And, while most probably knew that ugly things have been happening behind closed doors and underneath each and every bill passed (regardless of party affiliation) most voters had better things to do than to stop their busy lives to go kayaking and critically think about these matters. Having solitude to just think is a rare commodity these days. What the ACA has done in a major way is open the doors so we can all see the sausage being made. Plain and simple. We can see government incompetence in action. You don't have to be a programmer to recoil a the idea that a bad website that does not work cost taxpayers $600+ million dollars. And, you don't need a degree in computer science to probably guess that the final cost might reach double or triple that. And you don't need to even look at the code or know how it was written to start to understand that it is probably a complete mess and that three hundred million people are going to be forced to provide their most intimate data to this ill-conceived and ill-executed monster. Let's not even discuss the tens of millions who are predicted to suffer significant collateral damage due to the the letter of the law and the intended or unintended consequences. I don't care who you are. Write a law consisting of 30,000 pages and an ever-expanding set of rules. Don't take years to debate, analyze, simulate and evolve it. Then go and try to implement it. Well, there's an absolute certainty of having huge problems. Again, there's no partisan bend here. I am not arguing for or against the contents of the ACA. I am simply stating that something so massive that was created as the ACA ultimately was is almost guaranteed to be a minefield (or have one hidden inside somewhere). Imagine writing 30,000 pages of code in your favorite language and not testing it until you release it to your customers. I know, I know, this isn't a perfect analogy. I am just using it to stress the point that something that large, without suitable testing and "quality control" is bound to be a mess. Yet government operates this way. And we are the ones who get to eat what they cook-up. They certainly don't. Virtually nobody in the very top layers of this country's government has to live by a lot of the laws they pass. The Supreme Court certainly does not. Neither does congress or the office of the President. That should tell you something. They like to talk about the 1%, yet nobody talks about the 0.003%: The top 1,000 people in government who, to some extent, get to live above the law. The easiest way to have people not focus on you is to create focus elsewhere. The ACA, from my admittedly Libertarian, perspective is great in that here we have something that irrefutably exposes how bad it is to put almost anything in the hands of government. This is not what you probably heard in school. This is not what you probably heard your professors proselytize to the captive audience of a classroom. To those who have never lived outside the security of government jobs or jobs where real-world consequences are, well, not real, their view of reality is vastly distorted from "real" reality. And that's what they spew out to our youngsters. And, because they are young and impressionable, they take it in and believe it. Very soon they are repeating the mantra without every having set foot in the real world. Take this as an opportunity to watch and learn what reality looks like. If you are a reasonably accomplished coder you know, without a shred of a doubt, that $600 million dollars to implement the ACA portal --brilliantly or dismally-- is nothing else than obscene, if not criminal. A number in the range of one to, at worst, ten million (still sounds ridiculous) is what I hope most of you would think, no, would know, this should cost to do it right and to do it better. Leave the details of the 30,000 pages and all of the collateral damage they will surely cause out of the equation. Just look at the sausage being made and form an opinion. Now realize this: Everything. Absolutely everything in government. Is done. JUST. LIKE. THIS. You simply don't get to see it every day. Nobody reports on it. And it becomes water under the bridge. Now you can see it. Learn from it. Does this seem sensible to you? [0] http://en.wikipedia.org/wiki/Allegory_of_the_Cave http://en.wikipedia.org/wiki/Allegory_of_the_Cave

LekkoscPiwa 13y ago

I work at "agile" project for a federal government agency. These people need to stop using words meaning of which is a mystery to them. For them Agile means we have a daily meeting with developers now to get status updates and throw even more stuff on them. Right now working on "sprint zero" which exists to 'determine deliveries'. I mean they still operate it as they have always done. They just stole another buzz word that sounds cool. Yes Mr. President we do agile. And because Mr. President doesn't know what agile is, the agency doesn't know what it is, so we can continue business as usual, just make sure that our contractors speak sprints instead of iterations. Stupid.

tunesmith 13y ago

This seems to be a flawed analysis based off of a flawed analysis... First, agile principles don't imply that an agile approach will always succeed where a waterfall approach will fail. It merely asserts that agile increases (not guarantees) the likelihood of success. Second, the article's three sources didn't "proclaim that HealthCare.gov would not have failed if it had just used “modern” software practices, called agile development". The closest that comes is the third source that says, "None of these missteps would have occurred if the contractors had taken a gradual, agile approach." But in general, the articles were arguing that an agile approach would have improved matters, not guaranteed success. Third, agile is a spectrum. People attempt agile within larger waterfall restrictions all the time. It doesn't work very well, but it still might work better than being 100% waterfall. But it doesn't change the fact that this was a project with guaranteed high traffic on launch day and a hard deadline, and a prohibition to deliver a smaller MVP early. That right there is incompatible with the purest forms of agile.

ternaryoperator 13y ago
>"It merely asserts that agile increases (not guarantees) the likelihood of success." That's not my understanding. I understand Agile to assert that it's better able to accommodate changes to the project during development.

laughfactory 13y ago

It seems to me that the best virtue of the Agile methodology is simply in advocating for rapid iteration as input from users is received and getting to market sooner so input can begin to be gathered. Neither of these principles where obviously utilized in the development of Healthcare.gov. Anything beyond those two core principles is easily and frequently subverted by business interests outside of those actually doing the development.

rbanffy 13y ago

I almost convulsed when I read "“modern” software practices, called agile development, just like commercial companies do". No company in the club of companies governments hire does it. There is no agile way to cut through all procurement rules, all documentation requirements and all miscellaneous bureaucratic processes (required in order to prevent government funds misappropriation) present in any big project, much less one so publicly examined.

babesh 13y ago

Hire the best people you can. Support them. Get out of the way.

vannevar 13y ago

The author suffers from the common misconception that agile is a process for designing software. It is not. It is a process for implementing a design, while maintaining flexibility to feed implementation experience back up into the design process. It's this feedback that the author's process lacks. FTA: "This [process] will take a few months, but when it is done we will know exactly what we have to build." Most of the time, experience has shown this not to be true; only when you've gotten into the implementation do you know what you 'have to' build (within some reasonable timeframe). That's because large software projects are typically nonlinear complex systems, where small implementation details can sometimes have large design consequences. There is no prohibition in agile from using UML, and the process is certainly not hostile to design. It may be that some people believe that using agile implementation processes relieves them of the responsibility to design their software, and that it's OK for user stories to be generated off the top of their heads without analysis. Those people are mistaken. Nowhere does agile promise to design your system for you.

gesman 13y ago

The most successful, most praised by clients, most robust, most stable and most financially profitable applications I took part at were written by professionals using "waterfall" methods. This were absolutely not a test-driven development but rather strategies i personally developed that resulted in the most stable and bug free deployments. These were security applications, real time data backup and mirroring services. Agile in the wrong hands (or heads) doesn't work and in many cases is just a buzzword.

hga 13y ago
Indeed. Especially if you really understand what you're trying to accomplish, waterfall can work just fine. Come to think of it, the most successful "greensfield" project I ever headed was essentially waterfall, in large part because I'd been designing it in my head for several years before, had implemented variations on many components in those years, and had figured out why everyone else who'd tried it had failed to a greater or lessor extent.

pkolaczk 13y ago

Having observed a few big projects failed like that in my country, I can't deny the feeling that "throw big money onto company/institute(s) X to deliver a very big system that will replace the old system on day Y" is an awesome recipe for a spectacular failure, regardless of the software development process and skills of the programmers. Eventually the initial failure would be converted into something usable by spending 10x more money and 10x more time. Contrary, a minority of national-level projects that succeeded here were always almost things that were initially planned to be very small, experimental, with no hard deadline and implemented without a big-bang in media. The biggest distinction is that the small systems tend to be put in production only partially but much earlier, often as an opt-in, working in parallel to the old often paperwork based systems. So there was no huge switch to something new and the risk was pretty small. This is what agile means to me.

memracom 13y ago

So yet another magic bullet doesn't really work. Maybe one day we will learn that it is not the process that matters but the people. Good people will work with a good process that looks a lot like agile, but if you cram that same process down the throats of a random group of developers then you are going to get random results. The details of the methodology don't matter. If you look at what http://semat.org/ http://semat.org/ is proposing for the standard engineering methodology for developing software you will find that it IS agile, but it is not specified down to the nth detail and more importantly, it is extensible and adjustable so that any significantly sized project can build their own compliant methodology and process within the SEMAT framework. That part is so often missing, where the developers themselves discuss how to go about building the software, and management listens to them and defers to the developers because they are software engineering professionals.