6 ms·
Why US government IT fails so hard, so often
- DanielBMarkham 13y agoI just got through speaking in DC about Agile, startups, and functional programming, mostly to an audience that is intimately familiar with government work. It's really bad when you try to be diplomatic about how bad things are fucked up and get open laughter from the crowd. There's no BSing people who live this everyday. Talking with folks afterwards, I heard about systems that I could have rewritten in the late 90s in a few months that are still in the process of being rewritten, 15 years later. After literally hundreds of people have worked on them costing the taxpayer millions of dollars. And these are not unique or oddball edge cases. Stuff like that is standard operating procedure. I can't impress on the HN audience enough how terrible federal IT is overall. (Aside from some really great outliers). There's a vast and increasingly large gap between political rhetoric and the reality on the ground. This has been true for at least the last 20 years. I take issue with some of the details the supporting argument makes here, but that's mostly a quibble. The author is definitely headed in the right direction. I'm a veteran of several federal IT programs, and a real veteran. I have seen some really good federal teams, but never a good federal team of teams. It all falls apart once you get past a few people and start throwing some kind of structure and management in place. I understand that Agile is becoming required for some federal work, but I'll bet you a thousand bucks this kind of "Agile" isn't going to be the kind that the rest of us use. Think big tools, big management, lots of stupid measurements, and no accountability. All with the word "Agile" stamped on the front of it. ...The bottom line is that federal IT programs' success is measured by things that have nothing to do with how successful they are or by the metrics most of the world uses. While the business world (and Web companies in particular) now monitor user experience and productivity as a metric for IT success, the government keeps throwing out numbers that mask the truth: the only people who would use their systems are the ones that are forced to... Yep.
- Jtsummers 13y ago"Agile" when federalized will probably turn out like "Lean". Lean in manufacturing and similar processes, when done right, is a good thing. However, the government version of it is almost a complete perversion of the intent. Instead of bottom up, employee-driven changes, it's top down based strictly on metrics and the changes typically effected so that whoever's currently in charge can say, "I made <process> 10% more efficient by reducing the number of wasted <things>." Employees resist the changed processes/taskings because they fear a loss of employment, and are kept out of the loop when the changes are being devised. Federalized Agile would likely be similar.
- DanielBMarkham 13y agoYep. We saw this same thing with source control. Who doesn't like source control? But the government, in an effort to make sure there was absolutely no risk, mistakes, or accountability associated with software development, started adding more and more mandatory forms that had to be completed each time a developer checked something back in. This created such a mess for developers that it wasn't unusual for one check in to take hours to accomplish. You'd read a book while the system tried to accept your code changes. Agile is all about failing fast and learning quickly. Most contractual work for the government does not include the possibility of failure. At all. In fact, the way the contracts are written is such that the contractor takes on all responsibility for everything, even things there's no way they could control. In response to this idiotic situation, contractors start bidding at crazy low rates on these things, say 40 bucks an hour, then actually make their money on changes. So it's always in the contractors interest to capture and execute changes, while it's the government is trying to get the world delivered to them on a platter for five bucks. People use the term "perverse incentives", but this is like that taken to the next level. It's a multi-layered thing. It's total systemic failure. We'd call it systemic collapse, but the money keeps flowing in anyway, so that term isn't applicable, although it certainly feels like it. This kind of distrust doesn't make for anything but a living hell for everybody, government employees and contractors alike. So, with tons of rules and structures in place, the way things really work is through relationships. People who are easy to get along with and helpful get more government work. Both sides of the fence can't control the process, and nobody rocks the boat and everybody gets a piece of the pie. You end up from the government side with some sort of informal and unspoken version of I don't care if you make a lot of money, just make sure me and my friends look good Also it'd be nice to pick up a little lucrative consulting work once I've put in my 20 and can start double-dipping. Your average developer has absolutely no idea about this. If they've worked in IT in federal work, they probably have horror stories, but the more you see the worse it gets. Oddly enough, in the best cases you end up with people who really like working together, are motivated, but spend most of their time fighting a Sisyphean fight against the idiocy of the system.
- dragonwriter 13y ago> I understand that Agile is becoming required for some federal work, but I'll bet you a thousand bucks this kind of "Agile" isn't going to be the kind that the rest of us use. Agile can't be mandated; you can mandate methodologies (e.g., Scrum), but by the very act of mandating a methodology you are not agile. Agile is a culture, and it is a culture of flexibility in methodology with focus on the goal. And its a culture that is very much contrary to the entrenched culture of most large bureaucracies, government or private (and, in government, often in tension with binding mandates where the authority to make changes to the mandate is very distant from the locus at which one would normally expect efforts to adopt development approaches.)
- jfb 13y agoIt all falls apart once you get past a few people and start throwing some kind of structure and management in place. That's hardly confined to Federal IT projects, however.
- kilroy123 13y agoIsn't this how any government project works? IT, construction, etc. Sees like the government is terrible at building anything, especially when contractors are used to do the job. Probably why the government is so big and expense. I can only imagine what that $600+ billion gets us for the military. -_- http://en.wikipedia.org/wiki/Military_budget_of_the_United_States http://en.wikipedia.org/wiki/Military_budget_of_the_United_S...
- deleted 13y ago[deleted]
- iskander 13y ago> Sees like the government is terrible at building anything, especially when contractors are used to do the job. Strange to contrast that with a tremendously active and efficient era of public works from the 1890s through the 1950s. Just thinking of New York City, I suspect that the total cost of all the bridges, tunnels and highways (adjusted for inflation) might add up to a lot less than the 800 billion USD that disappeared into our recent stimulus bill. What changed? edit: Apparently the Brooklyn Bridge cost ~15 million USD to originally construct, which now would be somewhere between 300 million to 400 million USD (http://historical.whatitcosts.com/facts-brooklyn-bridge.htm http://historical.whatitcosts.com/facts-brooklyn-bridge.htm). Even though these are rough figures and inflation over large time horizons isn't a perfect measure, I'm still astounded. The government's totally broken healthcare website cost more to build than the Brooklyn Bridge.
- Jtsummers 13y agoThe relationship between government and business, the much more short term approach to making money for businesses and individuals, a loss of the concept of "public good" or "common good".
- jerf 13y agoI'm sorry, I don't really see how any of those explain the government being granted money (in staggering quantities), then spending it, then having nothing. Are you implying that because businesses have a short-term approach to making money that therefore, the government is forced to give money to unproductive contractors? Are you saying that because "the people" don't have a concept of "public good" that when the government bids something out they can't help but use a bidding approach that virtually guarantees failure? Vague platitudes about society at large aren't crashing government projects. Society at large aren't really involved in them, after all. Something specific between the interaction of the government and the government-chosen contractors is probably the problem. In a way, you're probably proving far more than you mean, accidentally; if the problem really is "society at large", then there is no solution and these large projects can't work, and therefore the most effective thing the government could possibly do is stop trying and wasting money. (A project with a prerequisite of "change the larger culture's attitudes" is simply a guaranteed failure, after all.) I suspect that's not the point you're going for. You probably really want to be arguing for a much more local reason.
- tjaerv 13y agoSurprised they didn't mention the all-time fiasco of the FBI's disastrous $170M Virtual Case File project: http://en.wikipedia.org/wiki/Virtual_Case_File http://en.wikipedia.org/wiki/Virtual_Case_File
- DanielBMarkham 13y agoThat's just small potatoes. The really big stuff is almost invisible. That's because the liability is too large, but every now and then you can see how things are going. Check out the IRS modernization efforts. (The quote and link are probably 5-10 years old. Haven't heard much about it one way or the other lately) ...The IRS has failed twice to modernize its system, in the 1970’s when taxpayer privacy issues surfaced and in the mid-1990’s when the IRS spent ten years and $2 billion on the project and had little to show for it. The current project launched in 1999 with a budget of $8 billion. The project is late and over-budget; problems have been well publicized after the release of the IRS Oversight Board investigation... http://www.kellogg.northwestern.edu/student/courses/tech914/summer2004/projectfailures/projectfailures/irs2.htm http://www.kellogg.northwestern.edu/student/courses/tech914/... AFAIK, the IRS dropped $10 Billion in IT money in just over a decade with little to nothing to show for it. I used to joke around and say heck, they could have paid me 2 Billion and I would have also provided them nothing, but look at the money they would have saved!
- tjaerv 13y agoThanks for that, I had no idea.
- DanielBMarkham 13y agoThe general rule is: the larger, older, and more politicized the agency is, the more awful the technology development. So NSA, which is fairly young, stays out of the political eye (until recently), and small? They do some really cool stuff. But the older, more controversial agencies, covering things like immigration, defense, taxes, health, and so on? Not good.
- cjoh 13y agoThis is all right, but a little vague. Here are some more specific problems: http://blog.dobt.co/post/63655420372/how-healthcare-gov-went-wrong http://blog.dobt.co/post/63655420372/how-healthcare-gov-went...
- GVIrish 13y agoThis is an excellent analysis. Particularly the point about setting the go-live date long before the scope of the project was understood, not to mention the fact that there's no way they could have had an open bidding process AND meet the deadline. In some ways it's a miracle this went live on-time, flaws and all. However, I suspect the true extent of the flaws with Healthcare.gov and the exchanges is not understood yet. If anything good comes out of this, I hope it's that the government finally starts paying attention to the woeful state of government IT projects.
- deleted 13y ago[deleted]
- uptown 13y agoYet if you look at what's been revealed about the NSA's capabilities, they're excelling in some areas of technological execution.
- EpicEng 13y agoA new version of Windows Server doesn't address the problem of loading 64 JE files and 11 CSS files on every page. There is some programming ineptitude at play as well, though I realize that the article was focusing specifically on the new ObamaCare site.
- acdha 13y agoThat's also a minor side issue: it'll cost you a sub-1-second page load time but it doesn't cause the kinds of problems people are talking about where the backend components are overloaded to the point of failure. The size of healthcare.gov is actually well under the average: http://www.webpagetest.org/result/131010_ZG_X5C/1/details/ http://www.webpagetest.org/result/131010_ZG_X5C/1/details/ (see http://www.webperformancetoday.com/2013/06/05/web-page-growth-2010-2013/ http://www.webperformancetoday.com/2013/06/05/web-page-growt...) For comparison, look at WashingtonPost.com: it uses considerably more data and more total requests and yet the page loads much faster: http://www.webpagetest.org/result/131010_GG_XDK/ http://www.webpagetest.org/result/131010_GG_XDK/ If you look carefully, though, you'll see a) that they aren't setting Cache-Control headers and are setting ETags (both performance killers) and b) there's this weird staggered response behaviour which indicates something is seriously wrong. It seems to handle much better in Chrome - 3s vs 70! - so presumably there's some interaction with the browser going on as well: http://www.webpagetest.org/result/131010_ZW_XD6/1/details/ http://www.webpagetest.org/result/131010_ZW_XD6/1/details/
- mmagin 13y agoAt an individual level: How many programmers who are good enough to get a job that pays well in the private sector want to take a job (either in the government or for its contractor) with all that bureaucratic overhead?
- vonmoltke 13y agoMore than I think most people here realize. Some people find particular, unique problems fascinating. Others have a sense of duty that outweighs material or personal concerns. Others just don't know better.
- ihsw 13y agoLarge organizations need homogeneous computer systems, so it stands to reason that centralized configuration management would be a perfect fit. So why hasn't the likes of puppet and chef caught on? Genuinely curious.
- jl6 13y agoLarge orgs want guarantees and certainty, and so typically look for software support from other large organizations. Easier to hold IBM to account than a small independent vendor who may have gone out of business by the time you need them. Yes, this is an expensive approach. Who is selling support for Puppet and Chef?
- archangel_one 13y agoAs per the article, they're very (very very) slowly migrating from Windows XP to Windows 7. I suspect Puppet and Chef are not even on their radar at this point.
- eitally 13y agoThe reason it fails is because of articles like this disincentivizing the talent they need from being interested in working for either a) .gov/.mil or b) government contractors. Additionally, the contracting process being divorced from the project execution weighs heavily.
- Jtsummers 13y agoArticles like this don't disincentivize a lot of people, and burying it would just let them be lured in and find out the hard way how bad it is. Articles like this air a problem that needs to be corrected. What disincentivizes talented (specifically young talent) people is the ageism inherent in the system and its perverse method of rewarding people (specifically talking civil service here, contracting's another beast). Unlike the ageism you hear about in the occasional article about Silicon Valley, its the young that have a hard time in civil service. Positions aren't available because too many people are waiting to retire (for various reasons). When they do get in there are few leadership roles open for advancement (see previous sentence). Their pay is strictly time and not merit based (well, not 100%, but NSPS is gone and only a handful of places use its equivalent of merit based pay increase). Career fields are disincentivized, for instance engineers are treated better than computer scientists and mathematicians even in offices where the latter make more sense (bonuses, faster promotion schedules for trainees, pay differential for the degree). Lack of proper training programs (this is office by office, so not sure how universal the problem is, but it's pretty shabby in a lot of places). People are expected to be hired in and, within months, become experts on arcane and outdated systems, with no support because the actual expert has just retired (that's whey there was an opening). People leave quickly, similar to the Teach for America article earlier in the day, because they burn out or realize there's no where to go, neither up nor down, just steady. On the other hand, in some regions older, talented folks can get a good deal. They can come in as a high grade GS employee with a middling step, automatically enrolled in a decent 401k (5% matching), good insurance (less critical with Obamacare?), and, after a couple years, effective tenure so they have stability which they might value for their family.
- richieb 13y agoNot much stability with the current Congress.
- GVIrish 13y agoThis article is only touching on a small portion of why government IT is terrible. People like to blame contractors but I think a large part of the problem is that the federal employees who are responsible for these big projects are rarely held accountable for failure. In fact, the lack of accountability for federal employees is why so much work was farmed out to contractors in the first place. Then there's the fact that a lot of the decisions in government IT are made by people who have little understanding of how successful software gets built. People don't understand the importance of defining the scope of the project properly. Or you get people that change requirements willy nilly. Or you find people that are just trying to protect their little fiefdom so you end up with wasteful duplication of effort or withholding of vital information. The Ars article indirectly touches on the fiefdom issue. When you have dysfunctional IT departments or applications, often times instead of fixing the dysfunction, someone just creates their own little separate IT effort to get around it. So the VIP's might get something functional while the peons get classic ASP & Access. What I find very interesting about the Healthcare.gov problem is that this may be the first time that a major federal IT project failure has major political consequences for elected officials. If Healthcare.gov ends up being a crash and burn failure, it could result in Republicans gaining enough power in the next election to overturn the whole ACA.
- RougeFemme 13y agoI don't think lack of accountability is what pushed the work out to contractors. And if that is the case, does that mean the accountability "stick" got pushed out to the contractors. I think work got pushed out to contractors from a desire to out-source and because contractors, for various reasons, generally do a much better job of staying abreast of best practices than the government employees. And I say this as someone who has worked on both sides of the fence. I agree that requirements are a big issue. In addition to the requirements issues you mention, there is the problem with the "business line owners" attempting to unnecessarily dictate implementation details, poorly defining the functional requirements in the first place and ignoring interfacing systems.
- GVIrish 13y agoWhere lack of accountability comes is the idea that contractors can be fired if they don't do a good job, whereas it is much more difficult to fire a full-time employee. So rather than solving that problem, most of the government's software development is farmed out to contractors. The other reasoning for contractors is that contractors don't get retirement benefits that FTE's do, so in theory that saves the govt money but in practice it probably doesn't, given that contractor bill rates are usually 200-300% or in excess of what FTE salaries are. Not to say that FTE's are great or anything, contractors do tend to me more up-to-date on technology but I think that is a result of the government employee hiring practices and compensation scale. Yeah I've seen a lot of "business line owner" issues like the ones you've mentioned. Another common one is that software is often built to mirror dysfunctional and wasteful business processes. So you either get software that is needlessly complex, or you build the software then have to re-do it because people realize how bad the underlying business process was.
- djhworld 13y agoHere in the UK most government contracts just go to the big enterprise shops (think Accenture, IBM, Logica, Cap Gemini, ATOS etc) who manage to negotiate fairly hefty long term deals and charge a healthy commission for taking them on and running them for years. This is why you run into legacy codebases that are 10-20 years old and will keep on going.
- deleted 13y ago[deleted]
- specialist 13y agoI've done IT at both megacorps and govt. Frankly, I don't see a difference. If anything, govt work has greater transparency and accountability. Contributing factors are, but not limited to: - scale - complexity - legacy / installed base - laws, regulations, rules - organizational processes (workflows) - quality of developers - resources - politics and agendas Just trying to support all the entrenched chaotic mutant use cases and edge cases will wreck any project plan. I much prefer working at younger, smaller companies. There's fewer roadblocks. But the healthcare and other benefits working for government rock, so here I am.