11 ms·

List of failed and over-budget custom software projects

154 points by monort 11y ago

jamessb 11y ago
IEEE Spectrum had a great series of interactive information graphics on 'Lessons from a decade of IT failures' [0]. In particular, "The Staggering Impact of IT Systems Gone Wrong" [1] and "Monuments to failure" [2] are good lists of failures, many of which are not included on Wikipedia's list. [0]: http://spectrum.ieee.org/static/lessons-from-a-decade-of-it-failures http://spectrum.ieee.org/static/lessons-from-a-decade-of-it-... [1]: http://spectrum.ieee.org/static/the-staggering-impact-of-it-systems-gone-wrong http://spectrum.ieee.org/static/the-staggering-impact-of-it-... [2]: http://spectrum.ieee.org/static/monuments-to-failure http://spectrum.ieee.org/static/monuments-to-failure
- thrownaway2424 11y ago
  I don't know if those are all in the same category. For example the Army cancelled Future Combat System because it didn't work. Would the project have been considered a "success" if they had put it into production instead? For my money, a real failure is like the giant BART fiasco in the 90s where they spent 80 million dollars on a new train control system that simply never existed, at all. They had to sue General Electric to get their money back.
  - AnimalMuppet 11y ago
    > I don't know if those are all in the same category. For example the Army cancelled Future Combat System because it didn't work. Would the project have been considered a "success" if they had put it into production instead? If it didn't work? Hopefully not. But military stuff is a bit different. Military systems tend to try to push the state of the art as far as possible, for as much edge in combat as possible. That makes for much higher chances of failure.
cthulhujr 11y ago
>This article contains one or more incomplete lists which may never be able to satisfy particular standards for completeness Well isn't that ironic.
- SmellyGeekBoy 11y ago
  Don't you think?
kozukumi 11y ago
The modern NHS system in the UK was the biggest fuck up I have ever seen. I worked with people who also worked on the project. They are seriously amazing people but apparently all the layers of management, red tape and politics was, according to my friends, the worst they have ever experienced. Two friends were so traumatised by the experience they ended up taking 2 years off work to travel/recover. The sad thing is it shouldn't have been such a difficult thing to develop. Complex sure but no more complex than many other similar systems. It was destroyed by too much meddling from the government adding, removing and changing the spec every other day and many, many managers just finding non-sense jobs that were totally pointless. It is shocking it went so over budget and was allowed to continue doing so for so, so long.
- teh_klev 11y ago
  Which NHS body in the UK are you talking about? There are four separate healthcare services in the UK which operate and are funded separately: NHS Scotland NHS England NHS Wales HSC Northern Ireland
  - noir_lord 11y ago
    NHS England - The National Programme for IT. It is as far as I've seen the 'greatest' fuckup in UK development history, 12.something billion down the drain, massive overspending, they basically made every mistake you could make short of setting the office on fire.
    - aledalgrande 11y ago
      They should make mismanagement of a government project become a felony.
      - dragonwriter 11y ago
        > They should make mismanagement of a government project become a felony. This would involve either: (1) Clear and unambiguous definition of "mismanagement" involving distant-from-the-specific-project procedural mandates, which would reduce "mismanagement" by the definition applied but increase (and mandate) what would be functionally mismanagement for specific real projects (lots of government regulation designed to prevent mismanagement all around the world, while usually not specifically felony criminal rules, works this way now), or (2) Being so ambiguous and vague so as to provide a basis for arbitrary prosecution, such that it would make sure no one wanted to touch management of a government project at all (at least -- assuming this was the US or one with similar fundamental rules as to what can be an enforceable criminal law -- until it was inevitably ruled unenforceable as impermissibly vague.)
tyingq 11y ago
Of course, there's a level of disclosure for Government run projects that means their blunders get exposed more easily. I'm quite sure there are some spectacular failures in the corporate world that won't make the list.
- deleted 11y ago
  [deleted]
- pascalmemories 11y ago
  Sadly, no, they are not more easily exposed. Most are quietly swept under the carpet and no-one (other than those working on it) ever know it what a crazy mess it was. It gets classed as something else, is re-tagged as a proof of concept (so was never supposed to deliver anything other than improve understanding of an issue) etc. The ones you hear about are the ones the government ministers foolishly chose to boast about in some answer in parliament or announce in some TV interview. Normally that's when consultants pounce on it and offer to help make it succeed [i.e. vampire all the money away in insane fees]. A minister announcing it is practically the death-knell of a project. The ones which succeed generally keep publicity and announcements to an absolute minimum and keep tight control. That's way more difficult than you imagine on a government project as civil servants go hunting for projects to buddy up to in order to pad their promotion cv. If your project budget is more than the last one they screwed, they're all over it so they can boast about the incremental increase in projects they contributed to - a major promotion point (as no-one checks if the project succeeded, the promotion is hinged on how much public money it pissed away). I was ecstatic to leave UK Gov projects behind.
- DubiousPusher 11y ago
  There's also an asymmetry in the way we think about public and private projects. When a piece of private software flops, it still exists as a product and however bad, often offsets some of its cost in sales. Since most public software is offered for free, no cost is visibly recouped. Healthcare.gov is actually a great example. It was a colossal failure at first. A quantity easily measured in dollars spent. However, it has largely functioned in the years since but measuring its positive economic impact is hard because the service is delivered free. None if that money has been reclaimed but it certainly has had some positive economic value.
mrweasel 11y ago
I should sit down and add the failed IT projects of the Danish government, but I don't have that kind of time. Our government is notorious for initiating software projects that ends in almost predictable failure. The latest project to fail was a system for the Danish Tax Office. It was suppose to automatically collect debt owed by citizens and companies.It's not entirely certain how that was suppose to work and it has been described as a utopia project with an unprecedented expectation of what kind of problem computers are capable of handling. The specification where 9000 pages long, tax law not included, and some how failing to supply a sufficient amount of actual requirements. It failed, of cause, costing almost $150.000.000 (DKK 1.000.000.0000), not including the 78 Billion Danish Kroner the tax office haven't been able to collect due the system not being delivered on time and generally not working.
- busterarm 11y ago
  Doesn't the government there (and most companies for that matter) contract IBM for big software projects? You know the old adage "nobody gets fired for buying IBM". I have a friend that's partner at a consultancy in Copenhagen and he says IBM is often their competition.
  - mrweasel 11y ago
    Mostly CSC actually. They have contract for systems that should have been part of licensing rounds, which would have allowed other to take over support, but no other where ever allowed to bid. The reason: CSC documented the systems so poorly that no one else will be able to handle the support. The government knows that it's illegal, yet CSCs contracts have automatically been renewed, multiple times. But yes, IBM have a large number of contracts as well. CSC just seems to be behind the worst projects. I suspect they bid for project and just focus on delivering exactly what's in the contract, knowing that it will never work. Either that or they are even more incompetent that anyone could imagine.
    - busterarm 11y ago
      Given my experience with the managers of CSC here in the States, that sounds exactly like what I'd expect. From my discussions with that friend I mentioned though it sounds like the attitude is changing and there's more work going to smaller shops like his (though friendships with government officials seems to help loads when it comes to securing work). Honestly it sounds like a really exciting environment to work in when we talk about it and I wish I had the time to really learn Danish so I could have a decent chance of securing good work out there.
orf 11y ago
Can someone explain to me how the fuck my government can spend 15.8bn[1] on a software project? We could build 5 Queen Elizabeth class aircraft carriers for that. Five! For the cost of a bloated Java app! Seriously, someone fill me in please. How can a software project possibly ever cost more than a billion pounds? You could employ 20,000 software engineers full time for £50K ($70k) a year for that much, and I expect the number of engineers working on Universal Credit is much lower than that and paid less. 1. https://en.wikipedia.org/wiki/Universal_Credit#Costs https://en.wikipedia.org/wiki/Universal_Credit#Costs
- detaro 11y ago
  They don't employ the developers. The development is done by companies that might pay the developers that much, but bill the client more per hour and bill the client more time. And of course you can't forget project managers, architects and other consultants that are more expensive. And have a lot of meetings. And hardware, and oracle databases/SAP licenses, and support contracts for those, and audits, and extra charges for change requests, and travel budgets, and... Just because one team of developers could do something in 200 man hours doesn't mean you can't spend and bill 2000 for it. And that's just development. Data migrations and user training/support are going to be massive issues if you replace or newly introduce a widely used system.
  - organsnyder 11y ago
    > Data migrations and user training/support are going to be massive issues if you replace or newly introduce a widely used system. Yup. My employer is undertaking a massive IT project (many many many millions of USD), and it's my understanding that training accounts for over half of the budget.
  - orf 11y ago
    > Just because one team of developers could do something in 200 man hours doesn't mean you can't spend and bill 2000 for it. It doesn't mean you should pay, or plan, to spend 20,000 man hours in meetings and devote huge chunks of your budget to expensive licenses for something that could take 200 man hours, especially when it's taxpayers money. Sure there are unavoidable overheads but why does our government keep splashing billions on them and failing, with little accountability? Training is going to be expensive, sure, but perhaps plow millions of pounds into that after the program is finished. Maybe don't use the same massive, costly consultancies that fail to deliver? The Obamacare website was a great example of this IMO. Costly consultancy failed to deliver, small agile teams brought in to save the day - if only a bit too late.
OhHeyItsE 11y ago
Would have been much easier to create a list of projects that were completed within budget (assuming that's ever actually happened).
- softawre 11y ago
  Hey, I wrote a little shell script yesterday that I had budgeted 30 minutes for and had it done in 20. But outside of that one thing.. lol
rdudekul 11y ago
I was trying to look for patterns in these projects. Looks like most of them were outsourced and most of them were US/UK Government projects. It would be interesting to see patterns in very successful US/UK Government projects. My personal experience has been that it almost always boils down to upper management/leadership issues, that result in hiring weak project managers which in turn leads to talent drain on the implementation side.
- gaius 11y ago
  What it boils down to is no-one is spending their own money, they have a bottomless pit from the poor taxpayer.
panglott 11y ago
Why are these all in the UK or US? Better transparency?
- slantyyz 11y ago
  >> Why are these all in the UK or US? Better transparency? Maybe because that particular Wikipedia page's editors are from the UK or US? [edit: just speculating, don't know this for a fact] If I recall correctly, the Gun Registry [1] in Canada was a huge, expensive debacle, with a lot of the costs coming from the underestimation of the software development. [1] https://en.wikipedia.org/wiki/Canadian_Firearms_Registry https://en.wikipedia.org/wiki/Canadian_Firearms_Registry
jordache 11y ago
when they list healthcare.gov has costed 1.5bn, does that include setting up the back office and staffing, or simply the project that responsible for putting the website?
strommen 11y ago
In 2006 I worked briefly on an air-traffic control system (ERAM) that was eventually delivered 5 years late and $400M over budget. I was a small cog in a giant government-contract-software machine, but it was easy to see the project was going to be a disaster. There are some pretty basic strategies to make building "large" software easier. All of them were attempted, but in an entirely broken way: - Break the architecture down into smaller components that can be tested independently. - Break the functionality down into smaller features that can be released independently. - Develop automated tests at the system level (i.e. integration tests, not unit tests). The software took 2-4 minutes to launch in development, and very few of us knew how to meaningfully test it. This meant that any change that compiled (and passed code review) would ship. Oh yeah - and the system used binary structures in shared memory to communicate across processes, which were running code written in two entirely different languages.
- dotnetchris 11y ago
  > the system used binary structures in shared memory to communicate across processes, which were running code written in two entirely different languages. What could go wrong
  - Someone1234 11y ago
    "We should have just used XML for everything!" - said every Java developer ever. "We should just have used binary for everything!" - said every C/C++ developer ever. "We should just have used Json for everything!" - said everyone currently.
    - fratlas 11y ago
      Any reason not to use JSON? Apart from needing lightweight traffic, in which case MQTT would be fine, right?
      - com2kid 11y ago
        * You need strict typing * You need well defined numeric formats * Message volume is high enough that the parsing costs of messages outweigh the savings on implementation time * You cannot guarantee plain text messages won't be managed in transit. You can get around the strict typing by adopting a schema format, but then you have a tooling problem that all the languages you need to support may not support your chosen JSON schema format. Other solutions such as protobufs exist and are far better for transmitting large amounts of binary data.
jayvanguard 11y ago
I'm surprised the outsourcing companies aren't listed on this. They are often huge contributors to the overruns.
amist 11y ago
As a side note, I cannot understand why these articles are written in a table format. Where there are cells with such a long description, bullets would be much suitable.
- organsnyder 11y ago
  Agreed on the formatting, but the sorting functionality can be handy.
Grue3 11y ago
...in UK and US. Apparently other countries never had any of these.
edgyswingset 11y ago
I remember, when as a resident of Oregon, I was wondering why we couldn't sign up with the national healthcare system. I then noticed that the Cover Oregon system was being built by Oracle. Suffice to say, I wasn't the least bit surprised that it was a complete and total catastrophe, and even less surprised that Oracle tried to sue Oregon for attempting to use the software. Nothing like LDD - Lawyer Driven Development.
- Yhippa 11y ago
  I remember this being quite the story. Anybody know what came of it? Still in litigation?
  - schmichael 11y ago
    Still in litigation. OR has spent more than $6.6m in legal fees fighting Oracle.
    - killface 11y ago
      I still don't understand why people trust Oracle consultants. Their list of failures is ten miles long. I guess nobody gets fired for hiring Oracle.
      - tdeck 11y ago
        Oracle is today's IBM, just as IBM is today's Infosys.
  - phonon 11y ago
    > Still in litigation? Yup.
- agumonkey 11y ago
  > for attempting to use the software I don't understand that
  - 11y ago
api 11y ago
I see a lot of billions, with a b, and then I look at what small teams do on bootstrapping or little seed rounds. The mind boggles.
- jandrese 11y ago
  A huge, huge, cost driver in many of the applications is the fact that they have to integrate with the old systems in numerous ways. A startup could probably build the basic version of many of these apps no problem, but that's only the tip of the iceberg. When you have to then import millions of customer records, which were not stored in any particular format (they're idiosyncratic with each doctor/office), and then integrate with the COBOL based system vendor 1 uses, and the IBM Mainframe that vendor 2 uses, oh, and those systems were never intended to interconnect with anything and the vendors are years behind schedule as well, and all while you have lawyers breathing down your neck with a 28 inch thick binder with all of the regulations you need to follow written in impenetrable legalese you can see how it can cost billions of dollars. Your startup guys heads would explode if you asked them to write a formal specification for an interconnection system between their elegant web frameworks and a big blob binary COBOL based record system on an IBM 360 with only hardcopy documentation written by a vendor that went out of business in the 80s. And yes, healthcare systems still run in environments like that for the reasons mentioned above: making a new system is too hard/expensive. Oh, and the vendor that had the lowest bid for the contract is planning to outsource to some country that doesn't speak English and is 12 hours offset from your local time.
  - jonesb6 11y ago
    It still seems like an excuse, "it's hard". The cost of "hard" shouldn't be $15B.
ergothus 11y ago
The primary problem with lists like this is that they don't compare to successes. For example: * I've been on projects where we're rewritten applications from the ground up that were totally successful and we were better able to build in the future because we had a clean setup. * I've seen government IT shops hire contractors to do a large project and ended up with something better than the internal devs would have done. Both of these are situations where the opposite case (time-consuming and expensive failure) gets a lot of attention. (and, in my experience, failure is more common). Show me one of those failures and my instinct is to say "Of COURSE that was a failure! It was obviously going to fail!". But, in reality, not always. Rather than building unhelpful conventional wisdom, we should be trying to understand the distinctions between the failures and the successes.
pessimizer 11y ago
SAP doesn't even get a mention? Waste Management: http://www.computerworld.com/article/2517917/enterprise-applications/sap--waste-management-settle-lawsuit.html http://www.computerworld.com/article/2517917/enterprise-appl... Levi's: http://www.theregister.co.uk/2008/07/10/levis_erp_costs/ http://www.theregister.co.uk/2008/07/10/levis_erp_costs/
- mixmastamyk 11y ago
  It's wikipedia, add it. ;)
- mschuster91 11y ago
  The problem with SAP isn't the software, it's "consultants" writing purposefully bad code - the customer can't assess the delivered quality and the consultant will keep his "ongoing maintenance" fee. (Okay, yes, the software itself is a problem too but it's not the major problem)
pkorzeniewski 11y ago
When it comes to government projects, I always wonder how many people involved treat it only as an opportunity to make as much money as possible without being caught - after several years you just sweep the whole thing under carpet so nobody remembers how much money has been wasted and you're set.
- zhte415 11y ago
  This is true not only of government projects, but of all projects. Anyone in charge of a budget is potentially corrupt. I'd argue that it is more common in corporates than in government, because the personal damage that could be inflicted is less. A company doesn't want reputation damaged by being involved in fraudulent bidding (so will sweep under the carpet readily), and as long as the job gets done, nobody cares (unless internal politics get stirred). The employee may be fired, but they can go somewhere else. And all is off the public record.
TheAndruu 11y ago
This is by no means a conclusive list.
codingdave 11y ago
That list is much short than the headline would imply. I was afraid of a list of thousand of projects, with a final footnote of "Every Enterprise software application ever written."
timwaagh 11y ago
this is why haskell has support for infinite lists
frik 11y ago
Now we need a list of overbudget customized COTS software projects. With all these companies running on Excel macros, years old SAP R/2 or 3 (just two common examples) and so on, the list will be extensive. (COTS = commercial off-the-shelf)
- rbanffy 11y ago
  That'd need a whole Wikipedia just for it.
deleted 11y ago
[deleted]
tyre 11y ago
96% of government IT procurements over $10m are deemed failures.[1] Absolutely shocking how ineffective the system of contractors and RFPs is. Part of it is a mindset, part of it is just needing better options. [1] https://www.washingtonpost.com/news/wonk/wp/2013/10/22/the-lessons-of-healthcare-gov-stretch-far-beyond-obamacare/ https://www.washingtonpost.com/news/wonk/wp/2013/10/22/the-l...
mcguire 11y ago
These lists seem a little short. What about: * The Distributed Computing Environment (or at least the OS/2 portion of it, anyway), https://en.wikipedia.org/wiki/Distributed_Computing_Environment https://en.wikipedia.org/wiki/Distributed_Computing_Environm... * Taligent, https://en.wikipedia.org/wiki/Taligent https://en.wikipedia.org/wiki/Taligent * IBM Workplace OS, https://en.wikipedia.org/wiki/IBM_Workplace_OS https://en.wikipedia.org/wiki/IBM_Workplace_OS And that's just a few from off my personal resume.
mrslave 11y ago
I will document my bias before reading the article in the hope that I am proven wrong. I estimate that most of these have occurred in apparatus of the state, including organizations that superficially are not but for all practical purposes (funding, license, regulation) are. Wise man once say: there's nothing easier in this world than spending someone else's money.
peterburkimsher 11y ago
Did nobody mention Xanadu?
natetan 11y ago
Great idea. The UK tried a NOMS (offender management) system and that was an expensive disaster too.. Would be nice to put down the contracting companies who were ripping of the Govt along with the civil servants who over saw the massive waste of tax payers money/time.
ap22213 11y ago
Note: very few, if any, of these examples failed because someone on the team didn't pass a coding test. Next time you interview someone remember: bad code could bring your project down, but there are 9 other factors that are more likely to bring your project down.
repeek 11y ago
For those interested in this topic, I highly recommend episode 181 of the Omega Tau podcast, "Why Megaprojects Fail (and what to do about it)" [1]. It's a fascinating listen into the psychology behind gigantic failures. [1] http://omegataupodcast.net/2015/09/181-why-megaprojects-fail-and-what-to-do-about-it/ http://omegataupodcast.net/2015/09/181-why-megaprojects-fail...
tetraverse 11y ago
"Sabina SEIFERT's Project: Software Project Failures" http://www.scism.lsbu.ac.uk/inmandw/past/9697/projects/prj127.html http://www.scism.lsbu.ac.uk/inmandw/past/9697/projects/prj12...
stretchwithme 11y ago
The Software Engineering Radio podcast did an episode on this recently: http://www.se-radio.net/2016/02/se-radio-epislode-250-jurgen-laartz-and-alexander-budzier-on-why-large-it-projects-fail/ http://www.se-radio.net/2016/02/se-radio-epislode-250-jurgen...
boxfire 11y ago
See Also: Agile Software Development
DyslexicAtheist 11y ago
As far as I can see all software built in Germany always ships on time and is within budget. Or maybe the page needs an update :-)

154 points by monort 11y ago

IEEE Spectrum had a great series of interactive information graphics on 'Lessons from a decade of IT failures' [0]. In particular, "The Staggering Impact of IT Systems Gone Wrong" [1] and "Monuments to failure" [2] are good lists of failures, many of which are not included on Wikipedia's list. [0]: http://spectrum.ieee.org/static/lessons-from-a-decade-of-it-failures http://spectrum.ieee.org/static/lessons-from-a-decade-of-it-... [1]: http://spectrum.ieee.org/static/the-staggering-impact-of-it-systems-gone-wrong http://spectrum.ieee.org/static/the-staggering-impact-of-it-... [2]: http://spectrum.ieee.org/static/monuments-to-failure http://spectrum.ieee.org/static/monuments-to-failure

thrownaway2424 11y ago
I don't know if those are all in the same category. For example the Army cancelled Future Combat System because it didn't work. Would the project have been considered a "success" if they had put it into production instead? For my money, a real failure is like the giant BART fiasco in the 90s where they spent 80 million dollars on a new train control system that simply never existed, at all. They had to sue General Electric to get their money back.
- AnimalMuppet 11y ago
  > I don't know if those are all in the same category. For example the Army cancelled Future Combat System because it didn't work. Would the project have been considered a "success" if they had put it into production instead? If it didn't work? Hopefully not. But military stuff is a bit different. Military systems tend to try to push the state of the art as far as possible, for as much edge in combat as possible. That makes for much higher chances of failure.

cthulhujr 11y ago

>This article contains one or more incomplete lists which may never be able to satisfy particular standards for completeness Well isn't that ironic.

SmellyGeekBoy 11y ago
Don't you think?

kozukumi 11y ago

The modern NHS system in the UK was the biggest fuck up I have ever seen. I worked with people who also worked on the project. They are seriously amazing people but apparently all the layers of management, red tape and politics was, according to my friends, the worst they have ever experienced. Two friends were so traumatised by the experience they ended up taking 2 years off work to travel/recover. The sad thing is it shouldn't have been such a difficult thing to develop. Complex sure but no more complex than many other similar systems. It was destroyed by too much meddling from the government adding, removing and changing the spec every other day and many, many managers just finding non-sense jobs that were totally pointless. It is shocking it went so over budget and was allowed to continue doing so for so, so long.

teh_klev 11y ago
Which NHS body in the UK are you talking about? There are four separate healthcare services in the UK which operate and are funded separately: NHS Scotland NHS England NHS Wales HSC Northern Ireland
- noir_lord 11y ago
  NHS England - The National Programme for IT. It is as far as I've seen the 'greatest' fuckup in UK development history, 12.something billion down the drain, massive overspending, they basically made every mistake you could make short of setting the office on fire.
  - aledalgrande 11y ago
    They should make mismanagement of a government project become a felony.
    - dragonwriter 11y ago
      > They should make mismanagement of a government project become a felony. This would involve either: (1) Clear and unambiguous definition of "mismanagement" involving distant-from-the-specific-project procedural mandates, which would reduce "mismanagement" by the definition applied but increase (and mandate) what would be functionally mismanagement for specific real projects (lots of government regulation designed to prevent mismanagement all around the world, while usually not specifically felony criminal rules, works this way now), or (2) Being so ambiguous and vague so as to provide a basis for arbitrary prosecution, such that it would make sure no one wanted to touch management of a government project at all (at least -- assuming this was the US or one with similar fundamental rules as to what can be an enforceable criminal law -- until it was inevitably ruled unenforceable as impermissibly vague.)

tyingq 11y ago

Of course, there's a level of disclosure for Government run projects that means their blunders get exposed more easily. I'm quite sure there are some spectacular failures in the corporate world that won't make the list.

deleted 11y ago
[deleted]
pascalmemories 11y ago
Sadly, no, they are not more easily exposed. Most are quietly swept under the carpet and no-one (other than those working on it) ever know it what a crazy mess it was. It gets classed as something else, is re-tagged as a proof of concept (so was never supposed to deliver anything other than improve understanding of an issue) etc. The ones you hear about are the ones the government ministers foolishly chose to boast about in some answer in parliament or announce in some TV interview. Normally that's when consultants pounce on it and offer to help make it succeed [i.e. vampire all the money away in insane fees]. A minister announcing it is practically the death-knell of a project. The ones which succeed generally keep publicity and announcements to an absolute minimum and keep tight control. That's way more difficult than you imagine on a government project as civil servants go hunting for projects to buddy up to in order to pad their promotion cv. If your project budget is more than the last one they screwed, they're all over it so they can boast about the incremental increase in projects they contributed to - a major promotion point (as no-one checks if the project succeeded, the promotion is hinged on how much public money it pissed away). I was ecstatic to leave UK Gov projects behind.
DubiousPusher 11y ago
There's also an asymmetry in the way we think about public and private projects. When a piece of private software flops, it still exists as a product and however bad, often offsets some of its cost in sales. Since most public software is offered for free, no cost is visibly recouped. Healthcare.gov is actually a great example. It was a colossal failure at first. A quantity easily measured in dollars spent. However, it has largely functioned in the years since but measuring its positive economic impact is hard because the service is delivered free. None if that money has been reclaimed but it certainly has had some positive economic value.

mrweasel 11y ago

I should sit down and add the failed IT projects of the Danish government, but I don't have that kind of time. Our government is notorious for initiating software projects that ends in almost predictable failure. The latest project to fail was a system for the Danish Tax Office. It was suppose to automatically collect debt owed by citizens and companies.It's not entirely certain how that was suppose to work and it has been described as a utopia project with an unprecedented expectation of what kind of problem computers are capable of handling. The specification where 9000 pages long, tax law not included, and some how failing to supply a sufficient amount of actual requirements. It failed, of cause, costing almost $150.000.000 (DKK 1.000.000.0000), not including the 78 Billion Danish Kroner the tax office haven't been able to collect due the system not being delivered on time and generally not working.

busterarm 11y ago
Doesn't the government there (and most companies for that matter) contract IBM for big software projects? You know the old adage "nobody gets fired for buying IBM". I have a friend that's partner at a consultancy in Copenhagen and he says IBM is often their competition.
- mrweasel 11y ago
  Mostly CSC actually. They have contract for systems that should have been part of licensing rounds, which would have allowed other to take over support, but no other where ever allowed to bid. The reason: CSC documented the systems so poorly that no one else will be able to handle the support. The government knows that it's illegal, yet CSCs contracts have automatically been renewed, multiple times. But yes, IBM have a large number of contracts as well. CSC just seems to be behind the worst projects. I suspect they bid for project and just focus on delivering exactly what's in the contract, knowing that it will never work. Either that or they are even more incompetent that anyone could imagine.
  - busterarm 11y ago
    Given my experience with the managers of CSC here in the States, that sounds exactly like what I'd expect. From my discussions with that friend I mentioned though it sounds like the attitude is changing and there's more work going to smaller shops like his (though friendships with government officials seems to help loads when it comes to securing work). Honestly it sounds like a really exciting environment to work in when we talk about it and I wish I had the time to really learn Danish so I could have a decent chance of securing good work out there.

orf 11y ago

Can someone explain to me how the fuck my government can spend 15.8bn[1] on a software project? We could build 5 Queen Elizabeth class aircraft carriers for that. Five! For the cost of a bloated Java app! Seriously, someone fill me in please. How can a software project possibly ever cost more than a billion pounds? You could employ 20,000 software engineers full time for £50K ($70k) a year for that much, and I expect the number of engineers working on Universal Credit is much lower than that and paid less. 1. https://en.wikipedia.org/wiki/Universal_Credit#Costs https://en.wikipedia.org/wiki/Universal_Credit#Costs

detaro 11y ago
They don't employ the developers. The development is done by companies that might pay the developers that much, but bill the client more per hour and bill the client more time. And of course you can't forget project managers, architects and other consultants that are more expensive. And have a lot of meetings. And hardware, and oracle databases/SAP licenses, and support contracts for those, and audits, and extra charges for change requests, and travel budgets, and... Just because one team of developers could do something in 200 man hours doesn't mean you can't spend and bill 2000 for it. And that's just development. Data migrations and user training/support are going to be massive issues if you replace or newly introduce a widely used system.
- organsnyder 11y ago
  > Data migrations and user training/support are going to be massive issues if you replace or newly introduce a widely used system. Yup. My employer is undertaking a massive IT project (many many many millions of USD), and it's my understanding that training accounts for over half of the budget.
- orf 11y ago
  > Just because one team of developers could do something in 200 man hours doesn't mean you can't spend and bill 2000 for it. It doesn't mean you should pay, or plan, to spend 20,000 man hours in meetings and devote huge chunks of your budget to expensive licenses for something that could take 200 man hours, especially when it's taxpayers money. Sure there are unavoidable overheads but why does our government keep splashing billions on them and failing, with little accountability? Training is going to be expensive, sure, but perhaps plow millions of pounds into that after the program is finished. Maybe don't use the same massive, costly consultancies that fail to deliver? The Obamacare website was a great example of this IMO. Costly consultancy failed to deliver, small agile teams brought in to save the day - if only a bit too late.

OhHeyItsE 11y ago

Would have been much easier to create a list of projects that were completed within budget (assuming that's ever actually happened).

softawre 11y ago
Hey, I wrote a little shell script yesterday that I had budgeted 30 minutes for and had it done in 20. But outside of that one thing.. lol

rdudekul 11y ago

I was trying to look for patterns in these projects. Looks like most of them were outsourced and most of them were US/UK Government projects. It would be interesting to see patterns in very successful US/UK Government projects. My personal experience has been that it almost always boils down to upper management/leadership issues, that result in hiring weak project managers which in turn leads to talent drain on the implementation side.

gaius 11y ago
What it boils down to is no-one is spending their own money, they have a bottomless pit from the poor taxpayer.

panglott 11y ago

Why are these all in the UK or US? Better transparency?

slantyyz 11y ago
>> Why are these all in the UK or US? Better transparency? Maybe because that particular Wikipedia page's editors are from the UK or US? [edit: just speculating, don't know this for a fact] If I recall correctly, the Gun Registry [1] in Canada was a huge, expensive debacle, with a lot of the costs coming from the underestimation of the software development. [1] https://en.wikipedia.org/wiki/Canadian_Firearms_Registry https://en.wikipedia.org/wiki/Canadian_Firearms_Registry

jordache 11y ago

when they list healthcare.gov has costed 1.5bn, does that include setting up the back office and staffing, or simply the project that responsible for putting the website?

strommen 11y ago

In 2006 I worked briefly on an air-traffic control system (ERAM) that was eventually delivered 5 years late and $400M over budget. I was a small cog in a giant government-contract-software machine, but it was easy to see the project was going to be a disaster. There are some pretty basic strategies to make building "large" software easier. All of them were attempted, but in an entirely broken way: - Break the architecture down into smaller components that can be tested independently. - Break the functionality down into smaller features that can be released independently. - Develop automated tests at the system level (i.e. integration tests, not unit tests). The software took 2-4 minutes to launch in development, and very few of us knew how to meaningfully test it. This meant that any change that compiled (and passed code review) would ship. Oh yeah - and the system used binary structures in shared memory to communicate across processes, which were running code written in two entirely different languages.

dotnetchris 11y ago
> the system used binary structures in shared memory to communicate across processes, which were running code written in two entirely different languages. What could go wrong
- Someone1234 11y ago
  "We should have just used XML for everything!" - said every Java developer ever. "We should just have used binary for everything!" - said every C/C++ developer ever. "We should just have used Json for everything!" - said everyone currently.
  - fratlas 11y ago
    Any reason not to use JSON? Apart from needing lightweight traffic, in which case MQTT would be fine, right?
    - com2kid 11y ago
      * You need strict typing * You need well defined numeric formats * Message volume is high enough that the parsing costs of messages outweigh the savings on implementation time * You cannot guarantee plain text messages won't be managed in transit. You can get around the strict typing by adopting a schema format, but then you have a tooling problem that all the languages you need to support may not support your chosen JSON schema format. Other solutions such as protobufs exist and are far better for transmitting large amounts of binary data.

jayvanguard 11y ago

I'm surprised the outsourcing companies aren't listed on this. They are often huge contributors to the overruns.

amist 11y ago

As a side note, I cannot understand why these articles are written in a table format. Where there are cells with such a long description, bullets would be much suitable.

organsnyder 11y ago
Agreed on the formatting, but the sorting functionality can be handy.

Grue3 11y ago

...in UK and US. Apparently other countries never had any of these.

edgyswingset 11y ago

I remember, when as a resident of Oregon, I was wondering why we couldn't sign up with the national healthcare system. I then noticed that the Cover Oregon system was being built by Oracle. Suffice to say, I wasn't the least bit surprised that it was a complete and total catastrophe, and even less surprised that Oracle tried to sue Oregon for attempting to use the software. Nothing like LDD - Lawyer Driven Development.

Yhippa 11y ago
I remember this being quite the story. Anybody know what came of it? Still in litigation?
- schmichael 11y ago
  Still in litigation. OR has spent more than $6.6m in legal fees fighting Oracle.
  - killface 11y ago
    I still don't understand why people trust Oracle consultants. Their list of failures is ten miles long. I guess nobody gets fired for hiring Oracle.
    - tdeck 11y ago
      Oracle is today's IBM, just as IBM is today's Infosys.
- phonon 11y ago
  > Still in litigation? Yup.
agumonkey 11y ago
> for attempting to use the software I don't understand that
- 11y ago

api 11y ago

I see a lot of billions, with a b, and then I look at what small teams do on bootstrapping or little seed rounds. The mind boggles.

jandrese 11y ago
A huge, huge, cost driver in many of the applications is the fact that they have to integrate with the old systems in numerous ways. A startup could probably build the basic version of many of these apps no problem, but that's only the tip of the iceberg. When you have to then import millions of customer records, which were not stored in any particular format (they're idiosyncratic with each doctor/office), and then integrate with the COBOL based system vendor 1 uses, and the IBM Mainframe that vendor 2 uses, oh, and those systems were never intended to interconnect with anything and the vendors are years behind schedule as well, and all while you have lawyers breathing down your neck with a 28 inch thick binder with all of the regulations you need to follow written in impenetrable legalese you can see how it can cost billions of dollars. Your startup guys heads would explode if you asked them to write a formal specification for an interconnection system between their elegant web frameworks and a big blob binary COBOL based record system on an IBM 360 with only hardcopy documentation written by a vendor that went out of business in the 80s. And yes, healthcare systems still run in environments like that for the reasons mentioned above: making a new system is too hard/expensive. Oh, and the vendor that had the lowest bid for the contract is planning to outsource to some country that doesn't speak English and is 12 hours offset from your local time.
- jonesb6 11y ago
  It still seems like an excuse, "it's hard". The cost of "hard" shouldn't be $15B.

ergothus 11y ago

The primary problem with lists like this is that they don't compare to successes. For example: * I've been on projects where we're rewritten applications from the ground up that were totally successful and we were better able to build in the future because we had a clean setup. * I've seen government IT shops hire contractors to do a large project and ended up with something better than the internal devs would have done. Both of these are situations where the opposite case (time-consuming and expensive failure) gets a lot of attention. (and, in my experience, failure is more common). Show me one of those failures and my instinct is to say "Of COURSE that was a failure! It was obviously going to fail!". But, in reality, not always. Rather than building unhelpful conventional wisdom, we should be trying to understand the distinctions between the failures and the successes.

pessimizer 11y ago

SAP doesn't even get a mention? Waste Management: http://www.computerworld.com/article/2517917/enterprise-applications/sap--waste-management-settle-lawsuit.html http://www.computerworld.com/article/2517917/enterprise-appl... Levi's: http://www.theregister.co.uk/2008/07/10/levis_erp_costs/ http://www.theregister.co.uk/2008/07/10/levis_erp_costs/

mixmastamyk 11y ago
It's wikipedia, add it. ;)
mschuster91 11y ago
The problem with SAP isn't the software, it's "consultants" writing purposefully bad code - the customer can't assess the delivered quality and the consultant will keep his "ongoing maintenance" fee. (Okay, yes, the software itself is a problem too but it's not the major problem)

pkorzeniewski 11y ago

When it comes to government projects, I always wonder how many people involved treat it only as an opportunity to make as much money as possible without being caught - after several years you just sweep the whole thing under carpet so nobody remembers how much money has been wasted and you're set.

zhte415 11y ago
This is true not only of government projects, but of all projects. Anyone in charge of a budget is potentially corrupt. I'd argue that it is more common in corporates than in government, because the personal damage that could be inflicted is less. A company doesn't want reputation damaged by being involved in fraudulent bidding (so will sweep under the carpet readily), and as long as the job gets done, nobody cares (unless internal politics get stirred). The employee may be fired, but they can go somewhere else. And all is off the public record.

TheAndruu 11y ago

This is by no means a conclusive list.

codingdave 11y ago

That list is much short than the headline would imply. I was afraid of a list of thousand of projects, with a final footnote of "Every Enterprise software application ever written."

timwaagh 11y ago

this is why haskell has support for infinite lists

frik 11y ago

Now we need a list of overbudget customized COTS software projects. With all these companies running on Excel macros, years old SAP R/2 or 3 (just two common examples) and so on, the list will be extensive. (COTS = commercial off-the-shelf)

rbanffy 11y ago
That'd need a whole Wikipedia just for it.

deleted 11y ago

[deleted]

tyre 11y ago

96% of government IT procurements over $10m are deemed failures.[1] Absolutely shocking how ineffective the system of contractors and RFPs is. Part of it is a mindset, part of it is just needing better options. [1] https://www.washingtonpost.com/news/wonk/wp/2013/10/22/the-lessons-of-healthcare-gov-stretch-far-beyond-obamacare/ https://www.washingtonpost.com/news/wonk/wp/2013/10/22/the-l...

mcguire 11y ago

These lists seem a little short. What about: * The Distributed Computing Environment (or at least the OS/2 portion of it, anyway), https://en.wikipedia.org/wiki/Distributed_Computing_Environment https://en.wikipedia.org/wiki/Distributed_Computing_Environm... * Taligent, https://en.wikipedia.org/wiki/Taligent https://en.wikipedia.org/wiki/Taligent * IBM Workplace OS, https://en.wikipedia.org/wiki/IBM_Workplace_OS https://en.wikipedia.org/wiki/IBM_Workplace_OS And that's just a few from off my personal resume.

mrslave 11y ago

I will document my bias before reading the article in the hope that I am proven wrong. I estimate that most of these have occurred in apparatus of the state, including organizations that superficially are not but for all practical purposes (funding, license, regulation) are. Wise man once say: there's nothing easier in this world than spending someone else's money.

peterburkimsher 11y ago

Did nobody mention Xanadu?

natetan 11y ago

Great idea. The UK tried a NOMS (offender management) system and that was an expensive disaster too.. Would be nice to put down the contracting companies who were ripping of the Govt along with the civil servants who over saw the massive waste of tax payers money/time.

ap22213 11y ago

Note: very few, if any, of these examples failed because someone on the team didn't pass a coding test. Next time you interview someone remember: bad code could bring your project down, but there are 9 other factors that are more likely to bring your project down.

repeek 11y ago

For those interested in this topic, I highly recommend episode 181 of the Omega Tau podcast, "Why Megaprojects Fail (and what to do about it)" [1]. It's a fascinating listen into the psychology behind gigantic failures. [1] http://omegataupodcast.net/2015/09/181-why-megaprojects-fail-and-what-to-do-about-it/ http://omegataupodcast.net/2015/09/181-why-megaprojects-fail...

tetraverse 11y ago

"Sabina SEIFERT's Project: Software Project Failures" http://www.scism.lsbu.ac.uk/inmandw/past/9697/projects/prj127.html http://www.scism.lsbu.ac.uk/inmandw/past/9697/projects/prj12...

stretchwithme 11y ago

The Software Engineering Radio podcast did an episode on this recently: http://www.se-radio.net/2016/02/se-radio-epislode-250-jurgen-laartz-and-alexander-budzier-on-why-large-it-projects-fail/ http://www.se-radio.net/2016/02/se-radio-epislode-250-jurgen...

boxfire 11y ago