4 ms·

Why it is Important that Software Projects Fail

54 points by fogus 17y ago

deleted 17y ago
[deleted]
yummyfajitas 17y ago
While bridge building may always be successful (as asserted by the author), close to budget really depends on how you define "close". Fixed link projects (bridges and tunnels) tend to cost 33% more than estimated (sigma=60%). Rail is even worse at 44% more than estimated, coming in under budget only about 12% of the time. http://flyvbjerg.plan.aau.dk/JAPAASPUBLISHED.pdf http://flyvbjerg.plan.aau.dk/JAPAASPUBLISHED.pdf
- ebrenes 17y ago
  And even if successful (in that the bridge does not immediately collapse), many bridges have been rendered inoperable due to misuse or neglect. Anyways, other interesting links on the subject: http://blogs.msdn.com/dave_froslie/archive/2005/03/07/389113.aspx http://blogs.msdn.com/dave_froslie/archive/2005/03/07/389113... http://blogs.msdn.com/steverowe/archive/2005/02/28/381910.aspx http://blogs.msdn.com/steverowe/archive/2005/02/28/381910.as...
grosales 17y ago
Wasn't Fred Brooks who said something along the lines of "always build one to throw away"? Even though, this is not supposed to be a satire (or is it?) , the fact that software engineering is a very young subject needs to be taken into consideration. On the other hand, We need failed software projects to know what not to do.
scotty79 17y ago
That kinda reminds me how when you upgrade hardware for your webapp new features quickly crawl in that utilise the hardware. They might not be needed but they become possible so they come into existence. To reduce amount of bureaucracy you'd have to pick some work-consuming area of the operation, make it obsolete by government act and fire all the people that were doing that work so they will not be available for other tasks inflating them. To avoid their complaints you should still pay them salaries for few years but allow and encourage them to find themselves another job outside government sector.
- Freebytes 17y ago
  This also happens when you upgrade to a new version of a programming language or operating system. You start to use features that previously did not exist or restructure your code to utilize such features. However, many times this is a great benefit to help with the readability of the code. I have done this myself with hardware and software.
fab13n 17y ago
Automation frees people from doing something productive, not from working. If we (as a society) leave them alone, they'll just clutter around their formally useful job, and become banking middle manager or something similar, which moves a lot of paperwork around, but produce little actual value. This is similar to what we experienced in Europe with nobles: the end of feodal times made noblemen mostly useless as warriors or local mobsters; since they had nothing useful to do with their free time, they became passionate with court intrigues, ballroom dancing, fancy wigs, duels, and other improductive occupations. However, it's been fixed: after centuries of wasted potential, capitalism created a draft toward productive activities, and the dominant classes worked at organizing the workforce, eventually leading to the industrial revolution. We have (again, as a society) to find a way to make today's socially useful activities attractive: most people will concede that an eldery care person is way more useful than many middle managers; yet the latters get more money and social recognition. I suspect that such issues aren't fixed spontaneously by the so-called "invisible hand of market".
- moron4hire 17y ago
  Humans tend to assume that their experiences are normative, and are surprised to see instances where their experiences are not shared. As a personal example, my friends are usually significantly shocked to learn I haven't seen the movie "The Boondock Saints". Yet they think nothing of the fact that they haven't seen Salvador Dali's "Un Chien Andalou", which I have. Our values are informed by our experiences, so we also tend to assume our values are shared by others. Assuming one's values are normative leads to assuming that the fact they aren't reflected in the world must be the result of some external entity, a secret society, or this nebulous "the system". Instead, if one assumes that one's views aren't normative, then it is clear why one's views are not reflected in the world in aggregate: it's a simple tautology. Occam's Razor applies. "The system" is not an entity unto itself, it is intractable from the people within it, it is nothing more than the people within it. The "invisible hand of the market" is merely a short-hand, anthropomorphizing term for "the aggregate activity of the people within the poorly defined borders of 'the market'." If something isn't a particular way, it's because "most people" demonstrably do NOT concede your point, or if they do, do not assign such great importance to it as to warrant change. I don't like that elderly care personnel are paid less than middle managers either, but I'm not suggesting to force everyone else to make that valuation. When you say, "we as a society have to find a way...", and if you are suggesting that way be via government intervention, then really you're saying, "I want to force everyone else in society to pay for my way..." (undemocratic, even totalitarian). Then we have to write a new tax code to pay for it, and we get this situation here in the OP. If instead you are suggesting that you will live by example and proselytize to your fellow man to do the same, then really you're using "the invisible hand of the market" to enact the change you desire.
conanite 17y ago
The author argues that failed software projects are nonetheless beneficial: Their development employs many bureaucrats, consultants and contractors and their expenditure supports an even larger number of people throughout society. ... by the same argument I should stroll around the city and burn some cars, thereby offering employment to many firefighters, car producers, actuaries and all their suppliers. The point about ever-expanding bureaucracies is well noted however. Perhaps it is our duty to design really bad programming languages and methodologies, to sell them to government organisations and similar bureaucracies in order to drive their likelihood of success even lower. Reminds me of the true purpose of Ada ... http://home.pipeline.com/~hbaker1/sigplannotices/gigo-1997-04.html http://home.pipeline.com/~hbaker1/sigplannotices/gigo-1997-0...
- crescendo 17y ago
  This is highly reminiscent of the broken window fallacy: http://en.wikipedia.org/wiki/Parable_of_the_broken_window http://en.wikipedia.org/wiki/Parable_of_the_broken_window
- jerf 17y ago
  Yes, but let's be fair, this is an incidental parenthetical to the main point, which is that the true value of these failed projects comes from the fact that the projects never increase the efficiency of the bureaucracy. If you simply cut that paragraph the essay is, if anything, improved, demonstrating it's not a critical point.
  - conanite 17y ago
    I agree, the point about employment was nitpicking. I should have made that clearer. I liked the essay overall.
rauljara 17y ago
If I'm following the author, increased efficiency (successful software projects) leads to increased time on the hands of bureaucrats. Increased time leads to increased meaningless tasks. So, the author's solution is not to find something meaningful for people to do with the time, or to simply give people more time off work (because according to the 2 references he cites, that would be impossible), but to celebrate the destruction of efficiency (failed software projects). There are many things wrong with the way our society rewards work/ constructs bureaucracies. I think, however, that it wouldn't be too hard to come up with some practical solutions rather than encourage subconscious intentional failure.
- forensic 17y ago
  >I think, however, that it wouldn't be too hard to come up with some practical solutions rather than encourage subconscious intentional failure. Your hubris knows no limits!
dennisgorelik 17y ago
That's a very strong Libertarian case. Government spending must be cut very significantly.
- Freebytes 17y ago
  I voted for Bob Barr simply because I believe the Fair Tax needs recognition. We should do away with the complexities even at the cost of jobs. The IRS employees could be put to use in other parts of the government, and H&R Block could focus more on accounting than simply paying people to use their software that anyone at home could use by themselves.
  - sethg 17y ago
    The entire Justice Department (not just the IRS) has a budget authority of about $25 billion, out of a budget with roughly $2.5 trillion in receipts and $3 trillion in outlays. So if the government not only eliminated the IRS but eliminated the entire Department of Justice, Federal spending would go down by less than one percent. Roll on the revolution, dude!
    - Freebytes 17y ago
      I did not say that we should do away with the archaic and convoluted systems the tax system currently implements for the benefit of saving federal tax dollars by the elimination of the IRS or Justice Department. Any suggestion to do away with the Justice Department is not clear thinking anyway. I said that we should eliminate unnecessary complications within the tax system. The Fair Tax would result in an increase in tax collection methods without the necessity of complications, it would tax illegal immigrants, and it would result in less of a burden for people that want to be fair to the system but have little knowledge of the tax laws.
- sethg 17y ago
  It seems to me that the problem the OP describes is not confined to governments. If some layer or department of a large company's bureaucracy is consuming about 1.5% of its revenues (just as the Australian tax office consumes about 1.5% of Australia's GDP), then the upper management will never have much interest in driving that proportion down to 0.5%. So as the company grows, that layer or department will grow as well in spite of automation, just as the tax office has grown.
ams6110 17y ago
On an everyday level, this reminds me of an observation made by one of Crichton's characters (I think it was the mathematician in Jurassic Park) that despite all the appliances and conveniences of modern technology, housekeeping still takes the same number of hours a week as it always has.
mmt 17y ago
Hard working couples struggle to buy the basic food and shelter which their grandfathers had purchased while their wives stayed at home. Is this an example of truth through mere assertion, or of pluralizing anecdote to equal data? In any case, I disagree with the assertion, because I think it's a stretch that such a couple would consider what was "basic" in 1955 to be acceptable. It's tough, though, since quality of life is so, well, qualitative. Another aspect I question about the mid-50s is how war surplus affected cost of goods. How many of those single income households were buying goods that had been subsidized by their wives' work for the war effort? There are strong hints of this effect in early computer power supplies, as 400Hz wasn't historically common in commercial power.

54 points by fogus 17y ago

deleted 17y ago

[deleted]

While bridge building may always be successful (as asserted by the author), close to budget really depends on how you define "close". Fixed link projects (bridges and tunnels) tend to cost 33% more than estimated (sigma=60%). Rail is even worse at 44% more than estimated, coming in under budget only about 12% of the time. http://flyvbjerg.plan.aau.dk/JAPAASPUBLISHED.pdf http://flyvbjerg.plan.aau.dk/JAPAASPUBLISHED.pdf

ebrenes 17y ago
And even if successful (in that the bridge does not immediately collapse), many bridges have been rendered inoperable due to misuse or neglect. Anyways, other interesting links on the subject: http://blogs.msdn.com/dave_froslie/archive/2005/03/07/389113.aspx http://blogs.msdn.com/dave_froslie/archive/2005/03/07/389113... http://blogs.msdn.com/steverowe/archive/2005/02/28/381910.aspx http://blogs.msdn.com/steverowe/archive/2005/02/28/381910.as...

grosales 17y ago

Wasn't Fred Brooks who said something along the lines of "always build one to throw away"? Even though, this is not supposed to be a satire (or is it?) , the fact that software engineering is a very young subject needs to be taken into consideration. On the other hand, We need failed software projects to know what not to do.

scotty79 17y ago

That kinda reminds me how when you upgrade hardware for your webapp new features quickly crawl in that utilise the hardware. They might not be needed but they become possible so they come into existence. To reduce amount of bureaucracy you'd have to pick some work-consuming area of the operation, make it obsolete by government act and fire all the people that were doing that work so they will not be available for other tasks inflating them. To avoid their complaints you should still pay them salaries for few years but allow and encourage them to find themselves another job outside government sector.

Freebytes 17y ago
This also happens when you upgrade to a new version of a programming language or operating system. You start to use features that previously did not exist or restructure your code to utilize such features. However, many times this is a great benefit to help with the readability of the code. I have done this myself with hardware and software.

fab13n 17y ago

Automation frees people from doing something productive, not from working. If we (as a society) leave them alone, they'll just clutter around their formally useful job, and become banking middle manager or something similar, which moves a lot of paperwork around, but produce little actual value. This is similar to what we experienced in Europe with nobles: the end of feodal times made noblemen mostly useless as warriors or local mobsters; since they had nothing useful to do with their free time, they became passionate with court intrigues, ballroom dancing, fancy wigs, duels, and other improductive occupations. However, it's been fixed: after centuries of wasted potential, capitalism created a draft toward productive activities, and the dominant classes worked at organizing the workforce, eventually leading to the industrial revolution. We have (again, as a society) to find a way to make today's socially useful activities attractive: most people will concede that an eldery care person is way more useful than many middle managers; yet the latters get more money and social recognition. I suspect that such issues aren't fixed spontaneously by the so-called "invisible hand of market".

moron4hire 17y ago
Humans tend to assume that their experiences are normative, and are surprised to see instances where their experiences are not shared. As a personal example, my friends are usually significantly shocked to learn I haven't seen the movie "The Boondock Saints". Yet they think nothing of the fact that they haven't seen Salvador Dali's "Un Chien Andalou", which I have. Our values are informed by our experiences, so we also tend to assume our values are shared by others. Assuming one's values are normative leads to assuming that the fact they aren't reflected in the world must be the result of some external entity, a secret society, or this nebulous "the system". Instead, if one assumes that one's views aren't normative, then it is clear why one's views are not reflected in the world in aggregate: it's a simple tautology. Occam's Razor applies. "The system" is not an entity unto itself, it is intractable from the people within it, it is nothing more than the people within it. The "invisible hand of the market" is merely a short-hand, anthropomorphizing term for "the aggregate activity of the people within the poorly defined borders of 'the market'." If something isn't a particular way, it's because "most people" demonstrably do NOT concede your point, or if they do, do not assign such great importance to it as to warrant change. I don't like that elderly care personnel are paid less than middle managers either, but I'm not suggesting to force everyone else to make that valuation. When you say, "we as a society have to find a way...", and if you are suggesting that way be via government intervention, then really you're saying, "I want to force everyone else in society to pay for my way..." (undemocratic, even totalitarian). Then we have to write a new tax code to pay for it, and we get this situation here in the OP. If instead you are suggesting that you will live by example and proselytize to your fellow man to do the same, then really you're using "the invisible hand of the market" to enact the change you desire.

conanite 17y ago

The author argues that failed software projects are nonetheless beneficial: Their development employs many bureaucrats, consultants and contractors and their expenditure supports an even larger number of people throughout society. ... by the same argument I should stroll around the city and burn some cars, thereby offering employment to many firefighters, car producers, actuaries and all their suppliers. The point about ever-expanding bureaucracies is well noted however. Perhaps it is our duty to design really bad programming languages and methodologies, to sell them to government organisations and similar bureaucracies in order to drive their likelihood of success even lower. Reminds me of the true purpose of Ada ... http://home.pipeline.com/~hbaker1/sigplannotices/gigo-1997-04.html http://home.pipeline.com/~hbaker1/sigplannotices/gigo-1997-0...

crescendo 17y ago
This is highly reminiscent of the broken window fallacy: http://en.wikipedia.org/wiki/Parable_of_the_broken_window http://en.wikipedia.org/wiki/Parable_of_the_broken_window
jerf 17y ago
Yes, but let's be fair, this is an incidental parenthetical to the main point, which is that the true value of these failed projects comes from the fact that the projects never increase the efficiency of the bureaucracy. If you simply cut that paragraph the essay is, if anything, improved, demonstrating it's not a critical point.
- conanite 17y ago
  I agree, the point about employment was nitpicking. I should have made that clearer. I liked the essay overall.

rauljara 17y ago

If I'm following the author, increased efficiency (successful software projects) leads to increased time on the hands of bureaucrats. Increased time leads to increased meaningless tasks. So, the author's solution is not to find something meaningful for people to do with the time, or to simply give people more time off work (because according to the 2 references he cites, that would be impossible), but to celebrate the destruction of efficiency (failed software projects). There are many things wrong with the way our society rewards work/ constructs bureaucracies. I think, however, that it wouldn't be too hard to come up with some practical solutions rather than encourage subconscious intentional failure.

forensic 17y ago
>I think, however, that it wouldn't be too hard to come up with some practical solutions rather than encourage subconscious intentional failure. Your hubris knows no limits!

dennisgorelik 17y ago

That's a very strong Libertarian case. Government spending must be cut very significantly.

Freebytes 17y ago
I voted for Bob Barr simply because I believe the Fair Tax needs recognition. We should do away with the complexities even at the cost of jobs. The IRS employees could be put to use in other parts of the government, and H&R Block could focus more on accounting than simply paying people to use their software that anyone at home could use by themselves.
- sethg 17y ago
  The entire Justice Department (not just the IRS) has a budget authority of about $25 billion, out of a budget with roughly $2.5 trillion in receipts and $3 trillion in outlays. So if the government not only eliminated the IRS but eliminated the entire Department of Justice, Federal spending would go down by less than one percent. Roll on the revolution, dude!
  - Freebytes 17y ago
    I did not say that we should do away with the archaic and convoluted systems the tax system currently implements for the benefit of saving federal tax dollars by the elimination of the IRS or Justice Department. Any suggestion to do away with the Justice Department is not clear thinking anyway. I said that we should eliminate unnecessary complications within the tax system. The Fair Tax would result in an increase in tax collection methods without the necessity of complications, it would tax illegal immigrants, and it would result in less of a burden for people that want to be fair to the system but have little knowledge of the tax laws.
sethg 17y ago
It seems to me that the problem the OP describes is not confined to governments. If some layer or department of a large company's bureaucracy is consuming about 1.5% of its revenues (just as the Australian tax office consumes about 1.5% of Australia's GDP), then the upper management will never have much interest in driving that proportion down to 0.5%. So as the company grows, that layer or department will grow as well in spite of automation, just as the tax office has grown.

ams6110 17y ago

On an everyday level, this reminds me of an observation made by one of Crichton's characters (I think it was the mathematician in Jurassic Park) that despite all the appliances and conveniences of modern technology, housekeeping still takes the same number of hours a week as it always has.

mmt 17y ago

Hard working couples struggle to buy the basic food and shelter which their grandfathers had purchased while their wives stayed at home. Is this an example of truth through mere assertion, or of pluralizing anecdote to equal data? In any case, I disagree with the assertion, because I think it's a stretch that such a couple would consider what was "basic" in 1955 to be acceptable. It's tough, though, since quality of life is so, well, qualitative. Another aspect I question about the mid-50s is how war surplus affected cost of goods. How many of those single income households were buying goods that had been subsidized by their wives' work for the war effort? There are strong hints of this effect in early computer power supplies, as 400Hz wasn't historically common in commercial power.