8 ms·

Why it is important that software projects fail (2008)

119 points by kwhitefoot 6y ago

3gg 6y ago
Amazing write-up, some really good kicks in there. The bloat in bureaucracy can be seen in the software itself too. Much of today's consumer software fundamentally performs the same function it did 20 years ago, yet it suffers from the same kind and number of bugs, runs slower, and uses more compute resources. Whatever advances have been in hardware are many times counteracted by the bloat of software. "Below 1% is easily affordable, so the bureaucracy will naturally grow beyond that size as predicted by Parkinson's law" applies to the software itself as much as it does to the bureaucracy it enables.
NiceWayToDoIT 6y ago
In some way, neither one project fails entirely - as it serves to distributes wealth from those that have too much to those that do not have enough. In some sense it would be even better for society if those who have lot of wealth risk even more into wild ideas and moonshot projects.
- ozim 6y ago
  But that is a broken window fallacy. It would be better for everyone if those who have a lot of wealth invest in useful projects.
  - NiceWayToDoIT 6y ago
    What is definition of useful? Many successful projects are not necessary useful, but still they create wealth, and many useful project never succeeded to make money to become sustainable... For example many paintings have huge price tag, and we could live without them. Many CO2 removal could devices are very useful but not financially sustainable ...
    - ozim 6y ago
      Let's skip "useful" and just go with projects that earn money for the person that invested in it. Then that person earns more and can hire 2 more devs or 3. When project fails devs have to search for new jobs and investor is probably not going to spend his money as easy again.
dustingetz 6y ago
I think this is a parody?
- rjkennedy98 6y ago
  > In this way the heavy burden of guilt can be lifted from the shoulders of the numerous project managers that have subconsciously devoted their careers to ensuring that projects rarely, if ever, succeed. Was totally thinking the same thing when I read this article. How could the above line be written seriously - PMs don't have guilt about destroying projects!
- readarticle 6y ago
  Some may misinterpret this article as satire. Surely it is not really desirable for software projects to fail. But the facts speak for themselves. Quite fun to read regardless!
- segfaultbuserr 6y ago
  No. I think he's serious. The same author also advocates that the manufacturing and distribution of high-performance processors should be outlawed and regulated as nuclear weapons in order to save humanity from a potential AI singularity [0]. Also, since the computing power we currently have may already be sufficient to run an AGI via distributed computing, destroying existing computers may be necessary. Finally, since he concluded that software projects don't really improve social productivity (in this article), that (high-performance) computers isn't a good idea to humanity in general, therefore, reverting back to the early 90s and asking everyone to use 10 MHz computers is an acceptable solution - you also solve the problem of software bloat simultaneously. In addition, he also suggested that patent trolls might actually be the saviors since they slow down the development of computing and AI. Later he extended these ideas to an entire book [1]. The author's ideas are highly unconventional, but I won't say it's unreasonable or even uncommon in tech - Bill Joy's Why The Future Doesn't Need Us shares similar concerns, the author simply took a much stronger position. [0] https://berglas.org/Articles/AIKillGrandchildren/AIKillGrandchildren.html https://berglas.org/Articles/AIKillGrandchildren/AIKillGrand... [1] http://computersthink.com/ http://computersthink.com/
lordnacho 6y ago
For a recent confirmation of this, witness the enormous amount of effort expended on GPDR compliance. None of it could have been done before computing power became so cheap that it needed to be wasted on doing it. Everyone complains about useless procedures, but I think the real culprit is that we have no framework for discussing value. By that I mean the fundamental economic problem of what should we do, who should do what, how should stuff be shared? In effect our current idea of value is a sort of unstated default: whatever you can get someone to do is valuable, unless it's illegal. And what becomes illegal is also governed by never-stated assumptions. So we end up with this society where anything that can take up productive capacity is done. There's always some sort of justification that might sound reasonable in some context - privacy of personal data is indeed important - but we end up not balancing the costs with the benefits, and you end up with absurd situations where schools can't send out mail to their alumni (yes, this actually happened to me).
- privacy_enjoyer 6y ago
  The reason the GDPR and other privacy legislation was introduced is because of the widespread abuse of people's private data for commercial and political gain. Attempting to address this isn't a waste of time or useless. It makes no sense to blame the people trying to solve this problem, rather than those who created it in the first place. If anything, the GDPR didn't go far enough. Though, that's often the case with legislation trying to stop corporate malfeasance. In my opinion - and speaking more generally, not just regarding privacy - not enough large companies have been obliterated for their harmful behaviours. They just pay a fine, or bribe some officials, and move on.
  - narag 6y ago
    Attempting to address this isn't a waste of time or useless. It is if you fail. It makes no sense to blame the people trying to solve this problem, rather than those who created it in the first place. Now we have two problems: the original problem that GDPR failed to solve and GDPR useless warnings.
    - 6y ago
dgellow 6y ago
The title is missing "(2008)"
perfunctory 6y ago
> As the primary net effect of software is to facilitate bureaucratic complexity it is therefor essential that software projects fail if society is to function effectively. Not only failed projects can prevent bureaucratic complexity, they can also save lives > automated system assigns a unique identifying code to each incoming call before passing it on — a method of keeping track of phone calls as they move through the system. But on April 9, the software responsible for assigning the codes maxed out at a pre-set limit; the counter literally stopped counting at 40 million calls. As a result, the routing system stopped accepting new calls, leading to a bottleneck and a series of cascading failures elsewhere in the 911 infrastructure. Had that automated system failed at the development phase... https://www.washingtonpost.com/news/the-switch/wp/2014/10/20/how-a-dumb-software-glitch-kept-6600-calls-from-getting-to-911/ https://www.washingtonpost.com/news/the-switch/wp/2014/10/20...
- amelius 6y ago
  You can blame the software, but you can also blame the organization for not having a backup system.
  - perfunctory 6y ago
    > Any backup 911 call centers that could have helped with the bottleneck never did, because they themselves were suffering from the same problems everyone else was, according to the FCC. Go figure.
    - ByteJockey 6y ago
      They mean a backup way to take phone calls, not backup 911 centers.
- segfaultbuserr 6y ago
  Wow, it sounds like an apocalyptic story that comp.risks has warned in the 1990s. (From those who don't know: comp.risks was a well-known Usenet forum on the systematic risks of computing technologies in society, and they publish a newsletter since the mid-80s.)
whoisthemachine 6y ago
I find myself fighting this fight all the time - we introduce a new tool, and all of a sudden a dozen processes (bureaucracy in a nutshell) spring up from its outputs or to control its inputs. Unnecessary processes should be considered like unnecessary code - maintenance overhead - and so the usefulness of any process should with some frequency be questioned.
Hokusai 6y ago
Interesting as food for though. Completely absurd in reality. It takes me 1 minute to review, all data is already filled in, my yearly income tax (not much ongoing a part from my salary). And, two clicks in my phone approve, sign and send the declaration to the tax agency. No writing in paper, no looking for numbers, no finding old mail with my payslips. I would avoid living in any country that makes it more complicated that this. If your public administration software sucks, you are voting the wrong people into public office. Do not blame the software.
- hu3 6y ago
  Interesting. What country is that?
  - tdrgabi 6y ago
    It works this way in Sweden. Not sure which country he was referring to.
    - kwhitefoot 6y ago
      And Norway, except that if you do not need to make any changes you do not need to do anything at all.
  - ihaveajob 6y ago
    My taxes in Spain were pretty much like that. You get a letter with the government's estimate. If you agree, you don't have to do anything else.
  - laputan_machine 6y ago
    This could be a number of European countries
  - AnotherGoodName 6y ago
    Could be Australia? Looking at others commenting though could be a lot of other countries. The USAs tax system where the online portion was essentially privatized by a single company (Turbotax) was pretty jarring.
deleted 6y ago
[deleted]
mceldeen 6y ago
This reminds me of the time that I tried to remove a little used feature of a product that I worked on. I could demonstrate that a total of four people had used the feature infrequently in the last year. It took forever to get the approval to remove it. I think that may also contribute to bureaucracy size. It’s next to impossible to remove something from the system as soon as it gets introduced. In other words, bureaucracy is monotonically increasing.
mwcampbell 6y ago
> As the primary net effect of software is to facilitate bureaucratic complexity it is therefor essential that software projects fail if society is to function effectively. This is a short-sighted and overly pessimistic view of the role of software. Let's bring in some sorely needed optimism that's especially relevant to the current pandemic: https://blog.cleancoder.com/uncle-bob/2020/04/05/ANewHope.html https://blog.cleancoder.com/uncle-bob/2020/04/05/ANewHope.ht... Another area where software is actually good for the world is accessibility. Lots of people with disabilities, including me and several of my friends, are able to live independent and happy lives largely thanks to software such as screen readers, speech synthesizers (useful for both blind people and people who can't speak), optical character recognition, and speech recognition. Not to mention that many products and services that are merely convenient for able-bodied people are life-changing for us. So I feel bad for people who are so caught up in the world of software that merely increases bureaucratic complexity, that they've lost sight of all of the good that software is doing, or at least don't get to contribute to that kind of software.
- bxparks 6y ago
  Yes, I think that the highlighted statement is overly hyperbolic. A better one would be something like: "A major effect of software is to facilitate bureaucratic complexity, therefore it is beneficial for some projects to fail." This revised statement would then lead to an interesting discussion about which software projects are beneficial to society and which should be allowed to fail. I would agree that a lot of enterprise software tend to increase workflow complexity. I think that's because in a corporate setting, the people with purchasing powers often don't understand how software technologies works, and they aren't the ones using the system.
- inamiyar 6y ago
  I agree, the accessibility point is important precisely because a lot of the growth of these bureaucracies is to accommodate more people. Being able to identify why these bureaucracies are growing (hint: it's not just because they can) allows us to better solve the problem, which is not software getting better (imo). An old(er) post that is vaguely relevant: https://news.ycombinator.com/item?id=21848860 https://news.ycombinator.com/item?id=21848860 An interesting look at the end of what makes projects slow down, and possibly insights into why bureaucracies grow.
georgeecollins 6y ago
> As the primary net effect of software is to facilitate bureaucratic complexity it is therefor essential that software projects fail if society is to function effectively. What about to play Go, to dispatch Uber drivers, to translate languages, to convert natural language queries into web searches?
ajarmst 6y ago
Red flags: Author has already claimed to be "bold" by the third word of the abstract. Author includes an abstract on a two page self-published paper where they further propose their own name to decorate a newly discovered economic "law". Author uses phrases "statistically significant" and "empirical proof" without betraying any familiarity with their normal meaning in formal prose. And, most importantly, author apparently believes that the observation that efficiencies introduced by new processes are invariably erased by increases in associated bureaucracy was novel and useful in 2008. People have been complaining about that since at least the Roman Republic.
bilater 6y ago
For better reading experience, add this to body in console.log: font-family: sans-serif; width: 80%; margin: auto; color: #1A202C; background-color: #F7FAFC; line-height: 1.5rem;
- steelframe 6y ago
  Is there a Chrome extension that can just do this sort of thing automatically for content that's linked to from forums like HN?
  - bilater 6y ago
    Not that I know of. Wouldn't be hard to build but I don't know how well this generalizes.
perfunctory 6y ago
Satire is at its best when it doesn't obviously look like one.
dang 6y ago
If curious see also 2019 https://news.ycombinator.com/item?id=20109316 https://news.ycombinator.com/item?id=20109316 2009 https://news.ycombinator.com/item?id=932956 https://news.ycombinator.com/item?id=932956
cryptica 6y ago
It's mostly a legal form of money laundering stealing tax payer funds. The Fed prints millions of dollars of fiat currency out of thin air, the government creates some bonds out of thin air and sells them to the Fed for the money they've just printed, then they give it to a big corporation via huge government contracts... The executives then use the free government money to buyback their company shares, which drives up stock prices, which leads to them getting huge bonuses... Then the project fails, the executives take the blame, get a golden handshake worth millions... Then the government will award equally large software contracts to that same company again in the future and repeat.

119 points by kwhitefoot 6y ago

3gg 6y ago

Amazing write-up, some really good kicks in there. The bloat in bureaucracy can be seen in the software itself too. Much of today's consumer software fundamentally performs the same function it did 20 years ago, yet it suffers from the same kind and number of bugs, runs slower, and uses more compute resources. Whatever advances have been in hardware are many times counteracted by the bloat of software. "Below 1% is easily affordable, so the bureaucracy will naturally grow beyond that size as predicted by Parkinson's law" applies to the software itself as much as it does to the bureaucracy it enables.

NiceWayToDoIT 6y ago

In some way, neither one project fails entirely - as it serves to distributes wealth from those that have too much to those that do not have enough. In some sense it would be even better for society if those who have lot of wealth risk even more into wild ideas and moonshot projects.

ozim 6y ago
But that is a broken window fallacy. It would be better for everyone if those who have a lot of wealth invest in useful projects.
- NiceWayToDoIT 6y ago
  What is definition of useful? Many successful projects are not necessary useful, but still they create wealth, and many useful project never succeeded to make money to become sustainable... For example many paintings have huge price tag, and we could live without them. Many CO2 removal could devices are very useful but not financially sustainable ...
  - ozim 6y ago
    Let's skip "useful" and just go with projects that earn money for the person that invested in it. Then that person earns more and can hire 2 more devs or 3. When project fails devs have to search for new jobs and investor is probably not going to spend his money as easy again.

dustingetz 6y ago

I think this is a parody?

rjkennedy98 6y ago
> In this way the heavy burden of guilt can be lifted from the shoulders of the numerous project managers that have subconsciously devoted their careers to ensuring that projects rarely, if ever, succeed. Was totally thinking the same thing when I read this article. How could the above line be written seriously - PMs don't have guilt about destroying projects!
readarticle 6y ago
Some may misinterpret this article as satire. Surely it is not really desirable for software projects to fail. But the facts speak for themselves. Quite fun to read regardless!
segfaultbuserr 6y ago
No. I think he's serious. The same author also advocates that the manufacturing and distribution of high-performance processors should be outlawed and regulated as nuclear weapons in order to save humanity from a potential AI singularity [0]. Also, since the computing power we currently have may already be sufficient to run an AGI via distributed computing, destroying existing computers may be necessary. Finally, since he concluded that software projects don't really improve social productivity (in this article), that (high-performance) computers isn't a good idea to humanity in general, therefore, reverting back to the early 90s and asking everyone to use 10 MHz computers is an acceptable solution - you also solve the problem of software bloat simultaneously. In addition, he also suggested that patent trolls might actually be the saviors since they slow down the development of computing and AI. Later he extended these ideas to an entire book [1]. The author's ideas are highly unconventional, but I won't say it's unreasonable or even uncommon in tech - Bill Joy's Why The Future Doesn't Need Us shares similar concerns, the author simply took a much stronger position. [0] https://berglas.org/Articles/AIKillGrandchildren/AIKillGrandchildren.html https://berglas.org/Articles/AIKillGrandchildren/AIKillGrand... [1] http://computersthink.com/ http://computersthink.com/

lordnacho 6y ago

For a recent confirmation of this, witness the enormous amount of effort expended on GPDR compliance. None of it could have been done before computing power became so cheap that it needed to be wasted on doing it. Everyone complains about useless procedures, but I think the real culprit is that we have no framework for discussing value. By that I mean the fundamental economic problem of what should we do, who should do what, how should stuff be shared? In effect our current idea of value is a sort of unstated default: whatever you can get someone to do is valuable, unless it's illegal. And what becomes illegal is also governed by never-stated assumptions. So we end up with this society where anything that can take up productive capacity is done. There's always some sort of justification that might sound reasonable in some context - privacy of personal data is indeed important - but we end up not balancing the costs with the benefits, and you end up with absurd situations where schools can't send out mail to their alumni (yes, this actually happened to me).

privacy_enjoyer 6y ago
The reason the GDPR and other privacy legislation was introduced is because of the widespread abuse of people's private data for commercial and political gain. Attempting to address this isn't a waste of time or useless. It makes no sense to blame the people trying to solve this problem, rather than those who created it in the first place. If anything, the GDPR didn't go far enough. Though, that's often the case with legislation trying to stop corporate malfeasance. In my opinion - and speaking more generally, not just regarding privacy - not enough large companies have been obliterated for their harmful behaviours. They just pay a fine, or bribe some officials, and move on.
- narag 6y ago
  Attempting to address this isn't a waste of time or useless. It is if you fail. It makes no sense to blame the people trying to solve this problem, rather than those who created it in the first place. Now we have two problems: the original problem that GDPR failed to solve and GDPR useless warnings.
  - 6y ago

dgellow 6y ago

The title is missing "(2008)"

perfunctory 6y ago

> As the primary net effect of software is to facilitate bureaucratic complexity it is therefor essential that software projects fail if society is to function effectively. Not only failed projects can prevent bureaucratic complexity, they can also save lives > automated system assigns a unique identifying code to each incoming call before passing it on — a method of keeping track of phone calls as they move through the system. But on April 9, the software responsible for assigning the codes maxed out at a pre-set limit; the counter literally stopped counting at 40 million calls. As a result, the routing system stopped accepting new calls, leading to a bottleneck and a series of cascading failures elsewhere in the 911 infrastructure. Had that automated system failed at the development phase... https://www.washingtonpost.com/news/the-switch/wp/2014/10/20/how-a-dumb-software-glitch-kept-6600-calls-from-getting-to-911/ https://www.washingtonpost.com/news/the-switch/wp/2014/10/20...

amelius 6y ago
You can blame the software, but you can also blame the organization for not having a backup system.
- perfunctory 6y ago
  > Any backup 911 call centers that could have helped with the bottleneck never did, because they themselves were suffering from the same problems everyone else was, according to the FCC. Go figure.
  - ByteJockey 6y ago
    They mean a backup way to take phone calls, not backup 911 centers.
segfaultbuserr 6y ago
Wow, it sounds like an apocalyptic story that comp.risks has warned in the 1990s. (From those who don't know: comp.risks was a well-known Usenet forum on the systematic risks of computing technologies in society, and they publish a newsletter since the mid-80s.)

whoisthemachine 6y ago

I find myself fighting this fight all the time - we introduce a new tool, and all of a sudden a dozen processes (bureaucracy in a nutshell) spring up from its outputs or to control its inputs. Unnecessary processes should be considered like unnecessary code - maintenance overhead - and so the usefulness of any process should with some frequency be questioned.

Hokusai 6y ago

Interesting as food for though. Completely absurd in reality. It takes me 1 minute to review, all data is already filled in, my yearly income tax (not much ongoing a part from my salary). And, two clicks in my phone approve, sign and send the declaration to the tax agency. No writing in paper, no looking for numbers, no finding old mail with my payslips. I would avoid living in any country that makes it more complicated that this. If your public administration software sucks, you are voting the wrong people into public office. Do not blame the software.

hu3 6y ago
Interesting. What country is that?
- tdrgabi 6y ago
  It works this way in Sweden. Not sure which country he was referring to.
  - kwhitefoot 6y ago
    And Norway, except that if you do not need to make any changes you do not need to do anything at all.
- ihaveajob 6y ago
  My taxes in Spain were pretty much like that. You get a letter with the government's estimate. If you agree, you don't have to do anything else.
- laputan_machine 6y ago
  This could be a number of European countries
- AnotherGoodName 6y ago
  Could be Australia? Looking at others commenting though could be a lot of other countries. The USAs tax system where the online portion was essentially privatized by a single company (Turbotax) was pretty jarring.

deleted 6y ago

[deleted]

mceldeen 6y ago

This reminds me of the time that I tried to remove a little used feature of a product that I worked on. I could demonstrate that a total of four people had used the feature infrequently in the last year. It took forever to get the approval to remove it. I think that may also contribute to bureaucracy size. It’s next to impossible to remove something from the system as soon as it gets introduced. In other words, bureaucracy is monotonically increasing.

mwcampbell 6y ago

> As the primary net effect of software is to facilitate bureaucratic complexity it is therefor essential that software projects fail if society is to function effectively. This is a short-sighted and overly pessimistic view of the role of software. Let's bring in some sorely needed optimism that's especially relevant to the current pandemic: https://blog.cleancoder.com/uncle-bob/2020/04/05/ANewHope.html https://blog.cleancoder.com/uncle-bob/2020/04/05/ANewHope.ht... Another area where software is actually good for the world is accessibility. Lots of people with disabilities, including me and several of my friends, are able to live independent and happy lives largely thanks to software such as screen readers, speech synthesizers (useful for both blind people and people who can't speak), optical character recognition, and speech recognition. Not to mention that many products and services that are merely convenient for able-bodied people are life-changing for us. So I feel bad for people who are so caught up in the world of software that merely increases bureaucratic complexity, that they've lost sight of all of the good that software is doing, or at least don't get to contribute to that kind of software.

bxparks 6y ago
Yes, I think that the highlighted statement is overly hyperbolic. A better one would be something like: "A major effect of software is to facilitate bureaucratic complexity, therefore it is beneficial for some projects to fail." This revised statement would then lead to an interesting discussion about which software projects are beneficial to society and which should be allowed to fail. I would agree that a lot of enterprise software tend to increase workflow complexity. I think that's because in a corporate setting, the people with purchasing powers often don't understand how software technologies works, and they aren't the ones using the system.
inamiyar 6y ago
I agree, the accessibility point is important precisely because a lot of the growth of these bureaucracies is to accommodate more people. Being able to identify why these bureaucracies are growing (hint: it's not just because they can) allows us to better solve the problem, which is not software getting better (imo). An old(er) post that is vaguely relevant: https://news.ycombinator.com/item?id=21848860 https://news.ycombinator.com/item?id=21848860 An interesting look at the end of what makes projects slow down, and possibly insights into why bureaucracies grow.

georgeecollins 6y ago

> As the primary net effect of software is to facilitate bureaucratic complexity it is therefor essential that software projects fail if society is to function effectively. What about to play Go, to dispatch Uber drivers, to translate languages, to convert natural language queries into web searches?

ajarmst 6y ago

Red flags: Author has already claimed to be "bold" by the third word of the abstract. Author includes an abstract on a two page self-published paper where they further propose their own name to decorate a newly discovered economic "law". Author uses phrases "statistically significant" and "empirical proof" without betraying any familiarity with their normal meaning in formal prose. And, most importantly, author apparently believes that the observation that efficiencies introduced by new processes are invariably erased by increases in associated bureaucracy was novel and useful in 2008. People have been complaining about that since at least the Roman Republic.

bilater 6y ago

For better reading experience, add this to body in console.log: font-family: sans-serif; width: 80%; margin: auto; color: #1A202C; background-color: #F7FAFC; line-height: 1.5rem;

steelframe 6y ago
Is there a Chrome extension that can just do this sort of thing automatically for content that's linked to from forums like HN?
- bilater 6y ago
  Not that I know of. Wouldn't be hard to build but I don't know how well this generalizes.

perfunctory 6y ago

Satire is at its best when it doesn't obviously look like one.

dang 6y ago

If curious see also 2019 https://news.ycombinator.com/item?id=20109316 https://news.ycombinator.com/item?id=20109316 2009 https://news.ycombinator.com/item?id=932956 https://news.ycombinator.com/item?id=932956

cryptica 6y ago

It's mostly a legal form of money laundering stealing tax payer funds. The Fed prints millions of dollars of fiat currency out of thin air, the government creates some bonds out of thin air and sells them to the Fed for the money they've just printed, then they give it to a big corporation via huge government contracts... The executives then use the free government money to buyback their company shares, which drives up stock prices, which leads to them getting huge bonuses... Then the project fails, the executives take the blame, get a golden handshake worth millions... Then the government will award equally large software contracts to that same company again in the future and repeat.