8 ms·

The staggering impact of IT systems gone wrong

34 points by lukebennett 11y ago

rabbiterm 11y ago
I clicked on this curious as to whether I had worked on any of these. And yep. 3 of them -- 3 out of the top 4 worst in the Americas. The DoD has some successes though.
- ep103 11y ago
  well, and this is just a personal suggestion, maybe you should stop sabotaging the companies you work for? Or hire yourself out for your ability to sabotage competitors? : D
dijit 11y ago
But there is a profession dedicated to preventing these kinds of failures: Systems Administrators. Proper infrastructure engineers (or "site reliability engineers" as google now calls them) are key for helping architect highly available systems like this; Yet the trend seems to be forcing software engineers follow tutorials from google on setting up key pieces of infrastructure nowadays. Edit: it's ettiquite to say why you're going to downvote someone, if what I said wasn't relevant please let me know.
- falcolas 11y ago
  Haven't you heard? With Amazon and Heroku and Google, SREs are obsolete! Your developers can manage their own infrastructure. /s Our own company learned the truth about requiring an operations team the easy way - they hired some and their reliability went up. Outages started being handled before it impacted customers. It's been fun watching the volume of "Oh Shit" emails dropping precipitously.
- mannykannot 11y ago
  I did not downvote you, but I disagree with your premise. Systems administration is only part of the problem, and only in some cases. A non-inclusive list of other causes include failing to understand the intended purpose of a system and the environment it is to operate in, and losing control of the design.
  - dijit 11y ago
    I would agree with what you said, but I would argue that the points you raised still fall into the realm of systems administration. Identifying scope and purpose, and it's impact and documenting the design are just some of the things an platform operations team is -supposed- to take care of.
    - mannykannot 11y ago
      If you expand the definition of systems administration to include everything, it doesn't get you any closer to a solution. The largest problems of development have nothing to do with administering anything (edit: for example, bad design often results from bad design decisions that are neither prevented nor fixed by seeing that the required documents have been written.) The idea that Winslow Taylor's principles can solve these problems is a fallacy - a commonly held one, but a fallacy nonetheless.
i_feel_great 11y ago
The Queensland Health payroll failure is now estimated to be $AU1.2b all up, with nothing to show for it.
- joshuarrrr 11y ago
  One of the authors here- we have another graphic coming about Queensland Health. The craziest part was when IBM estimated it at $6 million.
  - i_feel_great 11y ago
    $6m to $1.2b cost overrun. That must be some type of record.
    - joshuarrrr 11y ago
      Yeah, it seems like it. You can see how the budget ballooned here: http://spectrum.ieee.org/static/the-life-cycles-of-failed-projects http://spectrum.ieee.org/static/the-life-cycles-of-failed-pr... (select the 5th project)
jmnicolas 11y ago
I believe that these complex multi-billion $ projects fail for 2 reasons : - corruption : "let's milk the customer for all they have" - personal ambition : "let's choose what's best for my career instead of what's best for the project" Unless we radically change the way we do things the IEEE will need bigger and bigger green circles on their charts.
- awakeasleep 11y ago
  You're definitely falling prey to a short-sighted and overly cynical point of view. "Never attribute to malice that which is adequately explained by stupidity" and never use the stupidity explination to overlook the fundamental complexity of a software project. What you have here is a lot of non IT professionals, without siginificant IT experience, attempting the most complex IT projects ever considered, without the pressures of ship-or-sink, investors looking for progress reports, or even the benefit of working with their own employees (usually multiple groups of contractors work on these big projects). Also, they're not trying to build a MVP and iterate. They're trying to build Rome in a few years. When you think about the hurdles these projects face, of course they fail. The government trying to do a huge software project is like a kid seeing a master level parkour video on Youtube and not understanding the lifetime of practice that went into the sport. They're just going to jump off the roof and break their ankle. Of course in a sample size this large corruption is an issue. However, with zero corruption and only the best intentions, we'd see substantially the same results.
  - tomaskafka 11y ago
    Also, “Never attribute to stupidity that which is adequately explained by complexity.” http://www.cennydd.com/blog/its-not-what-you-think http://www.cennydd.com/blog/its-not-what-you-think
    - devonkim 11y ago
      Only those that haven't worked in DoD / federal systems could ever think that laziness / stupidity is not a major root cause of so many DoD IT project failures. There's a lot of successes when using smaller, elite contractors but complexity intertwines with stupidity as larger contractors must be brought in to handle the bureaucracy and their hiring standards are basically abysmal. Why? They were very much incentivized to hire butts in seats on contracts historically, not to be effective at executing them to technical satisfaction. Someone needs to call out DoD for what it really is in the US - federal Public Works program that's approved by neoconservatives.
ArkyBeagle 11y ago
1st Lt. Milo Minderbinder: Is it really that bad? Yossarian: It's cotton! 1st Lt. Milo Minderbinder: They've got to learn to like it!

34 points by lukebennett 11y ago

I clicked on this curious as to whether I had worked on any of these. And yep. 3 of them -- 3 out of the top 4 worst in the Americas. The DoD has some successes though.

ep103 11y ago
well, and this is just a personal suggestion, maybe you should stop sabotaging the companies you work for? Or hire yourself out for your ability to sabotage competitors? : D

dijit 11y ago

But there is a profession dedicated to preventing these kinds of failures: Systems Administrators. Proper infrastructure engineers (or "site reliability engineers" as google now calls them) are key for helping architect highly available systems like this; Yet the trend seems to be forcing software engineers follow tutorials from google on setting up key pieces of infrastructure nowadays. Edit: it's ettiquite to say why you're going to downvote someone, if what I said wasn't relevant please let me know.

falcolas 11y ago
Haven't you heard? With Amazon and Heroku and Google, SREs are obsolete! Your developers can manage their own infrastructure. /s Our own company learned the truth about requiring an operations team the easy way - they hired some and their reliability went up. Outages started being handled before it impacted customers. It's been fun watching the volume of "Oh Shit" emails dropping precipitously.
mannykannot 11y ago
I did not downvote you, but I disagree with your premise. Systems administration is only part of the problem, and only in some cases. A non-inclusive list of other causes include failing to understand the intended purpose of a system and the environment it is to operate in, and losing control of the design.
- dijit 11y ago
  I would agree with what you said, but I would argue that the points you raised still fall into the realm of systems administration. Identifying scope and purpose, and it's impact and documenting the design are just some of the things an platform operations team is -supposed- to take care of.
  - mannykannot 11y ago
    If you expand the definition of systems administration to include everything, it doesn't get you any closer to a solution. The largest problems of development have nothing to do with administering anything (edit: for example, bad design often results from bad design decisions that are neither prevented nor fixed by seeing that the required documents have been written.) The idea that Winslow Taylor's principles can solve these problems is a fallacy - a commonly held one, but a fallacy nonetheless.

i_feel_great 11y ago

The Queensland Health payroll failure is now estimated to be $AU1.2b all up, with nothing to show for it.

joshuarrrr 11y ago
One of the authors here- we have another graphic coming about Queensland Health. The craziest part was when IBM estimated it at $6 million.
- i_feel_great 11y ago
  $6m to $1.2b cost overrun. That must be some type of record.
  - joshuarrrr 11y ago
    Yeah, it seems like it. You can see how the budget ballooned here: http://spectrum.ieee.org/static/the-life-cycles-of-failed-projects http://spectrum.ieee.org/static/the-life-cycles-of-failed-pr... (select the 5th project)

jmnicolas 11y ago

I believe that these complex multi-billion $ projects fail for 2 reasons : - corruption : "let's milk the customer for all they have" - personal ambition : "let's choose what's best for my career instead of what's best for the project" Unless we radically change the way we do things the IEEE will need bigger and bigger green circles on their charts.

awakeasleep 11y ago
You're definitely falling prey to a short-sighted and overly cynical point of view. "Never attribute to malice that which is adequately explained by stupidity" and never use the stupidity explination to overlook the fundamental complexity of a software project. What you have here is a lot of non IT professionals, without siginificant IT experience, attempting the most complex IT projects ever considered, without the pressures of ship-or-sink, investors looking for progress reports, or even the benefit of working with their own employees (usually multiple groups of contractors work on these big projects). Also, they're not trying to build a MVP and iterate. They're trying to build Rome in a few years. When you think about the hurdles these projects face, of course they fail. The government trying to do a huge software project is like a kid seeing a master level parkour video on Youtube and not understanding the lifetime of practice that went into the sport. They're just going to jump off the roof and break their ankle. Of course in a sample size this large corruption is an issue. However, with zero corruption and only the best intentions, we'd see substantially the same results.
- tomaskafka 11y ago
  Also, “Never attribute to stupidity that which is adequately explained by complexity.” http://www.cennydd.com/blog/its-not-what-you-think http://www.cennydd.com/blog/its-not-what-you-think
  - devonkim 11y ago
    Only those that haven't worked in DoD / federal systems could ever think that laziness / stupidity is not a major root cause of so many DoD IT project failures. There's a lot of successes when using smaller, elite contractors but complexity intertwines with stupidity as larger contractors must be brought in to handle the bureaucracy and their hiring standards are basically abysmal. Why? They were very much incentivized to hire butts in seats on contracts historically, not to be effective at executing them to technical satisfaction. Someone needs to call out DoD for what it really is in the US - federal Public Works program that's approved by neoconservatives.

ArkyBeagle 11y ago

1st Lt. Milo Minderbinder: Is it really that bad? Yossarian: It's cotton! 1st Lt. Milo Minderbinder: They've got to learn to like it!