12 ms·

It's amazing how often giant IT projects like these go off the rails. Exploding costs and garbage implementations. If you start the sentence with: "A 5+ year go

It's amazing how often giant IT projects like these go off the rails. Exploding costs and garbage implementations. If you start the sentence with: "A 5+ year government IT project that cost over 100 million-" I already know how the story ends. How do we improve this common scenario? What are the root causes? The common themes are: 1. Lack of technical project competence at the decision maker level. 2. Scope creep. Where the one true system has to do everything. 3. A 'one-pass' approach where everything is expected to be delivered as a working system at the end of the project. Even fixing two of these gives us a solid shot at a successful project.

DelightOne 6y ago
D4. Not having the necessary data available through (good) APIs for everybody to use.
908B64B197 6y ago
4. The requirements are often flat out wrong. The Phoenix payroll system comes to mind, the Canadian government tried to shift the blame to IBM, but have their hands tied since IBM delivered exactly what was in the contract. It's just that what the government decided to put in the contract has little to do with how they really do payroll.
- breakfastduck 6y ago
  Basically the story of most govt IT investments.
  - jordanbeiber 6y ago
    Have you worked at big companies outside tech? Their IT budgets are often ginormous with very little to show for it. Consulting havens. Slow, if at all, moving project organizations. It’s about size, budget models and competences. Time for another rant: Note that Sweden have implemented “new public management” NPM which basically pushes government agencies to govern like the ever oh so successful free market companies. This has had many really bad side effects since the 90s and it’s right up the neoliberal alley. In my book it’s just silly and something only professional politicians together with consultants can cook. But it’s a different story, really.
  - lostlogin 6y ago
    This sort of anti-government comment seems to crop up regularly. My observation of corporate organisations attempting to guide software development is just as bad. I’m a proponent of doing these things in-house, however that path is far from straightforward and has masses of pitfalls too.
- genuine_smiles 6y ago
  > 4. The requirements are often flat out wrong. Isn't this basically a given? I find it hard to imagine that any organization could come up with good, complete requirements before they've had any software written.
  - jussij 6y ago
    Which is why most governments struggle with big IT projects. Governments by their nature tend to approach everything from a legal perspective. This then means the requirements of these big IT project end up being a mass of legal documents which try to describe what is being delivered by whom. Then when the whole thing falls apart it ends up in the courts and the court then decides who promised what based on those original contract documents.
    - jordanbeiber 6y ago
      Everyone struggles with big IT projects. It’s also why Gene Kim & co wrote “The Phoenix Project” [1]. Everyone involved in software-building, non-tech industry should read it. In the end it’s just lean turned agile software dev. Reduce waste. [1] https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592 https://www.amazon.com/Phoenix-Project-DevOps-Helping-Busine...
    - 6pac3rings 6y ago
      Take a triple bottom line solution to the problem. Take a look at how kmalloc_obj is going in DragonFly BSD. Government IT challenges three technical team leads to get to solution like kmalloc_obj. Performance pays for 1st prize and the rest get less money. Cut the time horizon from start to finish for the piece work to 18-months. Spread the risk of total failure to zero.
    - lostlogin 6y ago
      I was in a pub with a relative, and we got chatting to another random patron. The relative was asked about his CS job, and at some point details were being discussed. The relative said something like “we have made what was asked for but because we have run out of time, that’s what the customer is getting. We know what they actually want and need, but that’s not in the contract”. The person we were talking to was the customer.
- mrweasel 6y ago
  The Danish government tried to getting a new tax collection system built. The specificatio was 6000 page, not including the tax law that needed to be implemented. It facinates me that no one ever stopped to wonder if was actually possible to implement all that. It failed horribly, costing billion of DKK in implementation cost and even more in taxes that couldn’t be collected.
  - roncohen 6y ago
    The other problem here is that politicians are either oblivious or simple don't care about the cost associated with complexity in the laws. On top of that, the other unfortunate dynamic that exists is that politicians all work to get their little special case into new bills so they can point to that when their constituents ask how they influenced the new law. Taken together, the complexity of the law just accumulates. Tax law in Denmark being a particularly gross example.
    - mrweasel 6y ago
      I don't doubt that writing a new laws is complicated, but if I write code, and realise that I constantly need to tack on more code to handle special cases, then I will normally revisits my design. It is not my impression that politicians do the same with laws. One major issue, at least with Danish law makers, is that they want to target special groups, but that would be discriminating, so instead they attempt to target the behaviour of those group. This of cause will affect a number of people who where not in the original target group, so they add on exceptions and and details to narrow down the law. Also there never seems to be any clean up in the laws. If your laws/rule exploded from a few hundred pages to 30.000 pages, you should really revisit that thoughts behind that law.
      - reacharavindh 6y ago
        It's crazy out here in DK. The most embarrassing example of such crazy targeted laws is the "hand shake with the mayor" requirement for new Danish citizens, clearly target at a certain ethnicity that would hesitate to do just that. The intent is to make it difficult to that ethnicity, the result is just a veiled law that makes them "appear" just enough secular and non-discriminatory. There's a lot I love about life in Denmark. This is just to say that no place is perfect and DK also has stuff that all Danes should be rightly embarrassed about.
- magicalhippo 6y ago
  I was hired as a coder on a fun project like that. It wasn't a big project, I was the sole coder. What had been sold in was basically a Drupal install with some customization. I made sure they wrote a decent specification before I accepted the job. I delivered on time and we had the first test with the client. Everything went very well, and the client seemed happy. On the client side, the project was then moved from the project group to those who'd actually be using it. And then came the question from the new manager: Mgr: "This looks nice, but what about all the other sites?" Us: "Other sites? The contract was only for one site." Mgr: "Well, the whole point here was to have 17 sites with site-specific content written by site-specific users, managed centrally with a unified look as if it was one single site." Us: "Err... that's not what the specification we agreed on says." Mgr: "Well, as it stands this is useless to us." And so the simple three day job turned into many weeks.
  - _0ffh 6y ago
    Many appropriately paid weeks, I sincerely hope for you!?
    - magicalhippo 6y ago
      Since the specifications were sufficiently clear, the client could indeed see rather easily that the solution they wanted was not what was agreed upon. Had I not insisted on such detailed specs though, it would likely have been a lot harder to get them to pay up. So yeah, worked out fine for me, but taught me a valuable lesson.
- owroomexorcist 6y ago
  And guess who just bought another expensive IBM solution for payment management?
ilaksh 6y ago
There should be laws the require the use of open distributed extensible protocols. That means the government comes up with a way for the parties to collaborate and just enforces that. So the actual implementation is open to competition. Things like communication protocols or extensible APIs or schemas for exchanging APIs.
AdrianB1 6y ago
Cities are ran by politicians that in most cases are too incompetent to do anything better with their lives. When incompetent leaders make decisions, the incompetence is flowing to most, if not everything they do, including this kind of projects. There is no immediate fix for this situation, unless the decision would be made by some kind of city manager or board of experts that are recruited based on competence and held accountable for the results (up to prison, if needed). The old story with "politicians pay with their mandate, they will not be reelected if they fail" is a story for toddlers, that is not a punishment on par with the damages they make, will the Stockholm mayor pay back 100 million dollars?
- anm89 6y ago
  I'm sure people will be lining up to take these board positions when the punishment for project failure is prison. Maybe that's a little far, no?
  - hutzlibu 6y ago
    Or why not execute them instead? Worked very well for ages ... More seriously, there usually are no simple solutions to complex problems. And government is a very complex problem. So many people with so many opinions - and everyone involved afraid to say one wrong word or make one wrong decision. I don't think more fear helps there. I mean - prison for corruption - yes! But prison for incompetence, no. Then you also have to jail the people who put the incompetent person there in the first place and those people and so on.
    - juskrey 6y ago
      Just take back the salary when the city is net negative from their "experiments"
      - hutzlibu 6y ago
        Yeah, but would you like to have your salary reclaimed later on, if they are not satisfied with your work? Could you do calmy your job then? Or would the anxiety made you even more prone to misstakes?
peteretep 6y ago
Also internal company politics. What would fix much of this, for many projects, is a very experienced product person with final say over every feature, reporting directly to the CEO / board, and outranking everyone else in terms of decision making.
- hutzlibu 6y ago
  Sure, having skilled people with the right to make decisions solves almost everything. But who decides who is skilled enough? Then we are quickly in the realm of politics ..
  - utrechtsouth 6y ago
    > But who decides who is skilled enough? Trial and error using a free market place.
    - hutzlibu 6y ago
      When we are talking about centralized government institutions - then we kind of left the free market place.
      - utrechtsouth 6y ago
        Well, this thread gives an example of a private initiative that provides a better alternative. :)
        hutzlibu 6y ago
        And the mandatory governments solution trying to block it.
        utrechtsouth 6y ago
        It's not mandatory. Actually it's likely illegal.
jordanbeiber 6y ago
Projects are necessary for things that are built to last and not change much during their lifetime: - a bridge - airplanes - most houses - etc Hardware comes to mind - it’s all hardware on the list, basically. Software (outside certain realms ofc) like this? I’ve been, like many others here, doing this software thing for 20+ years now. Big and small, I’ve basically never seen anything spawned from a project-driven organization actually deliver great results. Most software is supposed to change, indefinitely - that’s the point! Everyone in this day and age should know that requirements change over even short periods of time, so why even bother trying to pin them down in detail up front - you’re going to do everyone involved a disservice. There is something to this agile thing and a “project” is it’s anti-pattern. Not mention how much a quick feedback loop will learn you about the operation side of things. Operations and change, it’s all you can build for and that is best done one step at a time. (This joke of a platform is spread across multiple (5?) vendors/partners no less. A couple of them probably started just for this, backed with vc funding. It’s most likely a glorious mess!) End rant.
- globular-toast 6y ago
  The trick is to make your projects tiny, not big. Instead of "build an all-encompassing system that does everything" you do "solve this very specific business problem". You could call these little projects sprints or something.
  - jordanbeiber 6y ago
    Think big, act small comes to mind. No tricks really, just common sense, right? :)
  - bborud 6y ago
    Oh, be careful. At my previous employer there was the concept of the "90 day sprint". Top management use the term without irony. All they have done is substitute "sprint" anywhere you would say "quarter". So now they are agile. I wish I was making this up.
    - 6y ago
cfn 6y ago
I once interviewed with a company that had been hired by the UK government to create a system for one of their agencies. They told me that the system had 1000 requirements (literally) had already been implemented once and failed but they wanted to have another go. Boggles the mind (and I refused their offer).
zhte415 6y ago
Just have everyone do PMP. Project management problems solved.
anvemaha 6y ago
In case anyone wants to read some research on the success rate of IT projects Standish Group's Chaos Reports are a good place to start. I read the 2014 one as a part of our project management uni course, but couldn't find it with a simple Google search so here's the 2015 one: [pdf] https://www.standishgroup.com/sample_research_files/CHAOSReport2015-Final.pdf https://www.standishgroup.com/sample_research_files/CHAOSRep...
- jordanbeiber 6y ago
  Many projects push technical debt and cut the wrong corners to stay on target. It looks like a success until you ask operations and people maintaining the deliverable. This is usually not factored in and ends up at at different cost center. You almost never get the full picture of continuity when dealing with a project organization hence it’s really, really hard to judge.
throw0101a 6y ago
> It's amazing how often giant IT projects like these go off the rails. Exploding costs and garbage implementations. If you start the sentence with: "A 5+ year government IT project that cost over 100 million-" I already know how the story ends. People don't notice when things go right. What about all the 5+ year, $100M projects you didn't hear about, because they never made the news, because the project went smoothly? Even in general conversation we tend to vent about how bad our day/week was, and not how awesome something went: * https://en.wikipedia.org/wiki/Negativity_bias https://en.wikipedia.org/wiki/Negativity_bias
daniellarusso 6y ago
This article and report was from awhile ago, but it was around only 1/3 of IT projects are completed ‘successfully’: https://www.zdnet.com/article/study-68-percent-of-it-projects-fail/ https://www.zdnet.com/article/study-68-percent-of-it-project...

DelightOne 6y ago

D4. Not having the necessary data available through (good) APIs for everybody to use.

908B64B197 6y ago

4. The requirements are often flat out wrong. The Phoenix payroll system comes to mind, the Canadian government tried to shift the blame to IBM, but have their hands tied since IBM delivered exactly what was in the contract. It's just that what the government decided to put in the contract has little to do with how they really do payroll.

breakfastduck 6y ago
Basically the story of most govt IT investments.
- jordanbeiber 6y ago
  Have you worked at big companies outside tech? Their IT budgets are often ginormous with very little to show for it. Consulting havens. Slow, if at all, moving project organizations. It’s about size, budget models and competences. Time for another rant: Note that Sweden have implemented “new public management” NPM which basically pushes government agencies to govern like the ever oh so successful free market companies. This has had many really bad side effects since the 90s and it’s right up the neoliberal alley. In my book it’s just silly and something only professional politicians together with consultants can cook. But it’s a different story, really.
- lostlogin 6y ago
  This sort of anti-government comment seems to crop up regularly. My observation of corporate organisations attempting to guide software development is just as bad. I’m a proponent of doing these things in-house, however that path is far from straightforward and has masses of pitfalls too.
genuine_smiles 6y ago
> 4. The requirements are often flat out wrong. Isn't this basically a given? I find it hard to imagine that any organization could come up with good, complete requirements before they've had any software written.
- jussij 6y ago
  Which is why most governments struggle with big IT projects. Governments by their nature tend to approach everything from a legal perspective. This then means the requirements of these big IT project end up being a mass of legal documents which try to describe what is being delivered by whom. Then when the whole thing falls apart it ends up in the courts and the court then decides who promised what based on those original contract documents.
  - jordanbeiber 6y ago
    Everyone struggles with big IT projects. It’s also why Gene Kim & co wrote “The Phoenix Project” [1]. Everyone involved in software-building, non-tech industry should read it. In the end it’s just lean turned agile software dev. Reduce waste. [1] https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592 https://www.amazon.com/Phoenix-Project-DevOps-Helping-Busine...
  - 6pac3rings 6y ago
    Take a triple bottom line solution to the problem. Take a look at how kmalloc_obj is going in DragonFly BSD. Government IT challenges three technical team leads to get to solution like kmalloc_obj. Performance pays for 1st prize and the rest get less money. Cut the time horizon from start to finish for the piece work to 18-months. Spread the risk of total failure to zero.
  - lostlogin 6y ago
    I was in a pub with a relative, and we got chatting to another random patron. The relative was asked about his CS job, and at some point details were being discussed. The relative said something like “we have made what was asked for but because we have run out of time, that’s what the customer is getting. We know what they actually want and need, but that’s not in the contract”. The person we were talking to was the customer.
mrweasel 6y ago
The Danish government tried to getting a new tax collection system built. The specificatio was 6000 page, not including the tax law that needed to be implemented. It facinates me that no one ever stopped to wonder if was actually possible to implement all that. It failed horribly, costing billion of DKK in implementation cost and even more in taxes that couldn’t be collected.
- roncohen 6y ago
  The other problem here is that politicians are either oblivious or simple don't care about the cost associated with complexity in the laws. On top of that, the other unfortunate dynamic that exists is that politicians all work to get their little special case into new bills so they can point to that when their constituents ask how they influenced the new law. Taken together, the complexity of the law just accumulates. Tax law in Denmark being a particularly gross example.
  - mrweasel 6y ago
    I don't doubt that writing a new laws is complicated, but if I write code, and realise that I constantly need to tack on more code to handle special cases, then I will normally revisits my design. It is not my impression that politicians do the same with laws. One major issue, at least with Danish law makers, is that they want to target special groups, but that would be discriminating, so instead they attempt to target the behaviour of those group. This of cause will affect a number of people who where not in the original target group, so they add on exceptions and and details to narrow down the law. Also there never seems to be any clean up in the laws. If your laws/rule exploded from a few hundred pages to 30.000 pages, you should really revisit that thoughts behind that law.
    - reacharavindh 6y ago
      It's crazy out here in DK. The most embarrassing example of such crazy targeted laws is the "hand shake with the mayor" requirement for new Danish citizens, clearly target at a certain ethnicity that would hesitate to do just that. The intent is to make it difficult to that ethnicity, the result is just a veiled law that makes them "appear" just enough secular and non-discriminatory. There's a lot I love about life in Denmark. This is just to say that no place is perfect and DK also has stuff that all Danes should be rightly embarrassed about.
magicalhippo 6y ago
I was hired as a coder on a fun project like that. It wasn't a big project, I was the sole coder. What had been sold in was basically a Drupal install with some customization. I made sure they wrote a decent specification before I accepted the job. I delivered on time and we had the first test with the client. Everything went very well, and the client seemed happy. On the client side, the project was then moved from the project group to those who'd actually be using it. And then came the question from the new manager: Mgr: "This looks nice, but what about all the other sites?" Us: "Other sites? The contract was only for one site." Mgr: "Well, the whole point here was to have 17 sites with site-specific content written by site-specific users, managed centrally with a unified look as if it was one single site." Us: "Err... that's not what the specification we agreed on says." Mgr: "Well, as it stands this is useless to us." And so the simple three day job turned into many weeks.
- _0ffh 6y ago
  Many appropriately paid weeks, I sincerely hope for you!?
  - magicalhippo 6y ago
    Since the specifications were sufficiently clear, the client could indeed see rather easily that the solution they wanted was not what was agreed upon. Had I not insisted on such detailed specs though, it would likely have been a lot harder to get them to pay up. So yeah, worked out fine for me, but taught me a valuable lesson.
owroomexorcist 6y ago
And guess who just bought another expensive IBM solution for payment management?

ilaksh 6y ago

There should be laws the require the use of open distributed extensible protocols. That means the government comes up with a way for the parties to collaborate and just enforces that. So the actual implementation is open to competition. Things like communication protocols or extensible APIs or schemas for exchanging APIs.

AdrianB1 6y ago

Cities are ran by politicians that in most cases are too incompetent to do anything better with their lives. When incompetent leaders make decisions, the incompetence is flowing to most, if not everything they do, including this kind of projects. There is no immediate fix for this situation, unless the decision would be made by some kind of city manager or board of experts that are recruited based on competence and held accountable for the results (up to prison, if needed). The old story with "politicians pay with their mandate, they will not be reelected if they fail" is a story for toddlers, that is not a punishment on par with the damages they make, will the Stockholm mayor pay back 100 million dollars?

anm89 6y ago
I'm sure people will be lining up to take these board positions when the punishment for project failure is prison. Maybe that's a little far, no?
- hutzlibu 6y ago
  Or why not execute them instead? Worked very well for ages ... More seriously, there usually are no simple solutions to complex problems. And government is a very complex problem. So many people with so many opinions - and everyone involved afraid to say one wrong word or make one wrong decision. I don't think more fear helps there. I mean - prison for corruption - yes! But prison for incompetence, no. Then you also have to jail the people who put the incompetent person there in the first place and those people and so on.
  - juskrey 6y ago
    Just take back the salary when the city is net negative from their "experiments"
    - hutzlibu 6y ago
      Yeah, but would you like to have your salary reclaimed later on, if they are not satisfied with your work? Could you do calmy your job then? Or would the anxiety made you even more prone to misstakes?

peteretep 6y ago

Also internal company politics. What would fix much of this, for many projects, is a very experienced product person with final say over every feature, reporting directly to the CEO / board, and outranking everyone else in terms of decision making.

hutzlibu 6y ago
Sure, having skilled people with the right to make decisions solves almost everything. But who decides who is skilled enough? Then we are quickly in the realm of politics ..
- utrechtsouth 6y ago
  > But who decides who is skilled enough? Trial and error using a free market place.
  - hutzlibu 6y ago
    When we are talking about centralized government institutions - then we kind of left the free market place.
    - utrechtsouth 6y ago
      Well, this thread gives an example of a private initiative that provides a better alternative. :)
      - hutzlibu 6y ago
        And the mandatory governments solution trying to block it.
        utrechtsouth 6y ago
        It's not mandatory. Actually it's likely illegal.

jordanbeiber 6y ago

Projects are necessary for things that are built to last and not change much during their lifetime: - a bridge - airplanes - most houses - etc Hardware comes to mind - it’s all hardware on the list, basically. Software (outside certain realms ofc) like this? I’ve been, like many others here, doing this software thing for 20+ years now. Big and small, I’ve basically never seen anything spawned from a project-driven organization actually deliver great results. Most software is supposed to change, indefinitely - that’s the point! Everyone in this day and age should know that requirements change over even short periods of time, so why even bother trying to pin them down in detail up front - you’re going to do everyone involved a disservice. There is something to this agile thing and a “project” is it’s anti-pattern. Not mention how much a quick feedback loop will learn you about the operation side of things. Operations and change, it’s all you can build for and that is best done one step at a time. (This joke of a platform is spread across multiple (5?) vendors/partners no less. A couple of them probably started just for this, backed with vc funding. It’s most likely a glorious mess!) End rant.

globular-toast 6y ago
The trick is to make your projects tiny, not big. Instead of "build an all-encompassing system that does everything" you do "solve this very specific business problem". You could call these little projects sprints or something.
- jordanbeiber 6y ago
  Think big, act small comes to mind. No tricks really, just common sense, right? :)
- bborud 6y ago
  Oh, be careful. At my previous employer there was the concept of the "90 day sprint". Top management use the term without irony. All they have done is substitute "sprint" anywhere you would say "quarter". So now they are agile. I wish I was making this up.
  - 6y ago

cfn 6y ago

I once interviewed with a company that had been hired by the UK government to create a system for one of their agencies. They told me that the system had 1000 requirements (literally) had already been implemented once and failed but they wanted to have another go. Boggles the mind (and I refused their offer).

zhte415 6y ago

Just have everyone do PMP. Project management problems solved.

anvemaha 6y ago

In case anyone wants to read some research on the success rate of IT projects Standish Group's Chaos Reports are a good place to start. I read the 2014 one as a part of our project management uni course, but couldn't find it with a simple Google search so here's the 2015 one: [pdf] https://www.standishgroup.com/sample_research_files/CHAOSReport2015-Final.pdf https://www.standishgroup.com/sample_research_files/CHAOSRep...

jordanbeiber 6y ago
Many projects push technical debt and cut the wrong corners to stay on target. It looks like a success until you ask operations and people maintaining the deliverable. This is usually not factored in and ends up at at different cost center. You almost never get the full picture of continuity when dealing with a project organization hence it’s really, really hard to judge.

throw0101a 6y ago

> It's amazing how often giant IT projects like these go off the rails. Exploding costs and garbage implementations. If you start the sentence with: "A 5+ year government IT project that cost over 100 million-" I already know how the story ends. People don't notice when things go right. What about all the 5+ year, $100M projects you didn't hear about, because they never made the news, because the project went smoothly? Even in general conversation we tend to vent about how bad our day/week was, and not how awesome something went: * https://en.wikipedia.org/wiki/Negativity_bias https://en.wikipedia.org/wiki/Negativity_bias

daniellarusso 6y ago

This article and report was from awhile ago, but it was around only 1/3 of IT projects are completed ‘successfully’: https://www.zdnet.com/article/study-68-percent-of-it-projects-fail/ https://www.zdnet.com/article/study-68-percent-of-it-project...