5 ms·

There's a dimension left out here on the Phoenix disaster: It wasn't really IBM's fault at all. The Canadian public service is really complex. There are multip

by Naga 9y ago

There's a dimension left out here on the Phoenix disaster: It wasn't really IBM's fault at all. The Canadian public service is really complex. There are multiple unions with multiple overlapping collective bargaining agreements, where the public service is allocated to different classes. These classes are paid specific rates, with retroactive pay being common for changing classes. The majority of the problems with Phoenix have been employees moving from their classes and being paid the correct amount. It has also adversely affected non-unionized positions. My understanding is that the Harper government (Prime Minister until 2015), who was responsible for the negotiation and for laying out the requirements, was trying to save a money and not responibly create a pay system. Two major factors that jump out at me: 1) Requirements did not call for training. The system was implemented and IBM was not required to train any operators on how the system functions, which is important because all new staff were hired to run the system and, 2) Due to the need for cost saving measures (the government was trying really hard to balance the budget, as the election was coming up), the previous payroll staff were terminated and a new payroll centre was opened in Miramichi, which is a small town in the middle of nowhere. So, on top of new software, the government lost all of its institutional knowledge regarding payroll and how things are supposed to work. It's actually hard to say how much of this is IBM's fault and how much is the governments, because the government doesn't know how to fix it. No one knows how Phoenix works and no one knows how it is supposed to work. It's just a big mess with no end in sight. Could IBM have done a better job? Probably, but garbage in, garbage out. For further reading, the Auditor-General's report: http://www.oag-bvg.gc.ca/internet/English/parl_oag_201711_01_e_42666.html http://www.oag-bvg.gc.ca/internet/English/parl_oag_201711_01...

drieddust 9y ago
As someone who works in same space, I can say this is happening because unrealistic expectations of customers are not challenged by bidding Vendors because they fear losing the project. This in turns happens because customer tries to outsmart vendors by pressing them for reducing the timelines to absolute impossible. If any vendor tries to play by the rules, they are thrown out. Smart vendors respond by crafting the contract smarty. Skipping training is one such example. It was crucial but .... In the end project is either canceled or goes over budged or barely manage to deliver anything.
- dv_dt 9y ago
  Yup, this kind of contracts game is what really needs modern innovation. I think the space is kind of living in an extensions of 70's assumptions with many decades of beauracratic rules added on to try to fix up fundamental problem. For another factor, it would be lower risk to allocate into many smaller projects and sign contracts in phases and sub-systems - but then the funding of the total project becomes more at risk politically, and then there is a higher technical load on the contracting dept to architect the boundaries.
  - drieddust 9y ago
    They are barely able to state the outcome of project so expecting them to do the necessary legwork to break projects into logical subprojects is too much to ask. No disrespect but we have come to a point where IT have started appearing magical to pointy hairy bosses who think juse because it's software everything is infinitely malleable with no impact on quality, cost, or time.
    - dv_dt 9y ago
      If you don't have an organization that can break it down to a smaller projects then there is a question if doing it as one large project has a higher or lower chance of failure.
      - drieddust 9y ago
        Shouldn't that depend upon complexity of change but who cares. CIO have a short tenure so collecting the bonuses and leaving before house of card collapses works. It's all about managing the personal bonus and career, who cares for the success 9f project?
        dv_dt 9y ago
        It depends on complexity, but I think it's more of a situation akin to refactoring or rewriting a complex code base. A total rewrite is always an attractive thought, but a real focus on continuous refactoring in smaller portion at a time is often lower risk, lower cost, and faster in the end - and I suspect it's the same with large bureaucratic processes too. It's also a way to sharpen the skills of the organization in small steps with lower risks for failure (and improving future project steps...).
toast0 9y ago
At some level, isn't IBM supposed to point out that training and transition are important for a large project and shouldn't be left out?
- coldacid 9y ago
  Sure, if they don't want to be the winning (read: lowest) bidder. Shit constraints --> shit results.
  - deleted 9y ago
    [deleted]
- otoburb 9y ago
  They most likely did raise this, but when asked to provide a more "competitive" bid, they axed training and who knows what else. This is common in large enterprise sales deals. For professional services deployment engagements, there is always the possibility of an amendment to the services scope (often referred to as a change order or change request). If the project is estimated to take over a year to deploy, by then you're in a new budget cycle and the vendor has come to an agreement with the procurement department to charge for additional services as required in subsequent phases.
  - LifeLiverTransp 9y ago
    Another nice thing about governments is that they do not deal in monetary damages but in media fallout- so something loud (collapse of a billion dollar project) is worser to them, then silently going over budget because additional points appear on the list. So biding yourself into a project to low gives you enough leverage when the point of no return is reached to still make a big plus.
  - opportune 9y ago
    This is why it's not accurate to claim that "it's not IBM's fault." They are fully culpable for making a bid that didn't include necessary components of the project priced in.
GnarfGnarf 9y ago
I concur. What I'm hearing is that the Canadian Federal payroll is extremely complex (like all government payrolls, I suppose). The Harper gov't justified the system by laying off everyone who knew the old payroll system. The software developed by IBM was so complex, that it required the expertise of the people who got laid off to operate it. The novices who took over didn't know how to make the software work, and chaos ensued. The Canadian government is as much at fault as the developers on this one. (I speak as one who is paying for this fiasco :o).
- foliovision 9y ago
  IBM has failed to build these state level systems successfully around the world (Canada, Australia, Pennsylvania, Slovenia, New Zealand). IBM should know how to go about building them right or IBM should not be in this business. The first $200 million of cost overruns I'm willing to pin on the Government of Canada. The subsequent billion dollars and the failure of the system to work at all is the vendor's fault. If the Government of Canada is entirely impossible to work with, the vendor should have shut the project down at 200% of budget rather than run it to six times budget and still fail.
walrus01 9y ago
the problems with part #2 have been pretty extensively discussed and detailed already: https://news.ycombinator.com/item?id=16494387 https://news.ycombinator.com/item?id=16494387 my earlier post: https://news.ycombinator.com/item?id=16495431 https://news.ycombinator.com/item?id=16495431 basically, it's a political game... the same jobs they "lost" in miramichi because of the firearms registry legal change, they tried to get back. Same pork barrel politics that congressmen in the US play with trying to bring jobs to rural districts.
- Naga 9y ago
  I missed that article two weeks ago, thanks for linking the discussion.
danmaz74 9y ago
> There's a dimension left out here on the Phoenix disaster: It wasn't really IBM's fault at all. I respectfully disagree. IBM management either knew what they were contracting was a recipe for disaster, or they were incompetent. In both cases, as they were supposed to be the billion-dollar expert on the matter, they bear great responsibility for this failure. And at an industry level, companies that promise the impossible push out of the market more honest ones, and they deserve all the bad PR IBM is getting on this one.
- mgkimsal 9y ago
  you're both right, but you're a bit more right. IBM should have been pointing out that losing the institutional knowledge may cause problems. they possibly assumed some of their milestones and deliverables would be while working with existing staff. if the existing staff is let go, many of the assumptions in the original estimate are just wrong at that point.
  - rgbrenner 9y ago
    The managers needed IBM to tell them about the concept of institutional knowledge? They need to be told that people with experience have learned their jobs? That new employees need training? This is like Management 101. Just how incompetent are these managers?
    - itronitron 9y ago
      The competent managers likely had no say in the matter.
      - bobdole1234 9y ago
        That's a trick question, there are no competent managers in government service for the most part. They either burn out from managing without the ability to recruit competent staff, or they moved elsewhere to make drastically more money. I'm sitting with a 4x multiple on my old public service job.
kpil 9y ago
I agree that buyers typically deceives themselves due to their limited understanding of highly complex large systems, and that their incompetence is amplified by a sub-optimal procurement process. It's obvious that there was zero actual knowledge involved related to implementing large systems - just by the fact that someone thought it was a good idea to take a behemoth system excreted out of Oracle, and then pass it through the corporate body of IBM, before inserting it into a huge government organisation, while at the same time getting rid of any process expertise and business knowledge they might actually have on-board. However - I don't think it's probable that any amount of training could save the situation to any measurable degree whatsoever. Is IBM to blame? Well, I think they knew what would happen.
CountSessine 9y ago
I think if Harper had stayed in office, he would have blamed this on the complicated payroll requirements. He likely would have leaned on PSAC/CUPE to simplify it in the next round of contract negotiations.

by Naga 9y ago

drieddust 9y ago

As someone who works in same space, I can say this is happening because unrealistic expectations of customers are not challenged by bidding Vendors because they fear losing the project. This in turns happens because customer tries to outsmart vendors by pressing them for reducing the timelines to absolute impossible. If any vendor tries to play by the rules, they are thrown out. Smart vendors respond by crafting the contract smarty. Skipping training is one such example. It was crucial but .... In the end project is either canceled or goes over budged or barely manage to deliver anything.

dv_dt 9y ago
Yup, this kind of contracts game is what really needs modern innovation. I think the space is kind of living in an extensions of 70's assumptions with many decades of beauracratic rules added on to try to fix up fundamental problem. For another factor, it would be lower risk to allocate into many smaller projects and sign contracts in phases and sub-systems - but then the funding of the total project becomes more at risk politically, and then there is a higher technical load on the contracting dept to architect the boundaries.
- drieddust 9y ago
  They are barely able to state the outcome of project so expecting them to do the necessary legwork to break projects into logical subprojects is too much to ask. No disrespect but we have come to a point where IT have started appearing magical to pointy hairy bosses who think juse because it's software everything is infinitely malleable with no impact on quality, cost, or time.
  - dv_dt 9y ago
    If you don't have an organization that can break it down to a smaller projects then there is a question if doing it as one large project has a higher or lower chance of failure.
    - drieddust 9y ago
      Shouldn't that depend upon complexity of change but who cares. CIO have a short tenure so collecting the bonuses and leaving before house of card collapses works. It's all about managing the personal bonus and career, who cares for the success 9f project?
      - dv_dt 9y ago
        It depends on complexity, but I think it's more of a situation akin to refactoring or rewriting a complex code base. A total rewrite is always an attractive thought, but a real focus on continuous refactoring in smaller portion at a time is often lower risk, lower cost, and faster in the end - and I suspect it's the same with large bureaucratic processes too. It's also a way to sharpen the skills of the organization in small steps with lower risks for failure (and improving future project steps...).

toast0 9y ago

At some level, isn't IBM supposed to point out that training and transition are important for a large project and shouldn't be left out?

coldacid 9y ago
Sure, if they don't want to be the winning (read: lowest) bidder. Shit constraints --> shit results.
- deleted 9y ago
  [deleted]
otoburb 9y ago
They most likely did raise this, but when asked to provide a more "competitive" bid, they axed training and who knows what else. This is common in large enterprise sales deals. For professional services deployment engagements, there is always the possibility of an amendment to the services scope (often referred to as a change order or change request). If the project is estimated to take over a year to deploy, by then you're in a new budget cycle and the vendor has come to an agreement with the procurement department to charge for additional services as required in subsequent phases.
- LifeLiverTransp 9y ago
  Another nice thing about governments is that they do not deal in monetary damages but in media fallout- so something loud (collapse of a billion dollar project) is worser to them, then silently going over budget because additional points appear on the list. So biding yourself into a project to low gives you enough leverage when the point of no return is reached to still make a big plus.
- opportune 9y ago
  This is why it's not accurate to claim that "it's not IBM's fault." They are fully culpable for making a bid that didn't include necessary components of the project priced in.

GnarfGnarf 9y ago

I concur. What I'm hearing is that the Canadian Federal payroll is extremely complex (like all government payrolls, I suppose). The Harper gov't justified the system by laying off everyone who knew the old payroll system. The software developed by IBM was so complex, that it required the expertise of the people who got laid off to operate it. The novices who took over didn't know how to make the software work, and chaos ensued. The Canadian government is as much at fault as the developers on this one. (I speak as one who is paying for this fiasco :o).

foliovision 9y ago
IBM has failed to build these state level systems successfully around the world (Canada, Australia, Pennsylvania, Slovenia, New Zealand). IBM should know how to go about building them right or IBM should not be in this business. The first $200 million of cost overruns I'm willing to pin on the Government of Canada. The subsequent billion dollars and the failure of the system to work at all is the vendor's fault. If the Government of Canada is entirely impossible to work with, the vendor should have shut the project down at 200% of budget rather than run it to six times budget and still fail.

walrus01 9y ago

the problems with part #2 have been pretty extensively discussed and detailed already: https://news.ycombinator.com/item?id=16494387 https://news.ycombinator.com/item?id=16494387 my earlier post: https://news.ycombinator.com/item?id=16495431 https://news.ycombinator.com/item?id=16495431 basically, it's a political game... the same jobs they "lost" in miramichi because of the firearms registry legal change, they tried to get back. Same pork barrel politics that congressmen in the US play with trying to bring jobs to rural districts.

Naga 9y ago
I missed that article two weeks ago, thanks for linking the discussion.

danmaz74 9y ago

> There's a dimension left out here on the Phoenix disaster: It wasn't really IBM's fault at all. I respectfully disagree. IBM management either knew what they were contracting was a recipe for disaster, or they were incompetent. In both cases, as they were supposed to be the billion-dollar expert on the matter, they bear great responsibility for this failure. And at an industry level, companies that promise the impossible push out of the market more honest ones, and they deserve all the bad PR IBM is getting on this one.

mgkimsal 9y ago
you're both right, but you're a bit more right. IBM should have been pointing out that losing the institutional knowledge may cause problems. they possibly assumed some of their milestones and deliverables would be while working with existing staff. if the existing staff is let go, many of the assumptions in the original estimate are just wrong at that point.
- rgbrenner 9y ago
  The managers needed IBM to tell them about the concept of institutional knowledge? They need to be told that people with experience have learned their jobs? That new employees need training? This is like Management 101. Just how incompetent are these managers?
  - itronitron 9y ago
    The competent managers likely had no say in the matter.
    - bobdole1234 9y ago
      That's a trick question, there are no competent managers in government service for the most part. They either burn out from managing without the ability to recruit competent staff, or they moved elsewhere to make drastically more money. I'm sitting with a 4x multiple on my old public service job.

kpil 9y ago

I agree that buyers typically deceives themselves due to their limited understanding of highly complex large systems, and that their incompetence is amplified by a sub-optimal procurement process. It's obvious that there was zero actual knowledge involved related to implementing large systems - just by the fact that someone thought it was a good idea to take a behemoth system excreted out of Oracle, and then pass it through the corporate body of IBM, before inserting it into a huge government organisation, while at the same time getting rid of any process expertise and business knowledge they might actually have on-board. However - I don't think it's probable that any amount of training could save the situation to any measurable degree whatsoever. Is IBM to blame? Well, I think they knew what would happen.

CountSessine 9y ago

I think if Harper had stayed in office, he would have blamed this on the complicated payroll requirements. He likely would have leaned on PSAC/CUPE to simplify it in the next round of contract negotiations.