31 ms·

Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

• Murkin 9y ago
  Would you sign to build a project who's MVP is 260,000 salary calculations with rules poorly documented and spread other hundreds of agencies? Maybe if they started with a subset and slowly built up..
  • cjalmeida 9y ago
    This problem was likely better served by many smaller systems that better fit the specific needs. Government is way too complex to try to cram all rules into a single monolith like Oracle's PeopleSoft.
    • rpeden 9y ago
      If what I've read is accurate, the various departments were already served by smaller systems that fit their specific needs. The intention with Phoenix was to save money by consolidating them all into one. Oops. At least part of the problem is that projects like this don't end up being awarded to those who are most capable of completing them successfully. Instead, they end up being awarded to companies and consultancies who are experts at shepherding proposals through the RFP process.
• fnordsensei 9y ago
  It's possible to infer from this that CA ordered an output from IBM, rather than an outcome. IBM can claim that they have indeed delivered the thing that was ordered, while CA remain disappointed that the effect they had in mind is missing. In Canada's defense though, it's sometimes hard to find someone who will even take on contracts of this magnitude. Getting good terms, even more so.
  • jandrese 9y ago
    These large government software projects end up costing way more than they should pretty much every time. The process is just broken. There's way too much design by committee and tying the hands of the people who are doing the actual work, preventing them from implementing the correct solution and instead making them implement the one the disconnected-from-reality specification writers decided on. Then they have to go back an iterate the correct solution, which involves more committee meetings at huge expense and rewriting documents and disseminating them to everybody...
    • threeseed 9y ago
      This is completely wrong. The CA government engaged the vendor, IBM, after they had decided on a list of requirements from what they wanted from a solution. In your argument you say that the hands of the people doing the actual work were tied. Well that's the vendor, and so no, their hands weren't tied. And not sure if you've ever worked at a project like this but the committee meetings aren't expensive. They are basically free. And there often should be far more of them. The real cost comes from major changes in the design too late in the project plan.
      • jandrese 9y ago
        Committee meeting take the time of multiple people. They are absolutely not free. People's time is by far the biggest expense in a development project like this. In the long run they can save money (getting everybody on the same page to avoid conflicts in the future), but you're paying up front for that. In practice the value of the meetings decreases with frequency (in the reducto ad absurdum case 100% of your time is spent in committee meetings and nothing gets done). Only engaging with the vendor after the requirements were drawn up is what I'm talking about. They contract with IBM and hand over a 2,000 page document describing their requirements, then discover that it's impossible to implement and end up spending many years and millions of dollars on trying to fix it.
      • pixl97 9y ago
        > but the committee meetings aren't expensive. They are basically free. And there often should be far more of them. The real cost comes from major changes in the design too late in the project plan. https://en.wiktionary.org/wiki/bikeshedding https://en.wiktionary.org/wiki/bikeshedding > The term was coined as a metaphor to illuminate Parkinson’s Law of Triviality. Parkinson observed that a committee whose job is to approve plans for a nuclear power plant may spend the majority of its time on relatively unimportant but easy-to-grasp issues, such as what materials to use for the staff bikeshed, while neglecting the design of the power plant itself, which is far more important but also far more difficult to criticize constructively. It was popularized in the Berkeley Software Distribution community by Poul-Henning Kamp[1] and has spread from there to the software industry at large. http://bikeshed.com/ http://bikeshed.com/
  • betaby 9y ago
    Often multiple smaller/medium-sized systems are better for the particular task than one large. Even more so for taxation and payroll systems where taxation/regulation varies greatly.
• guyzero 9y ago
  I hope to someday read a comprehensive post-mortem but I doubt there's anyone involved who can/would write one without a lot of bias against the other parties involved. This will go down in an annals of massive software project failures.
  • eyrarric 9y ago
    Actually, Australia went trough the same payroll procurement fiasco with IBM. You can read the report http://www.healthpayrollinquiry.qld.gov.au/__data/assets/pdf_file/0014/207203/Queensland-Health-Payroll-System-Commission-of-Inquiry-Report-31-July-2013.pdf http://www.healthpayrollinquiry.qld.gov.au/__data/assets/pdf...
  • throw7 9y ago
    I poked around to read more about this and the gov't did commission a report to look at what went wrong: http://www.cbc.ca/news/canada/ottawa/phoenix-federal-government-report-lessons-1.4339476 http://www.cbc.ca/news/canada/ottawa/phoenix-federal-governm...
• didgeoridoo 9y ago
  The name is just too perfect: https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business-ebook/dp/B00AZRBLHO https://www.amazon.com/Phoenix-Project-DevOps-Helping-Busine... To that point, I’m wondering if all IT initiatives codenamed “Phoenix” are doomed from the start, given that the implication is “let’s burn down everything we have that works, and rise anew from the ashes”...
  • projectileboy 9y ago
    Hah! I make this joke all the time with friends. I have literally been on three different failed projects named ‘Phoenix’
    • fleitz 9y ago
      I'm hesitant to use the phoenix framework for exactly this reason.
    • temp638642 9y ago
      Elaborate?
      • purerandomness 9y ago
        https://en.wikipedia.org/wiki/Phoenix_(mythology) https://en.wikipedia.org/wiki/Phoenix_(mythology)
    • da02 9y ago
      Why did they fail?
      • projectileboy 9y ago
        Well, one possibility is that I am the common factor in all of my bad relationships. But another possibility is that projects tend to get named 'Phoenix' for a reason, and that just as in real life, it's pretty hard to successfully be reborn from a pile of ashes.
• j45 9y ago
  The comments made about the government about excluding IBM from making future systems for the government stood out. The reality is payroll should be executed by payroll specialists. I recently had to research payroll providers and there are players who service 20-30% of the Canadian population already. This is another sign of folks not knowing how to source, design, oversee or implement software implementations.
  • 52-6F-62 9y ago
    This is also why Ceridian is still around in spite of all of their faults (or so I hear from any small/mid-size business practitioners I know). I take it that's who you're referring to?
    • willhslade 9y ago
      ADP too.
    • j45 9y ago
      There's actually a few at that size, Ceridian is one, ADP is another... Most have their pros and cons but ultimately can handle complex union type payrolls. Payroll, also is ripe for disruption.
      • 52-6F-62 9y ago
        >Payroll, also is ripe for disruption. From everything I hear about working with those companies—new and suitable replacements would be welcomed wholeheartedly. That is, if they can maintain the same level of professionalism and ability to handle situations like you mentioned. Reliability is of immense importance, so it's not as easy to just hack away at it and sell people an MVP and iterate.
        • AndyNemmity 9y ago
          Ripe for disruption, except that when you look at the laundry list of requirements, the thing that needs disrupting is the need for a billion different custom requirements, not a different payroll system.
• bloomingfractal 9y ago
  and that's why governments should hire software engineers that can be in the loop and understand not only the technical but the policy side. Hiring an army of contractors earning 300k/year to deliver "something" in the waterfall model is a recipe for disaster.
  • threeseed 9y ago
    Have you actually tried what you are saying ? Because it's a nice idea just completely unrealistic. Software engineers who are not just technical but SMEs as well as having excellent skills in stakeholder engagement are incredibly rare. And most of them know how good they are and are contracting at, you guessed it, around 300k/year. Talented engineers are usually quite savvy when it comes to money. Also for most projects like this it is Waterfall for the overall project (Requirements, Design, Development, UAT, Production) and Agile for the Design/Development parts. It's really the only way since the client does Requirements/UAT/Production and the vendor does Design/Development.
    • dragonwriter 9y ago
      > Have you actually tried what you are saying ? Because it's a nice idea just completely unrealistic. I've been in places where it's done. In fact, I've been involved for a long time in a public sector environment with three key systems, one of which is a central claims adjudication system, and two of which are accounting and tracking systems that interface with it, performing similar functions to each other but for two separate programs that share the adjudication system. Those three systems, for historical reasons, have three separate models: The adjudication is state-owned (that is, the code belongs to the state) and operated, developed almost entirely by a team of contractors with some involvement by state programming staff and a state development lead and also tight integration state-staff IT team that does requirements analysis and acceptance testing and manages most of the interface with state business users. One of the accounting/tracking systems is, and has been from the beginning, developed and maintained almost entirely in-house by state staff (and almost solely with programming staff.) Because of subsequent organizational consolidation, the state team working on this system overlaps the one working on the prior system. The other is a vendor-maintained MOTS system, with state IT oversight. Each has it's strengths and weaknesses, but the MOTS system is consistently the one that is the roadblock to adapting to changing business requirements, and both before and after consolidation the all-internal one was the one with the quickest best-case requirementd to delivery speed, and by far the least expensive for the value delivered.
• dman 9y ago
  Could this be the first time someone gets fired for hiring IBM? (Sorry could not resist after having heard of that quip so many times).
  • quickthrower2 9y ago
    Nope noone gets fired for choosing IBM
  • nikanj 9y ago
    Probably not. IBM will return a token amount of money so politicians save face. Someone might take the revolving door either direction.
  • WingH 9y ago
    Nobody gets fired, but nobody gets promoted either.
• ismail 9y ago
  I am guessing one of the reasons is probably the software selected... oracle
  • threeseed 9y ago
    Oracle Peoplesoft is used successfully at a number of organisations so blanket statements aren't helpful. But I would question whether it is better/cheaper in these cases to just write bespoke software from scratch rather than trying to modify a vendor's existing product.
  • dsfyu404ed 9y ago
    Oracle/PeopleSoft can do payroll for a defense contractor with tens of thousands of employees across multiple nations. I think proper configuration is what's lacking in this case.
    • debacle 9y ago
      Oracle costs a lot. It infects your entire stack, from hardware all the way up to personnel, and drastically inflates the cost from end to end. With no real utility advantage, either.
      • dsfyu404ed 9y ago
        That's not relevant to whether or not the software is capable of performing the task.
      • threeseed 9y ago
        Everything infects your entire stack. If I hire Java, Ruby, PHP etc people then everything from hardware up to personnel is going to be selected to optimise the delivery and execution of software. Kind of common sense that you would do this. I don't disagree that Oracle costs a lot and that they inflate margins pretty poorly but to say they have no real utility advantage is a bit ridiculous. Most of the world's enterprises are running core business systems off Oracle and if they move to the cloud it's usually to Oracle instances in the cloud not any other system.
• vidanay 9y ago
  So, what makes a payroll system at this scale so difficult? On the face of it, there are salary employees and there are hourly employees. How many taxing bodies does Canada have? Federal, provincial, and municipal? Is that a few dozen or a few hundred? Is Canadian tax code crazy complicated that doesn't allow for a rules based system? I'm honestly curious as to how a "simple" payroll system can go so far off the rails.
  • threeseed 9y ago
    FTA: ".. had yet to be encoded with many of the payroll complexities faced by a massive and diverse public service, such as unique shift rules for Coast Guard officers or prison guards."
  • BoorishBears 9y ago
    I remember thinking “Why does this simple state Jury Duty site need to be down from 9PM to 9AM” “I could write a site that did this job with 99.9% reliability in a weekend” Turns out disability laws require a real person be available to assist with the site. Imagine bidding on a website project and accidentally promising 24/7 live support. That’s government tech in a nutshell.
  • Demoneeri 9y ago
    There are multiple rules based on collective bargaining (union). "The software platform itself, a heavily modified version of Oracle’s PeopleSoft program, had yet to be encoded with many of the payroll complexities faced by a massive and diverse public service, such as unique shift rules for Coast Guard officers or prison guards."
  • deleted 9y ago
    [deleted]
  • mkohler 9y ago
    http://wiki.c2.com/?WhyIsPayrollHard http://wiki.c2.com/?WhyIsPayrollHard
    • vidanay 9y ago
• zaroth 9y ago
  Not having ever worked on anything like this, the death knell for these projects must be the inability to ever reduce functionality into a reasonable core set. Basically the 80/20 rule but where there are practically an infinite number of edge cases that stretch the project on indefinitely? You could serve 80% of the payroll burden with software that cost $10m. 90% coverage costs $100m. 98% costs $1,000m. And 100% costs $∞
  • microcolonel 9y ago
    Also there's a good chance that the decision to go with IBM as a vendor, and the conduct thereafter, was corrupt from the start. That would go a long way toward explaining why so little due diligence was observed in assessing whether or not to continue the project with them at any of the stages it could've been. Yes, people do make huge blunders in powering forward with overpowered, unnecessary solutions to problems they largely do not have; but blunders this big usually don't take this long to suss out.
    • saas_co_de 9y ago
      > blunders this big usually don't take this long to suss out This is actually pretty minor league compared to U.S. government failures. e.g. https://www.wired.com/story/us-border-patrol-hasnt-validated-e-passport-data-for-years/ https://www.wired.com/story/us-border-patrol-hasnt-validated...
      • microcolonel 9y ago
        Idunno, that doesn't really seem like such a big deal. The whole idea was rather pointless (in the face of all the bigger fish they have to fry) to begin with. With such headed, small minded planning, it may have been a blessing if they never bothered to implement it.
    • AndyNemmity 9y ago
      At the same time, "Nobody got fired for choosing IBM" is a statement for a reason. If you hire someone else, people will say, "Why didn't you hire the best?" If you hire IBM, you're avoiding that reasoning, even if on any given project IBM or other large enterprise vendors may not be the best.
• kabdib 9y ago
  Was not surprised to see Oracle and PeopleSoft involved. I hope they get the crap sued out of them. That stuff is toxic, and I pity any organization that Oracle got its claws into using that junk.
  • walrus01 9y ago
    Unless you are one of the people who makes a medium six figure salary as a deck officer on Larry Ellison's yacht, Oracle is utterly toxic.
    • rosege 9y ago
      I doubt they are on medium - probably low. I once met the IT guy onboard his yacht. Told me a funny story about trying to get the superbowl for one of Larry's mates one year.
      • walrus01 9y ago
        Captain position ignored for a moment, I have no idea what second or third mate on a giant Russian/saudi oligarch class of yacht might make in yearly gross, but if I were a billionaire I'd be looking for people with many years of offshore experience and a degree from a major maritime University... $175k usd?
  • da02 9y ago
    Back in 1999, I was in the elevator and I heard the admin. staff say PeopleSoft (the software) doesn't do anything. This was @ Polytechnic Uni. in Brooklyn. This conflicted with the school's public message where (officially) adopting PeopleSoft was a big step in the right direction. I always wondered if PeopleSoft improved since. I guess not?
    • funkymike 9y ago
      My employer recently switched from PeopleSoft to Oracle for HR stuff. So now instead of the system itself being slow to respond, it is plenty responsive when I put in for time off. However, if I fill out the form too quickly it rejects the input as invalid. They can't even get form input validation to work correctly.
• jellicle 9y ago
  Incidentally, the error was here: "The project was meant to save costs by firing 1,200 employees handling payroll at various departments around the country and replacing them with about 500 people in a centralized location using Phoenix to handle most of the government’s payroll needs." It's not software problems, or not primarily software problems anyway. They fired EVERYONE who knew anything about payroll in the entire Canadian government, all at once, and hired 500 brand new people in one call center to work with the brand new software as it rolled out. The new software doesn't include code for a lot of "special cases" which together represent a lot of cases. The new people have no idea how to solve most pay problems (people problems OR software problems). They get four days of training. http://www.cbc.ca/news/canada/ottawa/phoenix-falling-pay-centre-miramichi-1.3687476 http://www.cbc.ca/news/canada/ottawa/phoenix-falling-pay-cen... "Phoenix was supposed to cut our work in half. It actually doubled our work. So, you can't get rid of 2,000 people ... and double the work and expect the work to get done," one said." Overall, even if the software worked perfectly the execution would have been a disaster.
• dctoedt 9y ago
  Reminiscent of the state of Indiana's lawsuit against IBM for allegedly botching a project to automate the state's welfare system — lots of finger-pointing on both sides; the trial court's 65-page decision started out with the words, "Neither party deserves to win this case" [0]. The case has been up to the Indiana Supreme Court already [1]; on remand last summer, the trial court found that IBM is liable for USD $128 million [2]. [0] https://www.scribd.com/document/100544020/Indiana-IBM-Decision https://www.scribd.com/document/100544020/Indiana-IBM-Decisi... [1] http://www.in.gov/judiciary/opinions/pdf/03221601shd.pdf http://www.in.gov/judiciary/opinions/pdf/03221601shd.pdf [2] https://www.indystar.com/story/news/2017/08/07/court-ibm-owes-state-128-m-welfare-privatization-dispute/544789001/ https://www.indystar.com/story/news/2017/08/07/court-ibm-owe...
  • wahern 9y ago
    Apropos HN, he then says, officials set out to fix Indiana's poorly-performing welfare system by inserting an untested, theoretical experiment, and substitute personal caseworkers with computers and phone calls ("remote eligibility"). This is now admitted to be an error, and there is nothing in this case, or the Court's power, that can be done to correct it, or remedy the lost taxpayer money or personal suffering of needy Hoosiers.
    • toomanybeersies 9y ago
      ML based welfare eligibility screening seems a few short steps from Kafka. I'm not sure if that's what they were doing, but it seems like something that somebody would think is a great idea.
      • samstave 9y ago
        The ONLY person who would think all that is a good idea are the engineers who were able to update their linkedin with "implemented ML based solution to replace human resources with computer resources to streamline the welfare system for the state of X"
        • 9y ago
• moltar 9y ago
  The Big Rewrite. What can go wrong?
• jorblumesea 9y ago
  The whole big 4 IT consulting industry needs to die a swift death. It's basically institutionalized corruption. Overbill the client, under deliver with shoddy products made overseas, get that vendor lock in and watch the money flow in. The cost savings are largely an illusion sold by false promises of a better world.
  • Demoneeri 9y ago
    You are mistaken, I work for one of these. I have 12 years experience and I have only one failure to deliver and it was stopped in the design phase because it was too complex and the client wasn't ready.
    • jorblumesea 9y ago
      I worked for one too and I can't say the same thing. Entire projects were poorly thought out, designed and improperly executed. The people working for the company were incompetent and their "hiring bar" was if you had a pulse. And this was a big well known firm.
      • AndyNemmity 9y ago
        You're both right, I've seen both sides. The reality is, in all of them there are teams of amazing, insanely talented people, and teams of complete train wrecks. It's just life. In any large company, there's a massive variance in ability.
    • stjohnswarts 9y ago
      Same experience with me. I have mostly only consulted with large companies. There are some companies that insist on solid code and design. There are others that just live from quarter to quarter and the whims of the CEO and just throw out whatever is done at the end of the quarter. Having a great lawyer team chomping at the bit for litigation helps their confidence in living from quarter to quarter.
  • saas_co_de 9y ago
    Do some research into government contracting and you will realize that companies like Oracle are tame compared to many of the straight up scams being pulled. You will see countless examples of companies with no assets, no employees, virtual offices, etc, pulling down multi-million contracts.
• nradov 9y ago
  For historical perspective on another (in)famous payroll system project I encourage developers to read about the Chrysler C3 project. It ultimately failed, and yet provided the genesis of some agile development best practices that most of us use today. http://wiki.c2.com/?WasChryslerComprehensiveCompensationSuccess http://wiki.c2.com/?WasChryslerComprehensiveCompensationSucc... https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compensation_System https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compens... These things are hard to get right in any complex enterprise.
• walrus01 9y ago
  Canadian here. This has become a huge political football. People are blaming the current Liberal government, when the project was started by the Harper Conservatives. The full extent of just how fucked up it is has just now become public knowledge. It is also a political pork barrel project. > The project was meant to save costs by firing 1,200 employees handling payroll at various departments around the country and replacing them with about 500 people in a centralized location using Phoenix to handle most of the government’s payroll needs. What's not written there is that the decommissioned long gun registry was located in a rural part of New Brunswick. When it was killed by the conservatives, they had a big political issue on their hands because suddenly hundreds of federal government clerical workers were out of a job in this area. So what was the solution? Re-hire the same people and put them on the Phoenix payroll project. Never mind that not one of them had any experience with IBM, Oracle, PeopleSoft, payroll, finances, or anything else related. They wanted butts in chairs in front of keyboards and to keep those people on the federal workforce payroll. https://www.google.com/search?num=100&client=ubuntu&hs=ssW&channel=fs&ei=6leYWpieE4PgjwOUsoGIAw&q=long+gun+registry+new+brunswick+miramichi&oq=long+gun+registry+new+brunswick+miramichi&gs_l=psy-ab.3...6322.8176.0.8318.10.10.0.0.0.0.143.918.5j4.9.0....0...1.1.64.psy-ab..1.8.778...33i160k1j33i21k1.0.czpxz7HDuMU https://www.google.com/search?num=100&client=ubuntu&hs=ssW&c... quick edit, history lesson: early 1990s, Chretien Liberals: we need a gun registry liberal party: okay we passed a gun registry law liberal party: we need a pork barrel project to make these people in small town new brunswick happy, let's hire 500 people in miramichi liberal party: this gun registry has now cost 400% of what was originally budgeted liberal party, paul martin government loses an election to stephen harper and the conservative party harper conservatives: we promise to scrap the long gun registry harper conservatives: we've removed the long gun registry but now all these people are out of work and pissed off harper conservatives: we need to overhaul our old payroll system that runs on a bunch of old minicomputers, let's hire ibm harper conservatives: let's re-hire those people we fired and put the payroll center in small town new brunswick! harper conservatives lose the 2015 elections trudeau liberals: wtf is going on with this contract the previous government signed. wtf is going on with this clusterfuck of oracle, ibm, peoplesoft and porkbarrel politics of employing people who are not payroll specialists. trudeau liberals, later on: seriously this is broken beyond repair
  • 9y ago
• endorphone 9y ago
  The Canadian government attempted to build a gun registry, originally predicting it to cost $12M, then $85M, then $1000M, then $2000M, and then it was scrapped. While it's easy to blame vendors like IBM, I have to imagine feature-creep and "what about"-isms lead to such monstrosities.
  • walrus01 9y ago
    quick correction, Canada still has a gun registry for handguns. In one of the several elections that they won, the Harper Conservatives campaigned on removing the long gun registry (rifles and shotguns). Which they successfully did, putting hundreds of federal employees out of work in a small town part of New Brunswick. The political outcry from these unemployed people and the economic impact sent them on a new trip back to the pork barrel, centralizing all of the Phoenix payroll system clerical operations in... you guessed it... the same small town part of New Brunswick.
  • beamatronic 9y ago
    How many long guns are there in Canada anyway. Could you not use sheets.google.com? I'm serious.
    • walrus01 9y ago
      There are quite a lot, actually, though canada has a greater proportion of traditional type long guns like bird hunting shotguns, bolt action rifles, etc. Less people who want to buy a consumer-legal version of a HK416.
    • adrienne 9y ago
      Roughly a quarter of households in Canada own at least one firearm. Most of those are long guns.
    • flamedoge 9y ago
      You jest but really it shouldn't cost 2B for a database.
  • tigershark 9y ago
• manofstick 9y ago
  New Zealand in the 90's was trying to have an integrated policing system designed by IBM which burnt a lot of money (in the order of 100 million) and was abandoned [0]. I was working with one of the companies that came along to replace one part of the system with a much smaller component that just handled that individual piece and vividly remember the meeting with the IBM team. They had no code to show me, but they handed my a massive folder - probably verging on 10cm thick of use case diagrams. So many little stick figures staring up at me I was awestruck - but not in a positive way! [0] https://en.wikipedia.org/wiki/INCIS https://en.wikipedia.org/wiki/INCIS
  • abraae 9y ago
    Yep, INCIS was a cockup of massive proportions. I remember interviewing a BA (business analyst) who had just come off of the INCIS project. She said their entity model was enormous, with tens of thousands of entities. I couldn't understand how any project could have so many entities, let alone how anyone would ever hope to code up such a thing. She said the commandment to the analysts was to model everything so they did, just like they learnt in data modelling 101. So (INCIS was a project for the NZ Police), e.g.: - A police station has zero or more cells - A cell has one or more doors - A door has one and only one key - A door has many bars Another data point in INCIS - my sister at the time worked in the police. Her workplace ran out of pencils (they did a lot of note taking with pencils and paper). INCIS had so thoroughly depleted the police's budget that there was nothing left for...say.. computers with email and the like.
  • toomanybeersies 9y ago
    More recently, NZ had the Novopay debacle: https://en.wikipedia.org/wiki/Novopay https://en.wikipedia.org/wiki/Novopay The software itself, before fixing it, cost $182mm and covered payroll for 90,000 teachers. That's over $2000 per teacher. It cost another $45mm to fix it after this.
• deleted 9y ago
  [deleted]
• goodroot 9y ago
  As a Canadian who has worked at IBM, I'm not surprised that this didn't find success. In IBMs defense, I don't believe our public sector has the experience or aptitude required to act as a supporting interface for a job of this scale. With that being said, IBM isn't a successful technology company with a proven record building good software products. They were an unwise choice from the get-go. IBM is a successful financial engineering and sales firm that sells things, then scrambles to hire/acquires to get things done to an ever-evolving, always justifiably rough spec. They quack and then start the hack-show. What do you expect from the organization that invented "FUD" (https://en.wikipedia.org/wiki/Fear,_uncertainty_and_doubt https://en.wikipedia.org/wiki/Fear,_uncertainty_and_doubt) as a way to emotionally manipulate perspective clients into making a purchasing decision?
  • j45 9y ago
    I know folks who work at IBM as well - don't mean to put you on the spot - but IBM should be far better at leading and guiding customers to success. Except at $250-350/hour, below average performance is more profitable than successful and a lot of good people suffer on both sides.
    • Nelson69 9y ago
      Didn't IBM effectively give them that guidance? It seems kind of simple in a way, if IBM was hired to build X and failed to do so, then there will be a law suit or some form of arbitration and some money will be recovered or something. It's pretty standard breach of contract type stuff. You can audit everything too, maybe IBM was billing for stuff they shouldn't have; I kind of doubt it but maybe there was something really amiss. On the other hand if IBM was hired to build X, then it changed in to X' and then X'' and 10,000 changes later it turns our, Canada really wanted Y from the start, isn't IBM guiding them along their process to figure that out? It's an insanely expensive way to build and buy stuff. Is there supposed to be some sort of internal customer's advocate that looks at the project and says "hmm, this is a payroll system, it shouldn't cost a $billion typically, or this was a $100m contract that is already up to $300m, something must be wrong..." Don't get me wrong, in most sales driven organizations, they're high-fiving whoever inked that contract and trying to get invited to the parties he throws on his yacht. The catastrophic failure sucks for everybody, it's hard to see someone else finishing the job more inexpensively that getting IBM to do it though.
• CivBase 9y ago
  Does anyone know what the real problem was here? > Some people were paid too much, others not at all. Those issues snowballed with the deluge of requests to fix incorrect paychecks. > IBM said it’s “fulfilling its obligations on the Phoenix contract, and the software is functioning as intended.” Okay, so the cost is coming from incorrect paycheck values, but the software is working correctly? Does that mean it's human error? Can these issues be traced back to one or a few major instances of human error? Is this a widespread issue caused by a confusing or misleading UX design? The article is quick to talk about the projects costs and failures as well as the politicians involved, but I can't learn anything or blame anyone until I understand what went wrong.
  • halfnibble 9y ago
    My guess is that IBM believes the software works according to specification. And maybe it does. Maybe the specifications are wrong. Most likely, it's a combination of bad specs and programmed logic errors that don't always follow the specs that are correct. Of course, this requires testing and time to hash out. It would be a major mistake to scrap a project in it's final phase of refinement. They should have switched over slowly, department by department, handling and resolving issues along the way. But scrapping the entire project now only means another $1B investment and repeat down the road. Trudeau would join the ranks of the FAA in flushing multi-billion dollar software projects. It's a waste. I don't know about this project, but software engineers died on-the-job working on the FAA's AAS that never launched.
  • tempestn 9y ago
    From the anecdotes I've heard, it sounds like the support staff is just wildly insufficient to handle all the issues that crop up with a major change like this. Initially the government switched over a trial group of (iirc) something like 40k employees. There were a bunch of problems with people not being paid or being paid the wrong amounts, but for some incomprehensible reason (perhaps they wanted to ditch the old system to focus on fixing the new one or something crazy like that?) they switched over everyone before fixing those. Now, I personally know people who've been getting incorrect pay for over a year, who submit a complaint to their HR, who pass it up the chain, where... nothing seems to happen. Presumably because the 500 people or whatever it is tasked to deal with this are literally dealing with tens of thousands of issues, many of which aren't trivial changes (and so block the many that actually are trivial changes, but aren't even being seen). Basically it was released before it was ready, initial signs of issues were ignored, and now the human side of the system is overwhelmed. It's kind of like when a web server gets overwhelmed with traffic, and it isn't set to reject at a reasonable threshold, so things start getting queued and timing out, people start refreshing pages, and it just snowballs until the whole thing goes down. Which apparently is what has now happened.
• Analemma_ 9y ago
  It's been a long time since I've heard about any successful projects coming out of IBM's consulting and services businesses. Mostly it's one headline after another like this- "Billion dollar project scrapped". How much longer can they keep coasting on the reputation of what was, if we're being honest, a completely different business that just happened to have the same name?
  • dajt 9y ago
    It's incredible, isn't it? I can't believe any govt still hires them. They were behind the Australian census debacle too.
  • AndyNemmity 9y ago
    You don't hear about successful projects. They don't make news stories. I've done a ton of projects in consulting, not IBM, but you've never heard of any of them, because they worked, and the customer was happy (although maybe some delays, or other randomness.) It's self selecting that news is about massive failures.
• blantonl 9y ago
  The software platform itself, a heavily modified version of Oracle’s PeopleSoft program ... What implementation of Peoplesoft isn't heavily modified? Adapting business processes to COTS software packages is extremely challenging. Anyone remember the absolute bank that Peoplesoft consultants made back in the day when ERP and Business Process Re-engineering was all the rage?
  • kabdib 9y ago
    When I last "dealt with" PeopleSoft, the standard operating procedure was: 1) Pay PeopleSoft/Oracle a metric f--kton of money. 2) A CD arrives in the mail. (This little envelope cost $800,000?) 3) Your PeopleSoft consultants show up, laugh at that CD (tossing it into the microwave) and pull out their own distros with a zillion customizations that they proceed to spray over your servers. You have no idea what they are actually installing. 4) You get out a firehose, fill it with money, and keep showering the consultants with cash until you are sure they won't succeed in delivering anything worthwhile, and you fire them. I was only installing a local test instance of PeopleSoft (to do some integration work with another product). Somehow the PS consultants in the area got wind that someone in my company was doing an install, and I started getting all kinds of cold calls offering customization and optimization, with some of the most condescending and customer-hostile attitudes I've encountered. PS had pretty a pretty shitty ecosystem 15+ years ago, and it doesn't sound improved. I'd hate for anyone nontechnical to be at their mercy.
    • mkstowegnv 9y ago
      I was privvy to discussions surrounding a search for a university president. One of the candidates dropped out when he heard that the university was replacing a perfectly functional home brewed accounting system with PS - it had brought his previous university to its knees and he didn't want to ever deal with it again.
      • kabdib 9y ago
        That candidate showed remarkable common sense.
• tw1010 9y ago
  This reminds me of the part in the Robert Moses biography when he was made responsible over implementing a more meritocratic employee review system for the government. Just like this story, that one also ended in flames, though Moses made it out all right in the end. I wonder who the Moses character in this story was, and if we'll read about it in a biography in a few decades.
• tboyd47 9y ago
  > The project was meant to save costs by firing 1,200 employees handling payroll at various departments around the country and replacing them with about 500 people in a centralized location using Phoenix to handle most of the government’s payroll needs. > The government will spend C$431 million to keep the program running in the near term, on top of C$460 million already spent to put Phoenix in place and fix the problems it generated. So for each of the individual salaries Phoenix was supposed to eliminate, they spent C$600k ($500k USD) in up-front costs, and it will be another C$600k to clean up the mess. Turns out, software takes a lot of time and money to build. Who knew? I guess IBM didn't. Or maybe they didn't care? Maybe there should be some kind of advisory board system for large software projects, made up of disinterested-yet-well-informed techies, similar to what exists for corporations.
  • JumpCrisscross 9y ago
    > Maybe there should be some kind of advisory board system Committees are a bad way to solve a problem of mis-aligned interests.
    • tboyd47 9y ago
      Perhaps we (developers) could improve the industry as a whole if we can provide a solution to this misalignment of interests.
  • bhouston 9y ago
    > I guess IBM didn't. Or maybe they didn't care? I think from the perspective of IBM, this contract has been good. They made way more from the Canadian government than they initially estimated the contract to be.
  • gaius 9y ago
    Turns out, software takes a lot of time and money to build. Who knew? I guess IBM didn't. Or maybe they didn't care? Oh they knew, and they cared... about the money
  • dade_ 9y ago
    There is some missing math. They likely have 30 or 40 payroll systems all up for renewal/replacement. The old systems are probably loaded with features and forgotten requirements that would have been discovered during deployment.
• ll931110 9y ago
  There seems to be a gap to fill for companies/organizations building software for government. Wondering which reputable companies are filling this gap?
• redleggedfrog 9y ago
  IBM?! Oracle?! And you got away with only losing $780 million! Consider yourself lucky.
• verdverm 9y ago
  I have payroll issues with IBM still. They overpaid me when I left, then they got my end date incorrect and wanted more back than they deserve. This has been ongoing for 7 months, they can't seem to figure it out... just check the email logs, if they had any... They sent it to collections some time ago, now trying to get them to negotiate a payoff rather than dealing with IBM Employee "services"
  • djrogers 9y ago
    Whoa, doesn’t your state have an agency that handles that stuff? I was terminated once by a company that claimed they went bankrupt, so I filled out one form with the labor commission (had to in order to claim unemployment benefits - I was 22 and had no savings), and they got ALL up in my former employers business and straightened it out quickly with no extra effort in my part.
  • jhallenworld 9y ago
    Wow, collections.. I have a similar story: as an IBM employee I had a company Amex card. It works like this: you buy something on the card (in your name, against your personal credit), but you also fill out an expense reimbursement form (a horrible Java web-app) and IBM pays the card. All fine and good, but if you return something or if the vendor gives you a discount later or anything like that, you end up with a negative balance on the card. Now you leave the company, but you have an Amex card with a negative balance. I received Amex statements for six months before they figured out how to deal with it.
• hindsightbias 9y ago
  "ADP tech chief Stuart Sackman says his company is under constant attack globally and is hoping to encrypt more of its data using the new technology. A sixty year customer of IBM, ADP currently runs its largest domestic payroll engine on the systems, calculating 20 million paychecks per pay period, while also running its money movement, tax filing systems and cloud-based software suite on versions of the Z-series mainframes." https://www.forbes.com/sites/alexkonrad/2017/07/17/ibm-seeks-boost-with-hacker-proof-mainframes/#7304138ca9e8 https://www.forbes.com/sites/alexkonrad/2017/07/17/ibm-seeks...
  • Toast_25 9y ago
    As someone working in ifosec, i loathe how vague that article is about what exactly is going on in the chips.
• british_india 9y ago
  Didn't IBM switch to getting a majority of their developers from India? Enough said.
• giarc 9y ago
  So it doesn't have to be this way. I work for Alberta Health Services, we have 120,000 employees across the province. There are likely 5 separate unions plus out of scope staff (management). This organization was created after the merger of 5 separate health care organizations all with separate systems etc. They over time rolled everyone into the same pay system and I didn't hear about a single person missing their pay. The federal government transition has resulted in tons of people being not paid at all, being paid too little or being paid too much.
• bitmapbrother 9y ago
  I'm surprised there weren't any financial penalties for the IBM practice of over promising and delivering a broken system that generates more problems than it solves. They underbid and embellish proposals with BS to get the contract, staff if with overseas workers to get the maximum return and then hope for the best. This is how IBM does business and it's unfortunate that they continually get away with it by profiting on their incompetence.
• ttt111222333 9y ago
  > Canada to scrap IBM payroll plan gone awry costing $1B. The Phoenix project was originally chosen by Prime Minister Justin Trudeau’s Conservative predecessors 10 years ago to centralize the government payroll. Is it me or is this wording terrible? Trudeau is not a Conservative. It makes it sound like it is Trudeau's party that caused the problem.
  • mattnewton 9y ago
    Yeah why is his name there at all?
    • bcholmes 9y ago
      And why isn't "The Harper Government"(TM) not named?
      • loufe 9y ago
        I'm not a fan of the convservatives but it's still unfair to place the blame of a bad choice of client on a political party. A lot of non-partisans were involved in getting the project running.
        • tempestn 9y ago
          Sure, but it would make more sense to refer to them by name than as "Justin Trudeau's Conservative predecessors" or whatever.
• PayrollIssues 9y ago
  So: for anybody who is curious what's actually going on here: 1. Problems are far bigger than the specific payroll software system. There were tens of thousands of outstanding cases in 2015 before the new system went live. There were tens of thousands of outstanding cases in 2012 when the project started. And there were tens of thousands of outstanding cases every year before that on the "old great system". (why? read on) 2. IBM didn't create a new system from scratch - Govt chose PeopleSoft HCM, an industry-leading system which reliably pays people day in and day out at thousands of customers. Base solution is not the core issue. 3. When you buy PeopleSoft (or SAP, or JDE, or whatever), ideally you make your business fit the system's best practices, customize as little as possible. No client fully does it, but govt doesn't even try. IBM, together with govt employees, spent five years trying to hammer several tens of thousands of complex time and labor rules from more than hundred labour agreements into the system. 3. Implementing this system was half of the initiative, other half was reducing workforce for compensation advisers and moving them all to NB, hiring new grads as replacements. Who were surprisingly not exactly intimately familiar with these hundred labor agreements and tens of thousands of time and labour rules. We're talking one third of the support force at one tenth of experience. 4. So, the billion wasn't spent on "the system". It was mostly spent scrabling afterwards to re-hire old CA's as expensive consultants and trying to catch up the damage. 5. Because guess what, garbage in, garbage out. If your manager doesn't submit or approve your acting gig or leave of absence or maternity leave for several weeks, or months, or more, system won't magically know about it. Not the old one, not the new one, not some magical future one. If your HR person doesn't create you as a new employee, you won't get paid. If you don't submit your overtime slips on time, but instead collect them for 10 months so you can have "Xmas bonus", that also MAY just may create an issue. Ditto with retirement applications. Secondments to other departments. And if through this all employee groups perform denial of service attacks on the system because this is all a political game, that may also be a problem. There is a pervasive cultural issue that boggles mind as to why nobody does their HR things on time or correctly there. The system just provides useful target of the moment, but issues were there before and without cultural change they'll be there in the future. There are real problems and there are real people suffering because of them, and they have for decades, and yes it's gotten worse. Grandstanding over some magical new systems won't fix it though. This is a burner account, I had the dubious privilege of working on this few years back and am aware of some of the horror stories - from the present and the past. I'll stay the dear heck away from Federal Govt / Public Sector project in the future if I can.
• lopmotr 9y ago
  "there is reputational risk for IBM in not helping us fix this,”" - not true. IBM has a long history of exactly this kind of problem. Canada didn't care about their existing poor reputation and the next government to hire them won't either.
• kellysutton 9y ago
  This sounds like a good application of Gall's Law: A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.
• maerF0x0 9y ago
  I wish governments would just get on with it and open source their functionality for those of us who want to "serve our country"
  • dragonwriter 9y ago
    Passively wishing does not tend to influence government policy.
    • maerF0x0 9y ago
      Don't mistake my HN comment as being my _only_ action.
• toomanybeersies 9y ago
  To add to the list of failed (at least initially) payroll systems, we have New Zealand's Ministry of Education and their Novopay system: https://en.wikipedia.org/wiki/Novopay https://en.wikipedia.org/wiki/Novopay It cost $180mm and was borderline non-functional, and then cost another $40mm to fix to its now functional state. The same company that did Novopay also did the payroll for NZ Post and had similar issues.
• myrandomcomment 9y ago
  This is sad. IBM used to be the watch word for "It just works!" I started at IBM many years ago. When I went to my 1st ISP (back in the early 90s) and things broke I was like "what do you mean it broke? How is this okay?" It was quite a shock to move from a company where we had systems with 20+ years of uptime to having reboot the USENET server every 5 days.
• irascible 9y ago
  We only send our BEST people!!
• erpellan 9y ago
  The real achievement here is managing to run up a bill of 1B CAD while still failing to deliver. I mean, I can think of a few projects that probably cost several million/yr but 100x? Astonishing. There's an anecdote floating around about the consultant who heard of a billion-dollar government IT project who promptly offered to do it for 20 million. When asked how, they replied that they would sit on a beach for 5 years then pick up the phone and say, "We failed". Their contact got back in touch 5 years later and said, "I wish we'd given it to you".
  • tootie 9y ago
    CityTime was budgeted by City of New York for $63M and after spending $700M, a bunch of people ended up in jail. https://www.wnyc.org/story/143601-citytime-cautionary-tale/ https://www.wnyc.org/story/143601-citytime-cautionary-tale/
• rmrfrmrf 9y ago
  I've never seen a PeopleSoft implementation go well. It's always underbaked and overpriced.
• nlolks 9y ago
  Have you heard of Open Source? Government of France has. Enjoy the article. https://www.enterprisedb.com/blog/france-takes-lead-new-open-source-front https://www.enterprisedb.com/blog/france-takes-lead-new-open...
• S_A_P 9y ago
  Former ibm employees are even toxic. The cmo of openlink is a former ibm cloud marketer that has effectively run out the entire marketing staff
• watertom 9y ago
  I was the CTO for a large, Fortune 100 company. When I first took over the position I would regularly get requirements documents for internal projects that were 3 or 400 hundred pages. The project team would dutifully carry out the requirements gathering process, everything was meticulously documented, with data flows, and process maps, etc. Everything was a requirement, everything was mandatory, and everything had top priority. IT wasn’t allowed to say no, wasn’t allowed to criticize the business, and wasn’t allowed to analyze if what was specified made sense, or would work. Luckily I had a CIO that was willing to back me up when I started rejecting these projects. In six months I killed 10 or so very large multi-year software projects, in each case the business was forced to use existing software and change their processes. We saved a ton of money and implemented the solutions in weeks not years. These would have projects that had upwards of 100 people for 2, 3 or 4 years, all because the business wanted a “perfect” process. I consulted to the DoD and this pattern was repeated, over and over again. Most assuredly this is what happened with the Canadian Payroll project. Every person had their pet requirement(s) and they made sure they “got them in”, in the end the system that was specified couldn’t be built. It’s like SAP, if you buy SAP take it out of the box and adapt your processes to SAP, if you do you’ll have great experience. If you try to customize SAP, it going to get very expensive, you’ll have a ton of problems, you won’t be able to take upgrades and hou’ll Need a ton of staff just go try to keep it running day in and day out.
  • tyingq 9y ago
    Been there, and truly understand.
  • avinium 9y ago
    Governments everywhere would function a lot better if there were more people like you in them. I'm probably preaching to the choir, but government managers are deathly afraid to stick their necks out and say "no" to the ridiculous spec-by-committee approach to projects. It's a lot easier to shift the blame to contractors, particularly when most projects take so long that they're obsolete by the time they go live.
  • Cacti 9y ago
    I have had a similar experience. People hate change, and will do their best to not change their processes, ever, unless someone makes them (which rarely happens). They'd rather spends hundreds of millions of dollars inventing some new piece of software just so that don't have to spend a minute pressing a button they don't want to press. I am convinced 90% of corporate IT software would be irrelevant if the businesses were willing to change 5 or 10% of their process.
• makecheck 9y ago
  Sometimes I wonder how these projects are started; is it really that they go straight to “here’s a binding contract for millions”, without having them do any smaller projects in between? If it were me, at the very least I would start with something small and say: “deliver this, let me see how it works, give me every last line of code for further inspection, then we’ll talk more”. I mean, have them build a mini-site or something and make sure that they’re competent before unleashing them on the entire payroll system, right? And you don’t need some grand familiarity with the software industry to plan that way, it just makes sense to start smaller and make sure you know who you’re really dealing with (and also spending less money if it fails).
• patio11 9y ago
  Cross-posting a comment I made from 7 years ago when NYC's payroll project blew up: https://news.ycombinator.com/item?id=2716319 https://news.ycombinator.com/item?id=2716319 which I last cross-posted to the thread about Phoenix's payroll project blowing up: https://news.ycombinator.com/item?id=15303555 https://news.ycombinator.com/item?id=15303555
• dagw 9y ago
  Anybody have any insight on what it feels like working as consultant on such a huge failure? I (or my department, but honestly mostly me) once failed, as a consultant, to successfully deliver on a $50k contract and I felt like absolute shit for weeks and had trouble sleeping. I'd hate to think how it would feel to fail on this scale. Or is it a situation where the whole thing is so large and diffuse that no one can honestly be said to be responsible.
• harry8 9y ago
  Ibm global services, Accenture, computer associates, Oracle. See a company sign a big contact with them or their ilk, short that co. Their whole raison d'etre is to rip off old assholes in the c suite who can't find the on switch of the company or government money they're meant to be protecting. Hiring these consulting firms should get you fired every single time. It's a wonderful indicator of total incompetence. Notice how they're all deep into Washington? Yeah. Makes me sick these companies exist.
  • mrbill 9y ago
    Two common factors in a lot of these stories are "IBM GS" and "Oracle"...
• dalbasal 9y ago
  We really need to rethink the metasystems around these kinds of projects. A $1bn government payroll project is like the canonical "project that is guaranteed to suck" but realistically this extends pretty widely. These custom software builds, built around specific corporations, with all the sales and account management and expectation planning and making sure that *when" this fails, it is the other party's fault.... We need to find better ways of doing this stuff.
• cm2187 9y ago
  Hard to have an opinion without knowing the details but from my little experience I feel like the reason why these big project fail is not so much because they are too complex, I am sure a good (not excellent) developper could come up with an algorithm and data structure generic enough to accomodate all corner cases. But because these big projects start from a preexisting solution (PeopleSoft in this case), that wasn’t designed for this particular complexity, so it requires orders more energy to go around its limitations, and the people staffed by the likes of IBM on these projects are usually project managers with little to no programming skills and little domain knowledge. So you end up with a disaster for things that aren’t that complicated (custom shifts for coast guards and prison guards doesn’t look like a hard computer science problem).
• mathgladiator 9y ago
  I wrote my own payroll system for my house keeper, and it is surprising hard to get taxes right... I am not surprised.
• dgregd 9y ago
  The only solution to tax / accounting law and software problems is to create a reference accounting system and set it as a applicable law. In other words, to describe the law in unambiguous language. Do we really have to write a law in the same way as Romans did? Math as a science made progress after it invented a new precise language for itself. I'm not talking about the whole law. Only about that according to which the software is built. Since no one calculates taxes today by hand, it is worth to write the law so that it can be easily applied.
• kitd 9y ago
  Totally OT but The Phoenix project was originally chosen by Prime Minister Justin Trudeau’s Conservative predecessors 10 years ago seems an odd way to refer to Stephen Harper's government.
• tandr 9y ago
  I am genuine curious - Are there any recent examples of big, long but _successful_ projects that involves these 2 companies? (IBM and Oracle are mentioned in the article)