4 ms·

6 hours? They know 80's mainframe could do updates without any downtime right? Did someone mess up the Kubernetes config and shut it all down at the same time?

6 hours? They know 80's mainframe could do updates without any downtime right? Did someone mess up the Kubernetes config and shut it all down at the same time? The Canadian government is an example of how not to do software. Their Phoenix pay system is frequently used as a cautionnary tale for bad design and implementation [0]. [0] https://en.wikipedia.org/wiki/Phoenix_pay_system https://en.wikipedia.org/wiki/Phoenix_pay_system

triceratops 5y ago
> Instead of saving $70 million a year as planned, the report said that the cost to taxpayers to fix Phoenix's problems could reach a total of $2.2 billion by 2023. Ouch. And for those who think software engineering other than medical devices, aviation, or nuclear power plants isn't life-or-death: > Pascale Boulay, a Quebec coroner, determined that the 2017 death by suicide of a 52-year old woman from Val-des-Monts, Quebec, was preventable. The coroner assigned blame on the "flawed Phoenix pay system" that had "led her to emotional and financial ruin."
- bellyfullofbac 5y ago
  Also happened in the UK: https://www.theverge.com/2021/4/23/22399721/uk-post-office-software-bug-criminal-convictions-overturned https://www.theverge.com/2021/4/23/22399721/uk-post-office-s...
- 908B64B197 5y ago
  > And for those who think software engineering other than medical devices, aviation, or nuclear power plants isn't life-or-death I don't think a single real engineer was involved in this, sadly. Probably offshored contractors and clueless non-technical government employees. It's ironic because Quebec punch way above it's weight for engineering.
  - ayngg 5y ago
    It wasn't necessarily just IBM's fault either, it was complete incompetence by the government who continuously pushed the change through in the face of ample evidence (and IBM's own suggestion) that it should have been delayed. Not only did they push it through despite issues even after initial limited rollout, they made a ton of terrible decisions like to do training in house instead of from IBM as cost savings, made the new Public Service Pay Centre elsewhere for political reasons with new staff and they laid off most of the people that worked on the old system and decommissioned it so there was no going back.
    - 908B64B197 5y ago
      > It wasn't necessarily just IBM's fault either My understanding is that it wasn't IBM's fault at all. They delivered exactly what was aggreed upon. What the government asked made no sense. That's why they couldn't and didn't sue, and if you read about it in the media, there's no direct blame on IBM (a lot of it is implied; being to direct could be construed as defamation).
      - deleted 5y ago
        [deleted]
      - ayngg 5y ago
        Yeah that was what I thought too, but I don't have a full enough understanding to definitively say if IBM had any culpability or not.
      - yardstick 5y ago
        > My understanding is that it wasn't IBM's fault at all. They delivered exactly what was aggreed upon. What the government asked made no sense. If what the government asked made no sense, why did IBM agree to do it? Rhetorical question. Money talks.
        908B64B197 5y ago
        > If what the government asked made no sense, why did IBM agree to do it? My understanding is that the government gave IBM a complete spec of what they believed were their pay grades. But that wasn't what was agreed upon with the employees and unions (how do you calculate overtime on a coast guard boat?). So it all made sense to IBM, they shipped it and fufilled every requirements. They had no way of knowing these weren't the "correct" pay rules. It's the magic of government contracting: you write the spec and select the lowest bidder. Forget a 0 to the total weight the bridge must carry? You'll get very cheap proposals. Ohh and there's typically rules against a bidder contacting the government to get more information; that could be considered as interfering with the bidding process. It's also why older USPS vehicules had poor snow performance.
  - brailsafe 5y ago
    Most software people aren't "real" engineers in any capacity, especially not in Canada.
- icegreentea2 5y ago
  Phoenix was rotten from concept to post roll-out fuckery. It was broken before a single developer touched a line a code. Red flags independent of software development finesse or skill: - Centralized payroll operations to effectively the middle of no where (effectively ensuring that no existing payroll specialists would staff up the new center). Locate operations in the middle of no where to compensate for another government program and center being shut-down. - Sole source contract. To IBM no less. To customize and configure PeopleSoft. There's literally nothing that sends good vibes in that setup. - The very concept of running a big bang switch over of like 75% of the federal government to a new payroll system. - Launched without end to end, full system test - Launched KNOWING that critical functionality (like retroactive pay) was not implemented, and that critical requirements were failing tests. - Deciding to roll out despite their new centralized payroll center was already less efficient at handling payroll on legacy systems and that their central assumptions were already being challenged - Deciding to roll out despite receiving reports from client departments that the initial data transfer into the new system was already significantly incorrect - Quote: "In our opinion, they had received more than enough information and warning that Phoenix was not ready to be implemented, and therefore, they should not have proceeded as planned. Phoenix executives prioritized meeting schedule and cost over other critical elements, such as functionality and security, resulting in an incomprehensible failure of project management and oversight" You can read the audit here: https://www.oag-bvg.gc.ca/internet/english/parl_oag_201805_01_e_43033.html https://www.oag-bvg.gc.ca/internet/english/parl_oag_201805_0... Software is only tangentially related in that the "magic" of software lets people believe that such projects are even possible.
  - j_not_j 5y ago
    And none of the responsible GoC employees (I hesitate to call them "managers") was fired as a result of their mistakes.
    - kennywinker 5y ago
      Makes sense. Nobody Ever Got Fired for Buying IBM™
- OJFord 5y ago
  That seems oddly political for a coroner? (I'm not Canadian, this thread's the first I've heard of Phoenix Pay, I don't care.) Are suicides due to stress/depression usually so specific as to the cause that it's unavoidable? I would've assumed it'd just be 'emotional and financial ruin'.
  - 908B64B197 5y ago
    > I would've assumed it'd just be 'emotional and financial ruin'. In this case I think all her income was from a single source (the federal government). So it doesn't sound political to me as they were able to root cause the single source of financial ruin. It's like naming the drunk driver who slammed into a pedestrian while being completely alone on the road. You can always say "an anonymous driver" on the report ...
    - OJFord 5y ago
      I didn't know they did name the manslaughterer or even that they were drunk - that's what I meant by 'are they usually so specific that [highlighting Phoenix as the cause of the ruin] is unavoidabable'.
MattGaiser 5y ago
Thanks IBM.
deleted 5y ago
[deleted]
midasuni 5y ago
To err requires a computer. To really foul things up requires ansible.
jeromegv 5y ago
Canada Post is a crown corporation, they don't share employees with the government, it's entirely separate. And yes, Phoenix was awful, but separate point.

triceratops 5y ago

> Instead of saving $70 million a year as planned, the report said that the cost to taxpayers to fix Phoenix's problems could reach a total of $2.2 billion by 2023. Ouch. And for those who think software engineering other than medical devices, aviation, or nuclear power plants isn't life-or-death: > Pascale Boulay, a Quebec coroner, determined that the 2017 death by suicide of a 52-year old woman from Val-des-Monts, Quebec, was preventable. The coroner assigned blame on the "flawed Phoenix pay system" that had "led her to emotional and financial ruin."

bellyfullofbac 5y ago
Also happened in the UK: https://www.theverge.com/2021/4/23/22399721/uk-post-office-software-bug-criminal-convictions-overturned https://www.theverge.com/2021/4/23/22399721/uk-post-office-s...
908B64B197 5y ago
> And for those who think software engineering other than medical devices, aviation, or nuclear power plants isn't life-or-death I don't think a single real engineer was involved in this, sadly. Probably offshored contractors and clueless non-technical government employees. It's ironic because Quebec punch way above it's weight for engineering.
- ayngg 5y ago
  It wasn't necessarily just IBM's fault either, it was complete incompetence by the government who continuously pushed the change through in the face of ample evidence (and IBM's own suggestion) that it should have been delayed. Not only did they push it through despite issues even after initial limited rollout, they made a ton of terrible decisions like to do training in house instead of from IBM as cost savings, made the new Public Service Pay Centre elsewhere for political reasons with new staff and they laid off most of the people that worked on the old system and decommissioned it so there was no going back.
  - 908B64B197 5y ago
    > It wasn't necessarily just IBM's fault either My understanding is that it wasn't IBM's fault at all. They delivered exactly what was aggreed upon. What the government asked made no sense. That's why they couldn't and didn't sue, and if you read about it in the media, there's no direct blame on IBM (a lot of it is implied; being to direct could be construed as defamation).
    - deleted 5y ago
      [deleted]
    - ayngg 5y ago
      Yeah that was what I thought too, but I don't have a full enough understanding to definitively say if IBM had any culpability or not.
    - yardstick 5y ago
      > My understanding is that it wasn't IBM's fault at all. They delivered exactly what was aggreed upon. What the government asked made no sense. If what the government asked made no sense, why did IBM agree to do it? Rhetorical question. Money talks.
      - 908B64B197 5y ago
        > If what the government asked made no sense, why did IBM agree to do it? My understanding is that the government gave IBM a complete spec of what they believed were their pay grades. But that wasn't what was agreed upon with the employees and unions (how do you calculate overtime on a coast guard boat?). So it all made sense to IBM, they shipped it and fufilled every requirements. They had no way of knowing these weren't the "correct" pay rules. It's the magic of government contracting: you write the spec and select the lowest bidder. Forget a 0 to the total weight the bridge must carry? You'll get very cheap proposals. Ohh and there's typically rules against a bidder contacting the government to get more information; that could be considered as interfering with the bidding process. It's also why older USPS vehicules had poor snow performance.
- brailsafe 5y ago
  Most software people aren't "real" engineers in any capacity, especially not in Canada.
icegreentea2 5y ago
Phoenix was rotten from concept to post roll-out fuckery. It was broken before a single developer touched a line a code. Red flags independent of software development finesse or skill: - Centralized payroll operations to effectively the middle of no where (effectively ensuring that no existing payroll specialists would staff up the new center). Locate operations in the middle of no where to compensate for another government program and center being shut-down. - Sole source contract. To IBM no less. To customize and configure PeopleSoft. There's literally nothing that sends good vibes in that setup. - The very concept of running a big bang switch over of like 75% of the federal government to a new payroll system. - Launched without end to end, full system test - Launched KNOWING that critical functionality (like retroactive pay) was not implemented, and that critical requirements were failing tests. - Deciding to roll out despite their new centralized payroll center was already less efficient at handling payroll on legacy systems and that their central assumptions were already being challenged - Deciding to roll out despite receiving reports from client departments that the initial data transfer into the new system was already significantly incorrect - Quote: "In our opinion, they had received more than enough information and warning that Phoenix was not ready to be implemented, and therefore, they should not have proceeded as planned. Phoenix executives prioritized meeting schedule and cost over other critical elements, such as functionality and security, resulting in an incomprehensible failure of project management and oversight" You can read the audit here: https://www.oag-bvg.gc.ca/internet/english/parl_oag_201805_01_e_43033.html https://www.oag-bvg.gc.ca/internet/english/parl_oag_201805_0... Software is only tangentially related in that the "magic" of software lets people believe that such projects are even possible.
- j_not_j 5y ago
  And none of the responsible GoC employees (I hesitate to call them "managers") was fired as a result of their mistakes.
  - kennywinker 5y ago
    Makes sense. Nobody Ever Got Fired for Buying IBM™
OJFord 5y ago
That seems oddly political for a coroner? (I'm not Canadian, this thread's the first I've heard of Phoenix Pay, I don't care.) Are suicides due to stress/depression usually so specific as to the cause that it's unavoidable? I would've assumed it'd just be 'emotional and financial ruin'.
- 908B64B197 5y ago
  > I would've assumed it'd just be 'emotional and financial ruin'. In this case I think all her income was from a single source (the federal government). So it doesn't sound political to me as they were able to root cause the single source of financial ruin. It's like naming the drunk driver who slammed into a pedestrian while being completely alone on the road. You can always say "an anonymous driver" on the report ...
  - OJFord 5y ago
    I didn't know they did name the manslaughterer or even that they were drunk - that's what I meant by 'are they usually so specific that [highlighting Phoenix as the cause of the ruin] is unavoidabable'.

MattGaiser 5y ago

Thanks IBM.

deleted 5y ago

[deleted]

midasuni 5y ago

To err requires a computer. To really foul things up requires ansible.

jeromegv 5y ago

Canada Post is a crown corporation, they don't share employees with the government, it's entirely separate. And yes, Phoenix was awful, but separate point.