9 ms·

Tell HN: Canada Posts systems to be down for 6 hours of “unplanned maintenance”

87 points by neom 5y ago

neom 5y ago
I just got off the phone with Canada Post because my package had not shown up on the Canada Post tracking system when the package was handed off from USPS to them yesterday. Guy on the phone said they've been having "issues" and he can't do any service tickets or status lookups because their internal systems are all offline for the next 6 hours due to "unplanned maintenance"
belltaco 5y ago
Should be 'Tell HN', not 'Show HN'.
- gnicholas 5y ago
  Unless it was posted by the person who took the systems down. /s
perlgeek 5y ago
What are the odds this will turn out to be due to ransomware?
- admax88qqq 5y ago
  That or they just realized they're using log4j
  - perlgeek 5y ago
    Would be a pretty fun exploit if you printed it on the address field of a letter, OCR recognizes it, cannot parse it as a postal address, logs the failure using log4j, thus giving you access to their post routing system...
    - inkeddeveloper 5y ago
      Yes....fun...
    - rbobby 5y ago
      Sounds like a plan... until you realize why your label printer keeps restarting.
  - tyingq 5y ago
    Their cookies start with "JSESSIONID", so I'm guessing you're right.
  - m-p-3 5y ago
    And subsequently got a ransomware because of it.
kache_ 5y ago
Canada suffers from losing all their competent tech workers to jobs that actually pay them market rate :^)
- Naga 5y ago
  It's actually even worse for government (or crown corps like Canada Post), since they pay way below Canadian market rate, which is already way below U.S. salaries. The government then isn't really in competition for the best talent who want the most money.
  - poxrud 5y ago
    Like most government dev work, Canada Post outsources to Accenture for their projects.
    - dzhiurgis 5y ago
      This. Exactly everywhere, not government only. Why pay market rates to actual developers when you can get bottom of the barrel scum for $2000 a day
      - baseballMan 5y ago
        All about that capex vs opex!
  - totony 5y ago
    Not only that, most gov projects have to outsource (that's why we have such big consulting firms like CGI and LGS). This make project costs skyrocket. All that because they can't offer good wages (or condition - working conditions are pretty much set in stone).
  - xvf22 5y ago
    And in Ontario public service, raises are capped to 1% yearly including benefits.
- 5y ago
deleted 5y ago
[deleted]
908B64B197 5y ago
6 hours? They know 80's mainframe could do updates without any downtime right? Did someone mess up the Kubernetes config and shut it all down at the same time? The Canadian government is an example of how not to do software. Their Phoenix pay system is frequently used as a cautionnary tale for bad design and implementation [0]. [0] https://en.wikipedia.org/wiki/Phoenix_pay_system https://en.wikipedia.org/wiki/Phoenix_pay_system
- triceratops 5y ago
  > Instead of saving $70 million a year as planned, the report said that the cost to taxpayers to fix Phoenix's problems could reach a total of $2.2 billion by 2023. Ouch. And for those who think software engineering other than medical devices, aviation, or nuclear power plants isn't life-or-death: > Pascale Boulay, a Quebec coroner, determined that the 2017 death by suicide of a 52-year old woman from Val-des-Monts, Quebec, was preventable. The coroner assigned blame on the "flawed Phoenix pay system" that had "led her to emotional and financial ruin."
  - bellyfullofbac 5y ago
    Also happened in the UK: https://www.theverge.com/2021/4/23/22399721/uk-post-office-software-bug-criminal-convictions-overturned https://www.theverge.com/2021/4/23/22399721/uk-post-office-s...
  - 908B64B197 5y ago
    > And for those who think software engineering other than medical devices, aviation, or nuclear power plants isn't life-or-death I don't think a single real engineer was involved in this, sadly. Probably offshored contractors and clueless non-technical government employees. It's ironic because Quebec punch way above it's weight for engineering.
    - ayngg 5y ago
      It wasn't necessarily just IBM's fault either, it was complete incompetence by the government who continuously pushed the change through in the face of ample evidence (and IBM's own suggestion) that it should have been delayed. Not only did they push it through despite issues even after initial limited rollout, they made a ton of terrible decisions like to do training in house instead of from IBM as cost savings, made the new Public Service Pay Centre elsewhere for political reasons with new staff and they laid off most of the people that worked on the old system and decommissioned it so there was no going back.
abrowne 5y ago
Last year, someone signed up for PO box-equivalent with my firstnamelastname@gmail.com address. (Not the first time this guy in Montréal forgot that his email has a 0 before the @gmail.com. Or an error from the person entering it in.) I live in the US. I have lived in Canada, but over 10 years ago now. The only two options to contact Canada Post were phone or … Twitter — which I don't have an account for — no email or web form. And the phone CSR had to get a manager who still couldn't figure out how to deal with the issue since I wasn't a customer myself and just wanted my email removed from the account.
- jeromegv 5y ago
  You can try cmg@canadapost.postescanada.ca
  - abrowne 5y ago
    Good to know if there's a next time. I forgot to say that another option was to contact them by mail, so at least they're dogfooding ;-)
  - 1over137 5y ago
    The domain is canadapost-postescanada.ca
    - gompertz 5y ago
      It's cute in Canada, they would feel the need to go so incredibly far as having the domain bilingual, rather than just registering two separate domains and pointing them to the same address. If you actually (for God knows what reason) struggle so hard to see the similarity between French and English in this two word identifier, then surely putting them both on the same domain is only going to cause issues?
      - neom 5y ago
        Canadians take our bilingualism seriously. To a degree that calling it "cute" would be offensive to many. It's the law that French and English need to be displayed & provided "together" in government services, I presume this is to avoid any issues there. "The Official Languages Act is a Canadian law that came into force on September 9, 1969, which gives French and English equal status in the government of Canada" (although I believe in Quebec, French is by law to be displayed before or over English)
sh4un 5y ago
Sounds normal.
rndmind 5y ago

87 points by neom 5y ago

neom 5y ago

I just got off the phone with Canada Post because my package had not shown up on the Canada Post tracking system when the package was handed off from USPS to them yesterday. Guy on the phone said they've been having "issues" and he can't do any service tickets or status lookups because their internal systems are all offline for the next 6 hours due to "unplanned maintenance"

belltaco 5y ago

Should be 'Tell HN', not 'Show HN'.

gnicholas 5y ago
Unless it was posted by the person who took the systems down. /s

perlgeek 5y ago

What are the odds this will turn out to be due to ransomware?

admax88qqq 5y ago
That or they just realized they're using log4j
- perlgeek 5y ago
  Would be a pretty fun exploit if you printed it on the address field of a letter, OCR recognizes it, cannot parse it as a postal address, logs the failure using log4j, thus giving you access to their post routing system...
  - inkeddeveloper 5y ago
    Yes....fun...
  - rbobby 5y ago
    Sounds like a plan... until you realize why your label printer keeps restarting.
- tyingq 5y ago
  Their cookies start with "JSESSIONID", so I'm guessing you're right.
- m-p-3 5y ago
  And subsequently got a ransomware because of it.

kache_ 5y ago

Canada suffers from losing all their competent tech workers to jobs that actually pay them market rate :^)

Naga 5y ago
It's actually even worse for government (or crown corps like Canada Post), since they pay way below Canadian market rate, which is already way below U.S. salaries. The government then isn't really in competition for the best talent who want the most money.
- poxrud 5y ago
  Like most government dev work, Canada Post outsources to Accenture for their projects.
  - dzhiurgis 5y ago
    This. Exactly everywhere, not government only. Why pay market rates to actual developers when you can get bottom of the barrel scum for $2000 a day
    - baseballMan 5y ago
      All about that capex vs opex!
- totony 5y ago
  Not only that, most gov projects have to outsource (that's why we have such big consulting firms like CGI and LGS). This make project costs skyrocket. All that because they can't offer good wages (or condition - working conditions are pretty much set in stone).
- xvf22 5y ago
  And in Ontario public service, raises are capped to 1% yearly including benefits.
5y ago

deleted 5y ago

[deleted]

908B64B197 5y ago

6 hours? They know 80's mainframe could do updates without any downtime right? Did someone mess up the Kubernetes config and shut it all down at the same time? The Canadian government is an example of how not to do software. Their Phoenix pay system is frequently used as a cautionnary tale for bad design and implementation [0]. [0] https://en.wikipedia.org/wiki/Phoenix_pay_system https://en.wikipedia.org/wiki/Phoenix_pay_system

triceratops 5y ago
> Instead of saving $70 million a year as planned, the report said that the cost to taxpayers to fix Phoenix's problems could reach a total of $2.2 billion by 2023. Ouch. And for those who think software engineering other than medical devices, aviation, or nuclear power plants isn't life-or-death: > Pascale Boulay, a Quebec coroner, determined that the 2017 death by suicide of a 52-year old woman from Val-des-Monts, Quebec, was preventable. The coroner assigned blame on the "flawed Phoenix pay system" that had "led her to emotional and financial ruin."
- bellyfullofbac 5y ago
  Also happened in the UK: https://www.theverge.com/2021/4/23/22399721/uk-post-office-software-bug-criminal-convictions-overturned https://www.theverge.com/2021/4/23/22399721/uk-post-office-s...
- 908B64B197 5y ago
  > And for those who think software engineering other than medical devices, aviation, or nuclear power plants isn't life-or-death I don't think a single real engineer was involved in this, sadly. Probably offshored contractors and clueless non-technical government employees. It's ironic because Quebec punch way above it's weight for engineering.
  - ayngg 5y ago
    It wasn't necessarily just IBM's fault either, it was complete incompetence by the government who continuously pushed the change through in the face of ample evidence (and IBM's own suggestion) that it should have been delayed. Not only did they push it through despite issues even after initial limited rollout, they made a ton of terrible decisions like to do training in house instead of from IBM as cost savings, made the new Public Service Pay Centre elsewhere for political reasons with new staff and they laid off most of the people that worked on the old system and decommissioned it so there was no going back.

abrowne 5y ago

Last year, someone signed up for PO box-equivalent with my firstnamelastname@gmail.com address. (Not the first time this guy in Montréal forgot that his email has a 0 before the @gmail.com. Or an error from the person entering it in.) I live in the US. I have lived in Canada, but over 10 years ago now. The only two options to contact Canada Post were phone or … Twitter — which I don't have an account for — no email or web form. And the phone CSR had to get a manager who still couldn't figure out how to deal with the issue since I wasn't a customer myself and just wanted my email removed from the account.

jeromegv 5y ago
You can try cmg@canadapost.postescanada.ca
- abrowne 5y ago
  Good to know if there's a next time. I forgot to say that another option was to contact them by mail, so at least they're dogfooding ;-)
- 1over137 5y ago
  The domain is canadapost-postescanada.ca
  - gompertz 5y ago
    It's cute in Canada, they would feel the need to go so incredibly far as having the domain bilingual, rather than just registering two separate domains and pointing them to the same address. If you actually (for God knows what reason) struggle so hard to see the similarity between French and English in this two word identifier, then surely putting them both on the same domain is only going to cause issues?
    - neom 5y ago
      Canadians take our bilingualism seriously. To a degree that calling it "cute" would be offensive to many. It's the law that French and English need to be displayed & provided "together" in government services, I presume this is to avoid any issues there. "The Official Languages Act is a Canadian law that came into force on September 9, 1969, which gives French and English equal status in the government of Canada" (although I believe in Quebec, French is by law to be displayed before or over English)

sh4un 5y ago

Sounds normal.

rndmind 5y ago