5 ms·

Botching a prod update -- that happens to the best of us. And sorry to say I doubt Rogers would attract the best of us but I digress. But only kind of digress.

by chx 4y ago

Botching a prod update -- that happens to the best of us. And sorry to say I doubt Rogers would attract the best of us but I digress. But only kind of digress. When the human waste hits the circular cooler device, that's when we find out how well an organization is built and managed. Who is surprised here it took them this long to recover? The necessary no-blame culture, temporary removal of decision barriers, all-hands-on-deck to get back to normal, all the things the lucky of us take for granted are guaranteed to be missing here.

Victerius 4y ago
> And sorry to say I doubt Rogers would attract the best of us but I digress "The typical Rogers Communications Systems Administrator salary is $67,329" (https://www.glassdoor.ca/Salary/Rogers-Communications-Systems-Administrator-Salaries-E3783_D_KO22,43.htm https://www.glassdoor.ca/Salary/Rogers-Communications-System...) "The average system administrator salary in Canada is $71,981 per year or $36.91 per hour. Entry-level positions start at $61,425 per year, while most experienced workers make up to $94,400 per year." (https://ca.talent.com/salary?job=system+administrator https://ca.talent.com/salary?job=system+administrator)
- doubled112 4y ago
  Maybe related are the exceptions to labour laws while you make your 67K a year https://www.ontario.ca/document/industries-and-jobs-exemptions-or-special-rules/government-employees-and-professionals#section-3 https://www.ontario.ca/document/industries-and-jobs-exemptio... But I'm sure Rogers would never abuse any of those.
- eru 4y ago
  That's not a lot of money for software folks. Especially folks with access to the American labour market.
- amscanne 4y ago
  Note that $1 CAD is also $0.80 USD, which is often omitted but meaningful when comparing pay just over the border.
  - jonny_eh 4y ago
    And taxes, goods, and services are a little more expensive in Canada.
    - aledalgrande 4y ago
      and traveling overseas
      - Scoundreller 4y ago
        Is it? Flights from Canada to Europe or Asia can be cost-effective. It's domestic flights and CAN<->USA flights that can cost you.
        aledalgrande 4y ago
        They can be cost effective only if you can travel in quiet parts of the year. I was also mainly referring to buying things in Europe/US/UK with a salary paid in a weaker currency.
- rajamaka 4y ago
  Median house price in Vancouver - $1,341,643
  - doubled112 4y ago
    You could move to Toronto where the average house price is said to be $1,167,000. Ottawa is cheap at around $965,000.
Insanity 4y ago
I doubt any of the IT infrastructure in Canada is build by the “best of us”. And it shows, banking, healthcare, telecom, they are all stuck somewhere in the 90s.
- ilrwbwrkhv 4y ago
  I doubt any "best of us" stay in Canada. It is a fiercely mediocre place.
  - jonny_eh 4y ago
    The US is trying very hard to convince the best of us to move back to Canada.
    - dylan604 4y ago
      As well as making some of the natives wish they could go with (but that's pretty much been the case since at least late 60s.
  - fezfight 4y ago
  - chx 4y ago
    When I was deciding on where to immigrate to in 2006, I had job offers from the UK, Canada and the USA. I ruled out the USA because of the CFAA. I do not like second guessing everything I do online whether it violates a badly written federal law. Of course, United States v. Elcom Ltd. didn't make me much happier. Three Felonies A Day came out later and it just strengthens all this. We also should mention the almost complete lack of social net and the insane health care system. There are numerous arguments that by now the United States is a failed state https://www.thenation.com/article/world/trump-usa-election/ https://www.thenation.com/article/world/trump-usa-election/ https://eand.co/how-america-collapsed-and-became-a-fourth-world-country-db9829323a9a https://eand.co/how-america-collapsed-and-became-a-fourth-wo... etc.
- bloodyplonker22 4y ago
throwawaysleep 4y ago
A friend has one of his jobs with a telecom company. Everything there is political. He bets that the day the outage happened, as many devs as possible would have tried to call in sick as that is what happens at his firm (Telus/Bell). I would have. I call in sick during prod failures at one of my gov jobs. In jobs with no real rewards for excellence but decent job security, a crisis is a reason to get out the door.
no-dr-onboard 4y ago
> no-blame culture We have this where I work and I have mixed feelings about it. On one hand, it's certainly freeing that we don't get fired for mistakes. On another, it's strange to feel that the people who objectively caused the headache for everyone else go without serious attention to remediation. IMO, we don't have to name names, but Root Cause Analysis should include the decision tree that the primary motivator chose. No blame culture also hampers a lot of insider-threat analysis.
- robalfonso 4y ago
  No-blame culture and no-consequence culture are not the same thing. Just because you don’t get fired doesn’t mean the org should allow someone the capability to make the same mistake, whether that’s access/permissions/change in org etc
- Sebb767 4y ago
  I think the idea behind no-blame culture is that a single person shouldn't be able to cause a massive incident on their own (assuming no malice), so finding a victim to put the blame on isn't helpful anyway. Also, if you start blaming people, everyone will go in permanent cover-my-ass-mode and this will usually just lead to some poor junior, who couldn't play the game yet, getting the full blow. Of course, letting people run rampant and acting everything is fine is not great, but there's a good chance it might be an organizational problem and, if not, a HR problem (which is not necessarily related to a specific incident).
- saghm 4y ago
  I think the idea behind no-blame is that there are basically three ways that an outage could be the fault of an individual: maliciousness, incompetence, or bad process. In the first two cases, the individual should be fired, and in the third case, blaming the individual is counterproductive as it not only doesn't do anything to mitigate the issue but also distracts from the real cause. The only gray area I can think of is where it's not not clear from a single incident whether an individual is actually incompetent or if they're "borderline competent" (maybe more competent at other aspects of the job role and it's a balance to figure out if they're net value is worth keeping around), but in that case, I think that's also probably better to be dealt with at the management level and keep out of the root cause analysis. > MO, we don't have to name names, but Root Cause Analysis should include the decision tree that the primary motivator chose. I definitely agree with this! I actually have seen cases where people have asked for revisions to a root cause analysis not due to a lack of naming individuals but a lack of sufficient explanation for what the anonymous individual actually did to cause the outage, which I think is fair feedback to give. > No blame culture also hampers a lot of insider-threat analysis. Yeah, if no-blame culture is being used as a shield for maliciousness or incompetence, I think that could have a harmful effect on things. I guess I always just interpreted "no-blame" as actually meaning "no blame for honest mistakes", but I guess that's too long to be catchy.
throw0101a 4y ago
> Who is surprised here it took them this long to recover? They have no independent management infrastructure. Their management network rides logically on top of their main physical network. So when the latter went down you probably had people going to the physical locations of various DCs. But then you have to coördinate between them, and if your employees use your own cell phone service for communication, and that's down… So the first couple of hours was figuring out what went wrong, then it was having people scramble to locations to plug in serial consoles, then it was buying a bunch of SIM card from other companies, and finally folks managed to wrap their heads around the big picture and come up with a way to bootstrap the network.
- mjevans 4y ago
  An independent management infra would also help to assess as well as recover. There might be a need to require either such an independent network, or one that fully does not reuse your own internal network as a backup link for the management interface.
- yourad_io 4y ago
  > buying a bunch of SIM card from other companies Sounds like some employees already had those for such an event, which is impressive. From the article: > some employees started swapping out their SIM cards for Bell or Telus SIM cards that they had received back in 2015 as part of an emergency contingency plan established between the wireless carriers.
kache_ 4y ago
Not me. Canadian "tech" has 0 talent left. We're all working for the much better paying american mega corps Even shopify is getting bled dry
- Gunnerhead 4y ago
  As a Canadian who moved to the US to work in big tech, I love it down here!
deleted 4y ago
[deleted]

by chx 4y ago

Victerius 4y ago

> And sorry to say I doubt Rogers would attract the best of us but I digress "The typical Rogers Communications Systems Administrator salary is $67,329" (https://www.glassdoor.ca/Salary/Rogers-Communications-Systems-Administrator-Salaries-E3783_D_KO22,43.htm https://www.glassdoor.ca/Salary/Rogers-Communications-System...) "The average system administrator salary in Canada is $71,981 per year or $36.91 per hour. Entry-level positions start at $61,425 per year, while most experienced workers make up to $94,400 per year." (https://ca.talent.com/salary?job=system+administrator https://ca.talent.com/salary?job=system+administrator)

doubled112 4y ago
Maybe related are the exceptions to labour laws while you make your 67K a year https://www.ontario.ca/document/industries-and-jobs-exemptions-or-special-rules/government-employees-and-professionals#section-3 https://www.ontario.ca/document/industries-and-jobs-exemptio... But I'm sure Rogers would never abuse any of those.
eru 4y ago
That's not a lot of money for software folks. Especially folks with access to the American labour market.
amscanne 4y ago
Note that $1 CAD is also $0.80 USD, which is often omitted but meaningful when comparing pay just over the border.
- jonny_eh 4y ago
  And taxes, goods, and services are a little more expensive in Canada.
  - aledalgrande 4y ago
    and traveling overseas
    - Scoundreller 4y ago
      Is it? Flights from Canada to Europe or Asia can be cost-effective. It's domestic flights and CAN<->USA flights that can cost you.
      - aledalgrande 4y ago
        They can be cost effective only if you can travel in quiet parts of the year. I was also mainly referring to buying things in Europe/US/UK with a salary paid in a weaker currency.
rajamaka 4y ago
Median house price in Vancouver - $1,341,643
- doubled112 4y ago
  You could move to Toronto where the average house price is said to be $1,167,000. Ottawa is cheap at around $965,000.

Insanity 4y ago

I doubt any of the IT infrastructure in Canada is build by the “best of us”. And it shows, banking, healthcare, telecom, they are all stuck somewhere in the 90s.

ilrwbwrkhv 4y ago
I doubt any "best of us" stay in Canada. It is a fiercely mediocre place.
- jonny_eh 4y ago
  The US is trying very hard to convince the best of us to move back to Canada.
  - dylan604 4y ago
    As well as making some of the natives wish they could go with (but that's pretty much been the case since at least late 60s.
- fezfight 4y ago
- chx 4y ago
  When I was deciding on where to immigrate to in 2006, I had job offers from the UK, Canada and the USA. I ruled out the USA because of the CFAA. I do not like second guessing everything I do online whether it violates a badly written federal law. Of course, United States v. Elcom Ltd. didn't make me much happier. Three Felonies A Day came out later and it just strengthens all this. We also should mention the almost complete lack of social net and the insane health care system. There are numerous arguments that by now the United States is a failed state https://www.thenation.com/article/world/trump-usa-election/ https://www.thenation.com/article/world/trump-usa-election/ https://eand.co/how-america-collapsed-and-became-a-fourth-world-country-db9829323a9a https://eand.co/how-america-collapsed-and-became-a-fourth-wo... etc.
bloodyplonker22 4y ago

throwawaysleep 4y ago

A friend has one of his jobs with a telecom company. Everything there is political. He bets that the day the outage happened, as many devs as possible would have tried to call in sick as that is what happens at his firm (Telus/Bell). I would have. I call in sick during prod failures at one of my gov jobs. In jobs with no real rewards for excellence but decent job security, a crisis is a reason to get out the door.

no-dr-onboard 4y ago

> no-blame culture We have this where I work and I have mixed feelings about it. On one hand, it's certainly freeing that we don't get fired for mistakes. On another, it's strange to feel that the people who objectively caused the headache for everyone else go without serious attention to remediation. IMO, we don't have to name names, but Root Cause Analysis should include the decision tree that the primary motivator chose. No blame culture also hampers a lot of insider-threat analysis.

robalfonso 4y ago
No-blame culture and no-consequence culture are not the same thing. Just because you don’t get fired doesn’t mean the org should allow someone the capability to make the same mistake, whether that’s access/permissions/change in org etc
Sebb767 4y ago
I think the idea behind no-blame culture is that a single person shouldn't be able to cause a massive incident on their own (assuming no malice), so finding a victim to put the blame on isn't helpful anyway. Also, if you start blaming people, everyone will go in permanent cover-my-ass-mode and this will usually just lead to some poor junior, who couldn't play the game yet, getting the full blow. Of course, letting people run rampant and acting everything is fine is not great, but there's a good chance it might be an organizational problem and, if not, a HR problem (which is not necessarily related to a specific incident).
saghm 4y ago
I think the idea behind no-blame is that there are basically three ways that an outage could be the fault of an individual: maliciousness, incompetence, or bad process. In the first two cases, the individual should be fired, and in the third case, blaming the individual is counterproductive as it not only doesn't do anything to mitigate the issue but also distracts from the real cause. The only gray area I can think of is where it's not not clear from a single incident whether an individual is actually incompetent or if they're "borderline competent" (maybe more competent at other aspects of the job role and it's a balance to figure out if they're net value is worth keeping around), but in that case, I think that's also probably better to be dealt with at the management level and keep out of the root cause analysis. > MO, we don't have to name names, but Root Cause Analysis should include the decision tree that the primary motivator chose. I definitely agree with this! I actually have seen cases where people have asked for revisions to a root cause analysis not due to a lack of naming individuals but a lack of sufficient explanation for what the anonymous individual actually did to cause the outage, which I think is fair feedback to give. > No blame culture also hampers a lot of insider-threat analysis. Yeah, if no-blame culture is being used as a shield for maliciousness or incompetence, I think that could have a harmful effect on things. I guess I always just interpreted "no-blame" as actually meaning "no blame for honest mistakes", but I guess that's too long to be catchy.

throw0101a 4y ago

> Who is surprised here it took them this long to recover? They have no independent management infrastructure. Their management network rides logically on top of their main physical network. So when the latter went down you probably had people going to the physical locations of various DCs. But then you have to coördinate between them, and if your employees use your own cell phone service for communication, and that's down… So the first couple of hours was figuring out what went wrong, then it was having people scramble to locations to plug in serial consoles, then it was buying a bunch of SIM card from other companies, and finally folks managed to wrap their heads around the big picture and come up with a way to bootstrap the network.

mjevans 4y ago
An independent management infra would also help to assess as well as recover. There might be a need to require either such an independent network, or one that fully does not reuse your own internal network as a backup link for the management interface.
yourad_io 4y ago
> buying a bunch of SIM card from other companies Sounds like some employees already had those for such an event, which is impressive. From the article: > some employees started swapping out their SIM cards for Bell or Telus SIM cards that they had received back in 2015 as part of an emergency contingency plan established between the wireless carriers.

kache_ 4y ago

Not me. Canadian "tech" has 0 talent left. We're all working for the much better paying american mega corps Even shopify is getting bled dry

Gunnerhead 4y ago
As a Canadian who moved to the US to work in big tech, I love it down here!

deleted 4y ago

[deleted]