6 ms·
Rogers networks reliability and resiliency assessment after 2022-07-08 outage
- elchief 2y ago> Rogers staff relied on the company’s own mobile and Internet services for connectivity to communicate among themselves. When both the wireless and wireline networks failed, Rogers staff, especially critical incident management staff, were not able to communicate effectively during the early hours of the outage. Rogers had to send Subscriber Identity Module (SIM) cards from other mobile network operators to its remote sites to enable its staff with wireless connectivity to communicate with each other ooops. that's actually quite funny in case you didn't know, Rogers is one of canada's "big 3" telecom providers. The outage in 2022 basically crippled our economy for a couple days (most ATMs / interac didn't work)
- bcrl 2y agoThankfully none of my wavelength services from Rogers were impacted. =-)
- nightshift1 2y agoThat outage is actually a good argument against going cashless.
- endgame 2y agoHappens semi-regularly in Australia, too. More often because of fires or floods than network incompetence.
- maltalex 2y agoThe "lessons learned" are all very basic. Things like "separate the network management layer from the data network", "provide the network operation center with backup connectivity", etc. This is networking 101. Heck, this is engineering 101. The real question is how a network provider as large as Rogers managed to be so poorly engineered in the first place.
- intunderflow 2y agoMonopolization and corruption Rogers isn't the massive network it is because Canadian consumers love them and want to use them - they just have no choice, and the government does all it can to keep it that way
- ilrwbwrkhv 2y agoCanadians are sheep like in their mentally and incredibly risk averse. So there is zero competition.
- jszymborski 2y agoI don't think this is an acceptable way to speak about 41 million people on the basis of their nationality.
- piuantiderp 2y agoWhat is a good way to speak on the basis of nationality or it is always bad (tm)?
- lm411 2y agoNetworking 101 doesn't apply when you have collusion between the only two major telecom companies... and they both are bad. Why spend money on reliability when there are no consequences of not doing so?
- xyst 2y agoRogers is a monopoly. There is no incentive to improve. No incentive to hire good people on the floor to fix fundamental issues. Just middle management and c-level suite hoarding all of the money. At this point, the government can fine them but it’s just passed on to the consumers. Maybe a random regulatory fee. Can’t close them down otherwise risk massive downtime to critical infra (emergency services)
- gary_0 2y agoSomeone already mentioned the monopolism/corruption, but Canada also doesn't have much of an engineering/technology culture. Growing up in a BC suburb, it seemed like we were always 5-10 years behind the US on things like home computers, the Internet, cell phones, and e-commerce. And when that stuff did finally arrive, it cost far more than it did down South. And most of the people around me didn't seem to care; nobody I knew was interested in technology, either as an enthusiast or as a consumer. And anyone who is good at technology generally moves across the border for the much better wages--and actually being able to work at a real tech company. In short: brain drain is a factor too.
- ttul 2y agoNote that this failure didn’t prevent Rogers from buying Shaw and securing an even larger position in the national telecom system. In BC, for instance, there are only two mobile networks now (Telus and Rogers). That places an extremely high burden on two lazy oligopolists to maintain exceptional reliability. It’s a shame on Canada and on Canadians that foreign competitors are still prohibited from coming into the market.
- lm411 2y agoYep. Allowing Rogers to purchase Shaw was absurd. The Commissioner of Competition was very strongly opposed to the merger, the court ruled against him, and Canadians had to pay Rogers $13 million CAD because he opposed it. I was a Shaw Mobile (not Freedom) customer, and Rogers wanted me to drive 600km (both ways) through the Kootenay mountains in winter to get my new SIM cards. I told them where to go. Rogers is junk and Canada is an oligarchy.
- nikolajan 2y agoWhat made Canada special died decades ago, were it not for the strict visa process I would be living in the states (Vermont) right now, with a significantly higher salary (as a principal engineer), a lower income tax, healthier housing market, lower cost of living, and better quality of life.
- snapplebobapple 2y agoCanada used to say our healthcare was better but that hasnt been true for atleast a decade. Its more choose your version of crappy. Do you want to oay with your life savings (usa) or your life (canada)
- lm411 2y agoI (born and raised Canadian) moved my family to Canada 5 years ago. We've been on a waiting list for a doctor ever since. My only option to see a doctor here is to go the emergency room at the closest hospital.
- Insanity 2y agoFor fellow Canadians, I’m quite happy using Beanfield in Toronto. Cheap and decent. Their backbone is probably still Bell or Rogers though, but at least I pay a reasonable $60 for 2gbps down.
- da768 2y agoUsually smaller ISPs use the Bell/Rogers for last mile, but have their own backbone. IIRC Beanfield is completely independent, but is only available in Toronto
- bcrl 2y agoNo, Beanfield is probably not solely dependent on Bell and/or Rogers for transit. Toronto is one of the few cities in Canada where there are loads of transit providers to purchase bandwidth from (including most tier 1s). There are also multiple carriers with transport fibre going east and west from Toronto that is not owned by Bell or Rogers. Bell and Rogers are not the most competitive for transit, but they may well be in the mix that Beanfield uses. That said, if 151 Front Street West gets wiped off the face of the planet, the internet is probably going to be degraded for just about everyone in Ontario.
- shrubble 2y agoKye Prigg left Rogers shortly thereafter and ended up at Lumen (merger of CenturyLink and Level3), which is another company facing some issues: https://ir.lumen.com/news/news-details/2023/Lumen-Strengthens-Leadership-Team-adds-Kye-Prigg-as-Executive-Vice-President---Enterprise-Operations/default.aspx https://ir.lumen.com/news/news-details/2023/Lumen-Strengthen... "Kye most recently served as Senior Vice President of Access Networks and Operations for Rogers Communications. "
- canada_dry 2y agoA Roger's exec used to brag that they were able to release major changes faster than anyone because of their strategy which was "push it out... the customer's will find the remaining bugs". This was more than a decade ago, but it seems like it is still their culture.
- deleted 2y ago[deleted]
- WaitWaitWha 2y agoIn my opinion, from what I am reading, there is a root cause analysis in there under the heading "Reliability of Rogers network architecture". I removed the redundant and contradictory parts. > [...] both the wireless and wireline networks sharing a common IP core network, the scope of the outage was extreme in that it resulted in a catastrophic loss of all services. [...] It is a design choice by [...] Rogers, that seeks to balance cost with performance. Based on this write up and details I gathered, I believe the root cause, the fundamental reason for the failure is incorrect cost-to-performance balance at senior management.
- deleted 2y ago[deleted]
- great_psy 2y agoI have heard the deployment process at Rogers is something along the lines of: 1) get approval for your changes 2) ssh into a server 3) copy a backup of current jar file 4) copy the new instance 5) run some command to refresh
- jtchang 2y ago> The July 2022 outage was not the result of a design flaw in the Rogers core network architecture. Talk about some grade A gaslighting here. Reading the post mortem they first tell you it wasn't a design flaw then say they routed all their data through one core router ( including a lack of a management network). Then they say they are going to fix things by separating out the wireless and wired traffic. Why would you fix things if it wasn't a design flaw? Out of band access is like resilient architecture 101. Hell even homelabs generally have some way to do it. It's appalling that Rogers didn't have a way to access the core IP routers out of band. Yes it might mean having to use a competitors infrastructure but they ended up having to do it anyway. And with the failure of the service now all the infrastructure providers are under additional scrutiny. Rogers should be striking some agreements with other providers to carry core traffic in case of an outage such as in this DR situation. For example Visa, MC, Amex all have agreements in place to process each others auth data in case the other party goes down. The thinking here being an outage for credit cards makes everyone look bad.
- yelnatz 2y agoHilarious. Rogers runs the internet and phone lines so when it went down, the devs couldn't: 1) Remote into the boxes to see what's happening. 2) Talk to other devs because their phones are on Rogers network. My stress couldn't.
- atyvr 2y agoOutages are inevitable, but the Rogers outage in 2022 had some devastating consequences. Route leaks can happen to anyone, but the fact that it brought down their entire network, including voice and internet services, across all provinces, was unacceptable. What's even more concerning is that they had no out-of-band access, which meant no management access to their network. This explains why the outage lasted a whopping 24 hours. In my opinion, the lack of OOB was the most critical and yet the most preventable. Proper OOB is a must; I wouldn't operate a network without it, I don't understand why Rogers thought that was acceptable.
- JohnDouse 2y ago[dead]
- kn100 2y agoCan we for a second appreciate how freaking cool it is that a government agency published this?
- threesevenths 2y agoI wish they didn’t need to and we had transparency policies in place that meant Rogers would have published it at their expense. But yeah, better than. Nothing :)
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- luna4151 2y ago[dead]