4 ms·

Ask HN: Doesn't the Rogers outage prove – network as a single point of failure?

High Availability and Disaster Recovery and Auto-Scaling and a design in place to avoid SPOF (Single Point of Failure) always ignore Network being a SPOF. But, Network can be a Single Point of Failure too.

wmf 4y ago
This has been known for a long time. Almost every data center uses multiple ISPs for this reason.
- avi_vallarapu 4y ago
  Right. My point is - "How so many Services across Canada went down as they never designed their Services to overcome the network outage ? "
  - wmf 4y ago
    For things like ATMs I think having two network connections is pretty expensive. Companies also like to use "strategic outsourcing" where they use the same ISP for all their locations which results in the systemic outages we're seeing now.
    - avi_vallarapu 4y ago
      Well, the cost of outage vs the cost to maintain multiple ISP's is the key decision factor here.
  - ilrwbwrkhv 4y ago
    Again as I wrote elsewhere it's because Canada is a deeply mediocre country just getting by with others help and technology. Interac the main electronic money transfer system used Rogers as their primary network. Guess what they used for their backup network? Rogers. There is also very rampant high level corruption leading to anti competitive behaviour, oligarchy, money laundering and I can go on and on. Point is Canada is not a good example of networks and failures. It is a good example of incompetence and mediocrity.
    - solardev 4y ago
      Sounds a lot like America. Anybody can pay for redundant services in multiple data centers and networks. But almost nobody does. And corruption? Heh, here we just call them "donations".
      - ilrwbwrkhv 4y ago
        Yes but Canada has none of the highs of America. If you want to change the world, America is one of the only places.
        solardev 4y ago
        Gosh, I wish I could trade places. If Canada would accept us as refugees, I'd move there in a heartbeat. Change the world? Lol, maybe if you're Bezos or Zuckerberg. The rest of us mostly just stifle under their iron boots. If you don't get shot first. IMO a country that prioritizes its average citizens over its uber-rich isn't mediocre, it's well-managed. America is a hellscape of greed and corruption. Guess the grass is always greener, huh?
        ilrwbwrkhv 4y ago
        I am in the US. I spent a long time in Canada. The population of the US is huge. While the shootings I agree are egregious, in every other way the US is better. The average citizens in Canada have no upward mobility, no healthcare except for absolute emergencies and no bright hope for the future.
        solardev 4y ago
        Why do Canadians have no healthcare? Isn't there a national system? > in every other way the US is better > The average citizens in Canada have no upward mobility > no bright hope for the future. This also applies to everyone I know in the US, aside from those working in tech =/ It is especially bad in rural areas.
  - cloudsec9 4y ago
    While I'm all for ranting about too much control by too few players... How is this different then any "big" player having issues? (Think AWS, Big US ISP, large payment provider such as VISA/MC) I don't think most NEVER thought about network issues ... but rather a calculated risk, as long outages tend to be a rare occurrence. It's honestly cheaper to eat losses on a day like today then to pay for redundancy and availability.
- cloudsec9 4y ago
  Even multiple ISPs might not help. I remember a DC issue where they had two physically separate ISP attachments from different sides of the building. Cable cut takes out ENTIRE DC. Why? While the ISPs were different, at one (farther upstream) point they were both leasing fiber that ran through a single conduit -- so you need understanding of the physical structure of the network, not just "different ISPs".
leastonce 4y ago
Network is not a SPOF. Poorly implemented network can be though. My team manages a global network. We run outage simulations quarterly ( to check for unintended consequences of changes ). Part of that process is to down all circuits and services from a vendor. Could vendors be sharing upstream Concentrations that create SPOF. Sure they could. That’s why we contract a single vendor to create truly diverse services. We see 100x outages and failures from craplications vs network.

wmf 4y ago

This has been known for a long time. Almost every data center uses multiple ISPs for this reason.

avi_vallarapu 4y ago
Right. My point is - "How so many Services across Canada went down as they never designed their Services to overcome the network outage ? "
- wmf 4y ago
  For things like ATMs I think having two network connections is pretty expensive. Companies also like to use "strategic outsourcing" where they use the same ISP for all their locations which results in the systemic outages we're seeing now.
  - avi_vallarapu 4y ago
    Well, the cost of outage vs the cost to maintain multiple ISP's is the key decision factor here.
- ilrwbwrkhv 4y ago
  Again as I wrote elsewhere it's because Canada is a deeply mediocre country just getting by with others help and technology. Interac the main electronic money transfer system used Rogers as their primary network. Guess what they used for their backup network? Rogers. There is also very rampant high level corruption leading to anti competitive behaviour, oligarchy, money laundering and I can go on and on. Point is Canada is not a good example of networks and failures. It is a good example of incompetence and mediocrity.
  - solardev 4y ago
    Sounds a lot like America. Anybody can pay for redundant services in multiple data centers and networks. But almost nobody does. And corruption? Heh, here we just call them "donations".
    - ilrwbwrkhv 4y ago
      Yes but Canada has none of the highs of America. If you want to change the world, America is one of the only places.
      - solardev 4y ago
        Gosh, I wish I could trade places. If Canada would accept us as refugees, I'd move there in a heartbeat. Change the world? Lol, maybe if you're Bezos or Zuckerberg. The rest of us mostly just stifle under their iron boots. If you don't get shot first. IMO a country that prioritizes its average citizens over its uber-rich isn't mediocre, it's well-managed. America is a hellscape of greed and corruption. Guess the grass is always greener, huh?
        ilrwbwrkhv 4y ago
        I am in the US. I spent a long time in Canada. The population of the US is huge. While the shootings I agree are egregious, in every other way the US is better. The average citizens in Canada have no upward mobility, no healthcare except for absolute emergencies and no bright hope for the future.
        solardev 4y ago
        Why do Canadians have no healthcare? Isn't there a national system? > in every other way the US is better > The average citizens in Canada have no upward mobility > no bright hope for the future. This also applies to everyone I know in the US, aside from those working in tech =/ It is especially bad in rural areas.
- cloudsec9 4y ago
  While I'm all for ranting about too much control by too few players... How is this different then any "big" player having issues? (Think AWS, Big US ISP, large payment provider such as VISA/MC) I don't think most NEVER thought about network issues ... but rather a calculated risk, as long outages tend to be a rare occurrence. It's honestly cheaper to eat losses on a day like today then to pay for redundancy and availability.
cloudsec9 4y ago
Even multiple ISPs might not help. I remember a DC issue where they had two physically separate ISP attachments from different sides of the building. Cable cut takes out ENTIRE DC. Why? While the ISPs were different, at one (farther upstream) point they were both leasing fiber that ran through a single conduit -- so you need understanding of the physical structure of the network, not just "different ISPs".

leastonce 4y ago

Network is not a SPOF. Poorly implemented network can be though. My team manages a global network. We run outage simulations quarterly ( to check for unintended consequences of changes ). Part of that process is to down all circuits and services from a vendor. Could vendors be sharing upstream Concentrations that create SPOF. Sure they could. That’s why we contract a single vendor to create truly diverse services. We see 100x outages and failures from craplications vs network.