4 ms·

It's not as simple as N+1. Who watches the watchers? With a lot of these "redundant" systems you eventually have some SPOF which "manages" the redundant compo

by jzwinck 10y ago

It's not as simple as N+1. Who watches the watchers? With a lot of these "redundant" systems you eventually have some SPOF which "manages" the redundant components. For example, many years ago I used a network service provided by a quite diligent company. They had installed a large Cisco switch with redundant "supervisors" which are...something important. So back then the way this worked was that the "active" supervisor would handle everything, and the "standby" one would ping the active one and if it decided the active one had died, it would make itself active. This process takes about a minute. You can imagine what might happen if each supervisor starts to believe that the other is down. About ten years ago, Cisco released some new fancier (supposedly better) way to handle this. It is not enabled by default [1, don't bother reading]. [1] http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-6500-series-switches/prod_white_paper09186a0080088874.html http://www.cisco.com/c/en/us/products/collateral/switches/ca...

macintux 10y ago
I've always been (probably overly) paranoid about redundant switches. Seemed like a good way to introduce bizarre failure modes to prevent a fairly rare but easy to diagnose switch death.
- Sanddancer 10y ago
  It's not just complete switch death you have to worry about. Ports can die, cables can go bad, etc. Rare, but at the scale Delta's at, it'll happen frequently enough to plan for it.
nickpsecurity 10y ago
There's ways to deal with stuff like that in commercial sector and academia going back a long time. It is more complex than a simple ping. It's also common to have the protocols analyzed or even formally-verified for avoiding issues like you mentioned. Many systems will also have things like separate power supplies, certain distance apart, octocouplers, whatever to reduce hardware glitches. There's also both noise-immune (like 1802) and clockless (like Amulet) architectures to knock out those problems. Systems with a subset of these have been known to go years to over a decade without downtime. The usual culprit in stuff like you described is management. They either force knowledgeable staff to use less to save money in short-term at long-term expense or don't even hire staff good enough for high-assurance systems. Those do cost money given they're uncommon to rare. Although, one company (Ericsson) did go all out one time making their network nine 9's just to show it could be done. Till management and customers looked at the cost. :) EDIT to add textbook case study on management angle: https://www.youtube.com/watch?v=nsJpTTIEySo https://www.youtube.com/watch?v=nsJpTTIEySo
walrus01 10y ago
This is why core ISP POPs deploy routers in twin pairs. Each chassis has hotswap fans, n+1 power, dual redundant supervisors and hotswap line cards but is entirely possible for an entire chassis to go belly up. A single router chassis is not redundancy at all.

by jzwinck 10y ago

macintux 10y ago

I've always been (probably overly) paranoid about redundant switches. Seemed like a good way to introduce bizarre failure modes to prevent a fairly rare but easy to diagnose switch death.

Sanddancer 10y ago
It's not just complete switch death you have to worry about. Ports can die, cables can go bad, etc. Rare, but at the scale Delta's at, it'll happen frequently enough to plan for it.

nickpsecurity 10y ago

There's ways to deal with stuff like that in commercial sector and academia going back a long time. It is more complex than a simple ping. It's also common to have the protocols analyzed or even formally-verified for avoiding issues like you mentioned. Many systems will also have things like separate power supplies, certain distance apart, octocouplers, whatever to reduce hardware glitches. There's also both noise-immune (like 1802) and clockless (like Amulet) architectures to knock out those problems. Systems with a subset of these have been known to go years to over a decade without downtime. The usual culprit in stuff like you described is management. They either force knowledgeable staff to use less to save money in short-term at long-term expense or don't even hire staff good enough for high-assurance systems. Those do cost money given they're uncommon to rare. Although, one company (Ericsson) did go all out one time making their network nine 9's just to show it could be done. Till management and customers looked at the cost. :) EDIT to add textbook case study on management angle: https://www.youtube.com/watch?v=nsJpTTIEySo https://www.youtube.com/watch?v=nsJpTTIEySo

walrus01 10y ago

This is why core ISP POPs deploy routers in twin pairs. Each chassis has hotswap fans, n+1 power, dual redundant supervisors and hotswap line cards but is entirely possible for an entire chassis to go belly up. A single router chassis is not redundancy at all.