3 ms·
But the automated transfer switch seems to have failed Switch? N+1 baby. Even more on critical systems. If that switch breaking can cause you to cancel 1500 fl
by TheSpiceIsLife 10y ago
But the automated transfer switch seems to have failed
Switch? N+1 baby. Even more on critical systems. If that switch breaking can cause you to cancel 1500 flights there's a clear case to replicate the switchboard. Surely? Also, manual transfer switches are a thing too, just make sure you break-before-make.
- jzwinck 10y agoIt's not as simple as N+1. Who watches the watchers? With a lot of these "redundant" systems you eventually have some SPOF which "manages" the redundant components. For example, many years ago I used a network service provided by a quite diligent company. They had installed a large Cisco switch with redundant "supervisors" which are...something important. So back then the way this worked was that the "active" supervisor would handle everything, and the "standby" one would ping the active one and if it decided the active one had died, it would make itself active. This process takes about a minute. You can imagine what might happen if each supervisor starts to believe that the other is down. About ten years ago, Cisco released some new fancier (supposedly better) way to handle this. It is not enabled by default [1, don't bother reading]. [1] http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-6500-series-switches/prod_white_paper09186a0080088874.html http://www.cisco.com/c/en/us/products/collateral/switches/ca...
- macintux 10y agoI've always been (probably overly) paranoid about redundant switches. Seemed like a good way to introduce bizarre failure modes to prevent a fairly rare but easy to diagnose switch death.
- Sanddancer 10y agoIt's not just complete switch death you have to worry about. Ports can die, cables can go bad, etc. Rare, but at the scale Delta's at, it'll happen frequently enough to plan for it.
- nickpsecurity 10y agoThere's ways to deal with stuff like that in commercial sector and academia going back a long time. It is more complex than a simple ping. It's also common to have the protocols analyzed or even formally-verified for avoiding issues like you mentioned. Many systems will also have things like separate power supplies, certain distance apart, octocouplers, whatever to reduce hardware glitches. There's also both noise-immune (like 1802) and clockless (like Amulet) architectures to knock out those problems. Systems with a subset of these have been known to go years to over a decade without downtime. The usual culprit in stuff like you described is management. They either force knowledgeable staff to use less to save money in short-term at long-term expense or don't even hire staff good enough for high-assurance systems. Those do cost money given they're uncommon to rare. Although, one company (Ericsson) did go all out one time making their network nine 9's just to show it could be done. Till management and customers looked at the cost. :) EDIT to add textbook case study on management angle: https://www.youtube.com/watch?v=nsJpTTIEySo https://www.youtube.com/watch?v=nsJpTTIEySo
- walrus01 10y agoThis is why core ISP POPs deploy routers in twin pairs. Each chassis has hotswap fans, n+1 power, dual redundant supervisors and hotswap line cards but is entirely possible for an entire chassis to go belly up. A single router chassis is not redundancy at all.
- snowwindwaves 10y agoThe Manual transfer switches I have seen are mechanically interlocked or designed so that the sources can not be paralleled. In fact automatic transfer switches that are "open transition" have a mechanical interlock as well. Any situation in which it is up to a human to make sure they break-before-make is a grenade waiting to go off. The only thing I've seen close to that is old power plants where the generators are manually synchronized but usually there is still a sync check so the sources can't be connected more than X degrees out of phase, where X is 3 for large hydro units, 10 for standard diesels, and can be as wide as 20 or 30 for diesels at hospitals where they need to get the thing online ASAP
- csears 10y agoThey're not talking about Cisco/networking switches. Transfer switches are part of the electrical systems in the data center. https://en.wikipedia.org/wiki/Transfer_switch https://en.wikipedia.org/wiki/Transfer_switch