3 ms·
> Engineers should take a more proactive role in identifying and pushing whatever changes are required to prevent mass outages due to single points of failure.
by michaelt 3y ago
> Engineers should take a more proactive role in identifying and pushing whatever changes are required to prevent mass outages due to single points of failure.
In my experience even if you've eliminated single points of failure you can still get failures.
Sure, my server's got dual NICs and dual power supplies. But if the guy sent to replace server 12 in rack 345 accidentally gets server 12 in rack 346 I'm going to lose both NICs and both power supplies at once.
Sure, the network is fully redundant. But we naturally need to keep the settings on the two sets of kit in sync. We've had outages due to them getting out of sync in the past, not going to make that mistake again. So of course if there's a change to the firewall rules it automatically gets rolled out to both of the firewalls.
And so on.
- brodouevencode 3y agoYes, this is true. There's no such thing as 100% redundancy or uptime in the real world without significant cost. The likelihood of that happening goes down exponentially with each successive layer of redundancy or fallback in my experience.