4 ms·
I don't work as an SRE, but isn't that covered by providing engineers physical access to secure facilities in the absolute worst case? The article states: > T
by karlding 7y ago
I don't work as an SRE, but isn't that covered by providing engineers physical access to secure facilities in the absolute worst case?
The article states:
> The defense in depth philosophy means we have robust backup plans for handling failure of such tools, but use of these backup plans (including engineers travelling to secure facilities designed to withstand the most catastrophic failures, and a reduction in priority of less critical network traffic classes to reduce congestion) added to the time spent debugging.
- log_n 7y agoYou don't have to go that far. You could also have automated roll-back of individual servers if they sense something is off, for instance. Another alternative is low bandwidth flag based roll-backs (for instances such as this where the network is congested but not completely lost).