This was famously a problem for Google's distributed lock service, Chubby. They handled it by intentionally having outages to flush out ways it might have start

This was famously a problem for Google's distributed lock service, Chubby. They handled it by intentionally having outages to flush out ways it might have started to bear loads it wasn't designed for: https://sre.google/sre-book/service-level-objectives/#xref_risk-management_global-chubby-planned-outage https://sre.google/sre-book/service-level-objectives/#xref_r...