3 ms·
Because you want the failure modes to be graceful and recovery to be automatic. When the foundation of a technology stack has a failure, there are two differen
by saalweachter 2y ago
Because you want the failure modes to be graceful and recovery to be automatic.
When the foundation of a technology stack has a failure, there are two different axis of failure.
1. How well do things keep working without the root service? Does every service that can be provided without it still keep going?
2. How automatically does the system recover when the root service is restored? Do you need to bring down the entire system and restore it in a precise order of dependencies?
It's nice if your system can tolerate the missing service and keep chugging along, but it is essential that your system not deadlock on the root service disappearing and stay deadlocked after the service is restored. At best, that turns a downtime of minutes into a downtime of hours, as you carefully turn down every service and bring them back up in a carefully proscribed order. At worse, you discover that your system that hasn't gone down in three years has acquired circular dependencies among its services, and you need to devise new fixes and work-arounds to allow it to be brought back up at all.