2 ms·
I mean... that's what circuit breakers are for. If a component of a service is optional to its operation, then it wraps calls to that service in a circuit-break
by derefr 7y ago
I mean... that's what circuit breakers are for. If a component of a service is optional to its operation, then it wraps calls to that service in a circuit-breaker and fails requests that ask for that service. And if a component of a service is not optional to the operation of a service, then the failure should cascade to the service's dependent clients, and their dependent clients, and so on, so that there's backpressure all the way back to the origin of the request.
- closeparen 7y agoHealth checks can have several purposes. They're used by the routing control plane to determine inclusion in the load balancer pool. This is already a kind of circuit breaker and is similar to what an application-level circuit breaker would poll. But they're also used by the scheduler to determine whether the instance needs to be restarted. You don't want your thing in a restart loop just because a dependency is down! In fact, if a very widely shared dependency went down, and everyone was checking it in their health check, the scheduler control plane could quickly have a backlog measured in days trying to move all those instances. Our environment now supports distinct answers to those two questions but most service authors don't know about it.