2 ms·

So some of the responses to you seem to be missing the point. I agree with you, and with a health check being the wrong thing to circuit break on. Not matter h

So some of the responses to you seem to be missing the point. I agree with you, and with a health check being the wrong thing to circuit break on. Not matter how complex a health endpoint is, the circuit breaker is more so. The health endpoint might make a legit request, checking all downstream services, etc etc, to confirm that everything is working...but the actual traffic is many different requests, with different code flows. As an example, imagine service Bs health check actually makes a request. Great. Only, service B is misconfigured to talk to staging, whereas the upstream, service A, is in prod. The health check may always pass; after all, the staging request's data all gets resolved and handled appropriately. The upstream one, however, leads to constant failure, as that data doesn't exist. Health check passes 100% of the time. Actual upstream calls pass 0% of the time. Don't have your upstream calls circuit break on health checks; have them circuit break on actual responses. (And yes, you should have network isolation between staging and prod so that particular example can't happen. It is intentionally a contrived example)

derefr 7y ago
My original point was that a health-check is a good preflight check before the circuit-breaker goes "half open" in the article's terminology, letting traffic flow through again as a test to see whether the service has recovered. The circuit breaker should still do that—but it should do that after a health-check tells it the service has recovered. That way, you don't have random timeouts as your service abuses user-requests as a bad way of finding out the trivial information that it can't even connect to the service. Instead, the circuit breaker will first get an upper-bound on healthiness from the /health endpoint—and that'll be enough to tell it when even the upper bound says the service is unhealthy. Then, when the upper bound says "healthy", it'll seek a tighter bound on healthiness by letting through actual requests in the "half-open" state. In this way, depending on how you've written your /health endpoint, you can ensure that far fewer users get their request abused as the scope for the service's health-checking reconaissance-in-force mission; and so decrease your 99th-percentile latency. (Since, of course, a background poller hitting the /health endpoint is outside the request flow, and so doesn't count toward 99th-percentile latency.)

derefr 7y ago

My original point was that a health-check is a good preflight check before the circuit-breaker goes "half open" in the article's terminology, letting traffic flow through again as a test to see whether the service has recovered. The circuit breaker should still do that—but it should do that after a health-check tells it the service has recovered. That way, you don't have random timeouts as your service abuses user-requests as a bad way of finding out the trivial information that it can't even connect to the service. Instead, the circuit breaker will first get an upper-bound on healthiness from the /health endpoint—and that'll be enough to tell it when even the upper bound says the service is unhealthy. Then, when the upper bound says "healthy", it'll seek a tighter bound on healthiness by letting through actual requests in the "half-open" state. In this way, depending on how you've written your /health endpoint, you can ensure that far fewer users get their request abused as the scope for the service's health-checking reconaissance-in-force mission; and so decrease your 99th-percentile latency. (Since, of course, a background poller hitting the /health endpoint is outside the request flow, and so doesn't count toward 99th-percentile latency.)