4 ms·
This might be my bias as someone who mostly writes in actor-based concurrent languages, but: is there a reason to test the service’s back-to-life-ness by passin
by derefr 7y ago
This might be my bias as someone who mostly writes in actor-based concurrent languages, but: is there a reason to test the service’s back-to-life-ness by passing a real request through, rather than just having every open circuit trigger the creation of a periodic background “/health”-endpoint (or equivalent) poller actor for that backend within your service? To do the check within the client request lifecycle seems like it would needlessly increase client-request 99th-percentile latency for no real benefit, other than the single request which doesn’t time out, closes the circuit, and so gets served from the service.
ETA: you could even—presuming your system does solely idempotent stuff—take the exact request whose timeout caused the circuit to open, and pass that request (the particular query for SQL; the particular URI + POST/PUT data for REST) to the poller-actor you’re spawning to use as its health-check. Once that request doesn’t time out any more, you know you’re back online.
I’d recommend a real separate /health endpoint for the service you’re trying to contact, though, because 1. most applications do non-idempotent things, and 2. some queries time out because they touch edge-case bad/unscalable code paths, not because the remote-service-as-a-whole is down. If your `SELECT * FROM expensive_generated_report_view` SQL query times out, you really should not be reusing that query as a health-check against your RDBMS. (But nor should you just be doing `SELECT 1`!)
- dtech 7y agoA /health endpoint or similar mock might be available and responding fast, while a real request might fail or time out.
- derefr 7y agoWell, that’s not a “true” /health endpoint, then. A service’s /health endpoint should run through its regular non-trivial code paths and depend for its success on all of the dependent services normal requests to that service depend on. (Probably you’ll need to write it yourself, rather than using one supplied by your application framework.) For example, if you have a CRUD app fronting a database, your CRUD app’s /health endpoint should attempt to make a simple database query. If you have an ETL daemon that pulls from a third-party API and pushes to a message queue, it should probe both the readiness of the API and the message-queue before reporting its own readiness to work. (Of course, it is exactly in the case where the service has its own circuit-breaking logic with back-up paths “around” these dependencies, that it gets to say it’s healthy when its dependencies aren’t.) A test of the /health endpoint is, after all, what determines if a service is considered viable for routing by load-balancers; and, vice-versa, if the service is considered to be “stuck” and in need of a restart by platform auto-scaling logic. As such, your /health endpoints really should be configured in a way where they generate false positives—reporting being unhealthy when they’re really healthy—rather than false negatives. If you’ve got a pool of instances, better to have them paranoid of their own readiness in a way where your system will be constantly draining traffic from + restarting them, than to have them lackadaisical about their readiness in a way where they’re receiving traffic they can’t handle.
- coenhyde 7y agoSometimes its not a good idea for the health of a service to be determined by its connected parts (eg databases). For purely situational awareness this is fine. But if you use the healthcheck to determine if an instance of an application should be taken out of service you risk cascading failures; turning 1 problem into 10. It's usually better for the application to throw an error if it can't connect to the database. That said I do both approaches depending on the situation.
- derefr 7y agoI mean... that's what circuit breakers are for. If a component of a service is optional to its operation, then it wraps calls to that service in a circuit-breaker and fails requests that ask for that service. And if a component of a service is not optional to the operation of a service, then the failure should cascade to the service's dependent clients, and their dependent clients, and so on, so that there's backpressure all the way back to the origin of the request.
- closeparen 7y agoHealth checks can have several purposes. They're used by the routing control plane to determine inclusion in the load balancer pool. This is already a kind of circuit breaker and is similar to what an application-level circuit breaker would poll. But they're also used by the scheduler to determine whether the instance needs to be restarted. You don't want your thing in a restart loop just because a dependency is down! In fact, if a very widely shared dependency went down, and everyone was checking it in their health check, the scheduler control plane could quickly have a backlog measured in days trying to move all those instances. Our environment now supports distinct answers to those two questions but most service authors don't know about it.
- hinkley 7y agoTwo modes of cascading failure here: Request of Death, and cascading failures. If a request kills a particular server you should let the error flow upstream, otherwise it will just bounce from server to server until it's killed all of them. For the latter, someone related a real-world example of this to me the other day. Say you have a bunch of people managing customers. Every employee has 4 customers, and those take up all of their time. You get a new customer. Instead of hiring a new rep, you give someone a 5th customer to manage. They struggle, and eventually they quit. Now, all of your employees have 5 customers. Sooner or later one of those will also quit, and then it's a race to see who can get out the door fastest. The moral of that story is that all the load balancing in the world is for naught if you haven't done your capacity planning properly. And once the system starts to buckle it may be too late to bring new capacity online (since startup usually consumes more resources).
- TheRealPomax 7y agoyou don't mock it, you implement a real one that acts as a remote microservice. Sometimes it's a /health route on the same site, sometimes it's a status.example.com, etc. The only thing it should do is give you a static errors-or-not file that gets regenerated at some reasonable interval by an independently running process, off of all known dependencies that you control. If the health route says everything's up and running, then you can try a normal request, and if that fails, you back off anew (and you log an error entry that can be used to update the health endpoint's data generator, because something, somewhere, is misreporting)
- dtech 7y agoThe only way a health endpoint is not mocking anything is when it does a large variety of full valid requests, having the same complexity and functionality of the service. Oh and all of them should fail or succeed identically. At that point why not use the real request. Or you're overshooting. Say you have a service with 2 endpoints, both depend on the DB and 1 depends on an external API. External API goes down, you don't have a correct choice if you have 1 health endpoint for circuit breakers. If you open the breaker you needlessly disable requests to the healthy endpoint. If you leave it closed your circuit breaker is not working for the broken endpoint. You can of course use a health check in addition to the normal endpoint in your circuit breaker, but then its just a fail-faster for the scenarios that the health endpoint covers.
- derefr 7y ago> You can of course use a health check in addition to the normal endpoint in your circuit breaker, but then its just a fail-faster for the scenarios that the health endpoint covers. Er, yes, that was the thing I was trying to suggest. You want to fail fast (or, more precisely, keep the system in the "circuit open, degraded functionality code-path" state) so your users don't experience mysterious latency spikes when their request becomes the guinea-pig for the circuit-breaker. That'll still happen, but not as often if a health check can tell the service there's not even a chance the service is worth checking via user-request.
- lostcolony 7y agoSo some of the responses to you seem to be missing the point. I agree with you, and with a health check being the wrong thing to circuit break on. Not matter how complex a health endpoint is, the circuit breaker is more so. The health endpoint might make a legit request, checking all downstream services, etc etc, to confirm that everything is working...but the actual traffic is many different requests, with different code flows. As an example, imagine service Bs health check actually makes a request. Great. Only, service B is misconfigured to talk to staging, whereas the upstream, service A, is in prod. The health check may always pass; after all, the staging request's data all gets resolved and handled appropriately. The upstream one, however, leads to constant failure, as that data doesn't exist. Health check passes 100% of the time. Actual upstream calls pass 0% of the time. Don't have your upstream calls circuit break on health checks; have them circuit break on actual responses. (And yes, you should have network isolation between staging and prod so that particular example can't happen. It is intentionally a contrived example)
- derefr 7y agoMy original point was that a health-check is a good preflight check before the circuit-breaker goes "half open" in the article's terminology, letting traffic flow through again as a test to see whether the service has recovered. The circuit breaker should still do that—but it should do that after a health-check tells it the service has recovered. That way, you don't have random timeouts as your service abuses user-requests as a bad way of finding out the trivial information that it can't even connect to the service. Instead, the circuit breaker will first get an upper-bound on healthiness from the /health endpoint—and that'll be enough to tell it when even the upper bound says the service is unhealthy. Then, when the upper bound says "healthy", it'll seek a tighter bound on healthiness by letting through actual requests in the "half-open" state. In this way, depending on how you've written your /health endpoint, you can ensure that far fewer users get their request abused as the scope for the service's health-checking reconaissance-in-force mission; and so decrease your 99th-percentile latency. (Since, of course, a background poller hitting the /health endpoint is outside the request flow, and so doesn't count toward 99th-percentile latency.)