4 ms·

> Well, that’s not a “true” /health endpoint, then You cannot make such a "true" health endpoint, it's super easy to make a service that contains a paradox abo

by dtech 7y ago

> Well, that’s not a “true” /health endpoint, then You cannot make such a "true" health endpoint, it's super easy to make a service that contains a paradox about what such an endpoint should do. 1 service with 2 endpoints A and B. A relies on an external service and the database, A and B both rely on the database. What to do if the external service is down bringing A down? Either your health endpoint is useless because A is down and it reports the service is fine or you just cascaded the downtime to B while there is no reason to do so. Same situation for 1 endpoint but any branch dependent on the request etc. etc. Of course you can use a health endpoint for determining restarts, load balancers etc, but its not replacement for circuit breakers on your calls.

derefr 7y ago
> 1 service with 2 endpoints A and B. A relies on an external service and the database, A and B both rely on the database. What to do if the external service is down bringing A down? Make one /health/a endpoint and one /health/b endpoint. Client-service A uses /health/a to check if the service is "healthy in terms of A's ability to use it." Client-service B likewise pings /health/b. In a scenario with many different dependent services (a crawler that can hit arbitrary third-party sites, say, or something like Hadoop that can load arbitrary not-necessarily-existent extensions per job) these endpoints should be something clients can create/register themselves, ala SQL stored procedures; or the service can offer a connection-oriented health-state-change streaming endpoint, where the client can hold open a connection and be told when about readiness-state-change events. But to be clear, these are edge-case considerations: in most cases, a service has only critical-path dependencies (which it needs to bootstrap itself, or to "do its job" in the most general SLA sense); and optional dependencies (which it doesn't actually need, and can offer degraded functionality when the service isn't available via circuit-breaking.) It's a rare—and IMHO not-well-factored—service that has dependencies that are on the critical path for some use-cases but not others. Such a service should probably be split into two or more services: a core service that all use-cases depend on; and then one or more services that each just do the things unique to a particular use-case, with all their dependencies being on the critical path to achieve their functionality of serving that specific use-case. Then, those use-case-specific services can be healthy or unhealthy. An example of doing this factoring right: CouchDB. It has a core "database" service, but also a higher-level "view-document querying" service, that can go down if its critical-path dependency (a connection to a Javascript sandbox process) isn't met. Both "services" are contained in one binary and one process, but are presented as two separate collections of endpoints, each with their own health endpoint. An example of doing this factoring wrong: Wordpress. It's got image thumbnailing! Comment spam filtering! CMS publication! All in one! And yet it's either monolithically "healthy" or "unhealthy"; "ready" or "not ready" to run. That is clearly ridiculous, right?
- fatnoah 7y ago
  >Make one /health/a endpoint and one /health/b endpoint. Client-service A uses /health/a to check if the service is "healthy in terms of A's ability to use it." Client-service B likewise pings /health/b. I've done exactly this, and it worked well in my case were the # of related services was pretty small. Each endpoint would return an HTTP status code indicating overall health with additional details stating exactly which checks succeeded or not.
- dtech 7y ago
  > And yet it's either monolithically "healthy" or "unhealthy"; "ready" or "not ready" to run. That is clearly ridiculous, right? That's my point. If your editor cannot create new articles that's a problem that needs to be resolved. I probably wouldn't want all my editors smashing the submit button until it worked, leading to additional overload problems. So I open the "circuit breaker" for submission by sending an email to all the editors to please stop submitting while wordpress is broken. But shutting down the complete website would make the problem much worse because readers wouldn't see anything. Conclusion: It's neither 100% healthy nor 100% unhealthy. Healthiness cannot be captured by a single boolean.
hinkley 7y ago
> you cannot make such a "true" health endpoint Well, you can make such an endpoint, you already have. It's called... Your endpoint. The answer to the top level question is, "because it's easier, more accurate, and more maintainable to call a real endpoint than to try to maintain an endpoint whose sole purpose is to predict whether your other endpoints are actually working." Aka: Just Ask.

by dtech 7y ago

derefr 7y ago

> 1 service with 2 endpoints A and B. A relies on an external service and the database, A and B both rely on the database. What to do if the external service is down bringing A down? Make one /health/a endpoint and one /health/b endpoint. Client-service A uses /health/a to check if the service is "healthy in terms of A's ability to use it." Client-service B likewise pings /health/b. In a scenario with many different dependent services (a crawler that can hit arbitrary third-party sites, say, or something like Hadoop that can load arbitrary not-necessarily-existent extensions per job) these endpoints should be something clients can create/register themselves, ala SQL stored procedures; or the service can offer a connection-oriented health-state-change streaming endpoint, where the client can hold open a connection and be told when about readiness-state-change events. But to be clear, these are edge-case considerations: in most cases, a service has only critical-path dependencies (which it needs to bootstrap itself, or to "do its job" in the most general SLA sense); and optional dependencies (which it doesn't actually need, and can offer degraded functionality when the service isn't available via circuit-breaking.) It's a rare—and IMHO not-well-factored—service that has dependencies that are on the critical path for some use-cases but not others. Such a service should probably be split into two or more services: a core service that all use-cases depend on; and then one or more services that each just do the things unique to a particular use-case, with all their dependencies being on the critical path to achieve their functionality of serving that specific use-case. Then, those use-case-specific services can be healthy or unhealthy. An example of doing this factoring right: CouchDB. It has a core "database" service, but also a higher-level "view-document querying" service, that can go down if its critical-path dependency (a connection to a Javascript sandbox process) isn't met. Both "services" are contained in one binary and one process, but are presented as two separate collections of endpoints, each with their own health endpoint. An example of doing this factoring wrong: Wordpress. It's got image thumbnailing! Comment spam filtering! CMS publication! All in one! And yet it's either monolithically "healthy" or "unhealthy"; "ready" or "not ready" to run. That is clearly ridiculous, right?

fatnoah 7y ago
>Make one /health/a endpoint and one /health/b endpoint. Client-service A uses /health/a to check if the service is "healthy in terms of A's ability to use it." Client-service B likewise pings /health/b. I've done exactly this, and it worked well in my case were the # of related services was pretty small. Each endpoint would return an HTTP status code indicating overall health with additional details stating exactly which checks succeeded or not.
dtech 7y ago
> And yet it's either monolithically "healthy" or "unhealthy"; "ready" or "not ready" to run. That is clearly ridiculous, right? That's my point. If your editor cannot create new articles that's a problem that needs to be resolved. I probably wouldn't want all my editors smashing the submit button until it worked, leading to additional overload problems. So I open the "circuit breaker" for submission by sending an email to all the editors to please stop submitting while wordpress is broken. But shutting down the complete website would make the problem much worse because readers wouldn't see anything. Conclusion: It's neither 100% healthy nor 100% unhealthy. Healthiness cannot be captured by a single boolean.

hinkley 7y ago

> you cannot make such a "true" health endpoint Well, you can make such an endpoint, you already have. It's called... Your endpoint. The answer to the top level question is, "because it's easier, more accurate, and more maintainable to call a real endpoint than to try to maintain an endpoint whose sole purpose is to predict whether your other endpoints are actually working." Aka: Just Ask.