5 ms·

And that's why you always set health checks on servers behind a load balancer that should take bad ones out as soon as possible. You then get another interestin

by vasco 9y ago

And that's why you always set health checks on servers behind a load balancer that should take bad ones out as soon as possible. You then get another interesting problem, which is if the server is only spitting errors when serving requests, it'll then be marked healthy again and forever toggle between healthy and unhealthy. But that you can also solve.

indigodaddy 9y ago
If you are doing an L7 http health check, then you should be able to validate a specific URI's expected successful/healty response.
mabbo 9y ago
The mistake I've seen here is having the "health check" do little or nothing. In many web services, I've seen a /ping or /health api that the load balancer calls to ask "are you healthy?". But people get rushed or lazy or requirements change, and they wind up with that api just doing "return true". Now you've got a host that can't access a database or can't do much of anything- but it can return true! Health check APIs should always do some level of checks on dependencies and complex internal logic (maybe a few specific unit/integration tests?) to ensure things are truly healthy.
- s_kilk 9y ago
  At work we have two endpoints on every service, `/status` and `/health-check`. The former is basically a "return true" endpoint, which can tell you if the service is alive and reachable. The latter will usually do something like "select 1;" from any attached databases and only succeed if everything is OK.
  - randerson 9y ago
    And the former is the one you want your Load Balancers to be checking. With a deep health check, even a brief database outage will cause every web server to be taken out of rotation, and then you're completely down for at least as many health check intervals as it takes for the LB to consider a host healthy again. Same goes for any other shared resource that is likely to affect all web servers if it becomes unavailable.
    - vasco 9y ago
      Presumably your service would not be in a very useful state anyway if the main data store it needs to function is out, so which kind of health-check you use will depend on the failure mode you want to expose to your users.
    - gaius 9y ago
      Databases are 10-100x as reliable as the application tier in my experience
      - qooleot 9y ago
        Although the OP didn't describe as such, but health checks that do a simple query are also testing the connectivity to the db. Someone might have hardcoded a db address in an environment variable, or there are connection pooling issues.
    - azernik 9y ago
      Or you just have smarter load balancers, that realize when all of their servers are acting funky and stop taking servers out of rotation.
      - Terr_ 9y ago
        Or to riff on this idea further: You make the concept of a node's "relative stability" part of the load-balancer logic the same way that "relative idleness" is a factor. Then if all 100/100 nodes get taken down by some shared problem, the system simply degenerates into picking the idle-est of the 100.
    - sammorrowdrums 9y ago
      That's not necessarily true. In kubernetes, for example, you have Liveness and Readiness probes, which can both have a period, timeout, initial delay and importantly a number of failures before you kill the service and spawn a new one. This allows you to have less frequent checks that are more in-depth, and basic ones that are just the `return true` type. I guess you're correct that you can have a widespread db outage that makes many of them fail, but then there should be new ones coming very fast, as you can set the deployment minimum for services taking requests. I think you can get very close to stability even in this circumstance. https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/ https://kubernetes.io/docs/tasks/configure-pod-container/con...
- trjordan 9y ago
  Outlier detection also fixes this, without having to implement health checks that replicate the whole functionality (complexity) of the rest of the service. 10 500s in a row? Go to the timeout chair until you get better.
- nathan_f77 9y ago
  There's a great gem for Rails that you can use for comprehensive health checks: https://github.com/ianheggie/health_check https://github.com/ianheggie/health_check It can check the database connection, redis, cache, email, up-to-date migrations, and S3 credentials.
- aaronmu 9y ago
  The best way I can think of is to aggregate errors over time, categorize them and build a health check around those metrics.
deaps 9y ago
I was going to say something similar. I get that this was a very basic explanation of what load balancing can be but very rarely have I seen it implemented in such a fashion, as described - that is, 'serving toast, when finished with the toast, that toaster gets more toast, etc.' There are so many types of load balancing, and so many different ways to guard, simply with best practices, against what this story explains. I also found the 'quadruple hump' in a graph reference difficult to follow without more of a backstory of what the graph was representing. I also don't understand why it would be so difficult to grep through the logs of that load balancer (or load balancers) and find that common fault on the backend.

by vasco 9y ago

indigodaddy 9y ago

If you are doing an L7 http health check, then you should be able to validate a specific URI's expected successful/healty response.

mabbo 9y ago

The mistake I've seen here is having the "health check" do little or nothing. In many web services, I've seen a /ping or /health api that the load balancer calls to ask "are you healthy?". But people get rushed or lazy or requirements change, and they wind up with that api just doing "return true". Now you've got a host that can't access a database or can't do much of anything- but it can return true! Health check APIs should always do some level of checks on dependencies and complex internal logic (maybe a few specific unit/integration tests?) to ensure things are truly healthy.

s_kilk 9y ago
At work we have two endpoints on every service, `/status` and `/health-check`. The former is basically a "return true" endpoint, which can tell you if the service is alive and reachable. The latter will usually do something like "select 1;" from any attached databases and only succeed if everything is OK.
- randerson 9y ago
  And the former is the one you want your Load Balancers to be checking. With a deep health check, even a brief database outage will cause every web server to be taken out of rotation, and then you're completely down for at least as many health check intervals as it takes for the LB to consider a host healthy again. Same goes for any other shared resource that is likely to affect all web servers if it becomes unavailable.
  - vasco 9y ago
    Presumably your service would not be in a very useful state anyway if the main data store it needs to function is out, so which kind of health-check you use will depend on the failure mode you want to expose to your users.
  - gaius 9y ago
    Databases are 10-100x as reliable as the application tier in my experience
    - qooleot 9y ago
      Although the OP didn't describe as such, but health checks that do a simple query are also testing the connectivity to the db. Someone might have hardcoded a db address in an environment variable, or there are connection pooling issues.
  - azernik 9y ago
    Or you just have smarter load balancers, that realize when all of their servers are acting funky and stop taking servers out of rotation.
    - Terr_ 9y ago
      Or to riff on this idea further: You make the concept of a node's "relative stability" part of the load-balancer logic the same way that "relative idleness" is a factor. Then if all 100/100 nodes get taken down by some shared problem, the system simply degenerates into picking the idle-est of the 100.
  - sammorrowdrums 9y ago
    That's not necessarily true. In kubernetes, for example, you have Liveness and Readiness probes, which can both have a period, timeout, initial delay and importantly a number of failures before you kill the service and spawn a new one. This allows you to have less frequent checks that are more in-depth, and basic ones that are just the `return true` type. I guess you're correct that you can have a widespread db outage that makes many of them fail, but then there should be new ones coming very fast, as you can set the deployment minimum for services taking requests. I think you can get very close to stability even in this circumstance. https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/ https://kubernetes.io/docs/tasks/configure-pod-container/con...
trjordan 9y ago
Outlier detection also fixes this, without having to implement health checks that replicate the whole functionality (complexity) of the rest of the service. 10 500s in a row? Go to the timeout chair until you get better.
nathan_f77 9y ago
There's a great gem for Rails that you can use for comprehensive health checks: https://github.com/ianheggie/health_check https://github.com/ianheggie/health_check It can check the database connection, redis, cache, email, up-to-date migrations, and S3 credentials.
aaronmu 9y ago
The best way I can think of is to aggregate errors over time, categorize them and build a health check around those metrics.

deaps 9y ago

I was going to say something similar. I get that this was a very basic explanation of what load balancing can be but very rarely have I seen it implemented in such a fashion, as described - that is, 'serving toast, when finished with the toast, that toaster gets more toast, etc.' There are so many types of load balancing, and so many different ways to guard, simply with best practices, against what this story explains. I also found the 'quadruple hump' in a graph reference difficult to follow without more of a backstory of what the graph was representing. I also don't understand why it would be so difficult to grep through the logs of that load balancer (or load balancers) and find that common fault on the backend.