3 ms·

> The backend node reports how hot it is to the LB and gets a relative weight. If it can take more connections, it raises its relative weight, as it takes on mo

> The backend node reports how hot it is to the LB and gets a relative weight. If it can take more connections, it raises its relative weight, as it takes on more load, it lowers its relative weight. Some load balancers support readiness checks. When a backend is over-worked it can fail readiness to excuse itself from more work. We implement one variation of this by looking at load metrics (cpu, disk, mem, net) [0]. readiness is binary (ok / !ok) and not weighted (ex: no. of conns). The backend, however, is free to load shed (reject already sent requests) as it sees fit. [0] https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html https://www.brendangregg.com/blog/2017-08-08/linux-load-aver...

sethammons 3y ago
in our case, the idea was not to shed all load (bool on/off), but to lower load slowly and to smooth traffic. Another confounding factor was that there were multiple services running on a single node and any of the services may be the one needing to push back a bit. We were able to have each service control the load onto the given node this way -- and if all nodes reported they were overloaded, they would all end up with the same weight meaning we kept processing as much as possible. If we just pulled them from the lb, you could crash the entire cluster during a period of overload.
xyzzy_plugh 3y ago
Load averages can be very useful, but it's problematic to combine them with something like readiness checks. I must point out the corresponding comment in loadavg.c which is required reading: https://github.com/torvalds/linux/blob/b97d64c722598ffed42ece814a2cb791336c6679/kernel/sched/loadavg.c#L5-L7 https://github.com/torvalds/linux/blob/b97d64c722598ffed42ec... The parent's concept of reporting "hotness" is really powerful. It lets a central authority (the load balancer in this scenario) make an informed decision about where to send traffic: the coolest backend. In the case where all systems are actually quite hot, it probably makes sense in most scenarios to continue sending traffic as best-effort. In the case of readiness checks, nacking a check removes the backend from the pool of eligible new-traffic recipients. At least this is true for systems I've encountered. The solution you propose is problematic as it's possible for all systems to decline traffic resulting in no available backends -- rarely a desirable state -- which the load balancer usually can't do much about. A good system will allow you to decouple the signals (loadavg, connections, latency, CPU, etc.) and the health decision (readiness checks) and the interpretation of that decision (load balancer policy) to provide the best of both these worlds. Now, there are many scenarios where simply excusing yourself works fine -- processing expensive batch jobs, for example -- so this can work nicely, but for typical production traffic scenarios I'd advise against this approach.
- heavenlyblue 3y ago
  > as it's possible for all systems to decline traffic resulting in no available backends I don't see why this is a problem? This is when we should start rejecting requests on the frontend
  - paulmd 3y ago
    and also spinning up additional instances on the backend
  - sethammons 3y ago
    presumably you want your service to be up. If you can smooth the load out to other nodes, your service stays up. If you have a bad strategy, you start losing capacity as nodes get removed. Let's say each node can handle 5k rps and you have 10 nodes. You can handle 50k rps. If you are receiving 40k rps, a good strategy will put each node at 80% capacity. A bad strategy will knock a node out, reducing your total capacity, putting extra pressure on the rest of the system, causing more failures, and more pressure. This is called a thundering herd. At some point, your only option is load shedding. But with a bad LB strategy, you start load shedding much much earlier than you should. This is a bad experience for customers that is avoidable with good LB strategies.
    - ignoramous 3y ago
      Fair. Our solution is to make sure a biased load balancer (like in TFA) isn't sending the workload towards a select few machines while others may be not be as over worked, in as simple way as possible. We run load balancers in a fail open mode. As in, if every backend is excusing itself, then none are excused. But as you point out, load balancing is a hairy beast unto itself.
- ignoramous 3y ago
  > Load averages can be very useful, but it's problematic to combine them with something like readiness checks. We don't use loadavg. The linked article in my parent comment makes the same point as you are (:

sethammons 3y ago

in our case, the idea was not to shed all load (bool on/off), but to lower load slowly and to smooth traffic. Another confounding factor was that there were multiple services running on a single node and any of the services may be the one needing to push back a bit. We were able to have each service control the load onto the given node this way -- and if all nodes reported they were overloaded, they would all end up with the same weight meaning we kept processing as much as possible. If we just pulled them from the lb, you could crash the entire cluster during a period of overload.

xyzzy_plugh 3y ago

Load averages can be very useful, but it's problematic to combine them with something like readiness checks. I must point out the corresponding comment in loadavg.c which is required reading: https://github.com/torvalds/linux/blob/b97d64c722598ffed42ece814a2cb791336c6679/kernel/sched/loadavg.c#L5-L7 https://github.com/torvalds/linux/blob/b97d64c722598ffed42ec... The parent's concept of reporting "hotness" is really powerful. It lets a central authority (the load balancer in this scenario) make an informed decision about where to send traffic: the coolest backend. In the case where all systems are actually quite hot, it probably makes sense in most scenarios to continue sending traffic as best-effort. In the case of readiness checks, nacking a check removes the backend from the pool of eligible new-traffic recipients. At least this is true for systems I've encountered. The solution you propose is problematic as it's possible for all systems to decline traffic resulting in no available backends -- rarely a desirable state -- which the load balancer usually can't do much about. A good system will allow you to decouple the signals (loadavg, connections, latency, CPU, etc.) and the health decision (readiness checks) and the interpretation of that decision (load balancer policy) to provide the best of both these worlds. Now, there are many scenarios where simply excusing yourself works fine -- processing expensive batch jobs, for example -- so this can work nicely, but for typical production traffic scenarios I'd advise against this approach.

heavenlyblue 3y ago
> as it's possible for all systems to decline traffic resulting in no available backends I don't see why this is a problem? This is when we should start rejecting requests on the frontend
- paulmd 3y ago
  and also spinning up additional instances on the backend
- sethammons 3y ago
  presumably you want your service to be up. If you can smooth the load out to other nodes, your service stays up. If you have a bad strategy, you start losing capacity as nodes get removed. Let's say each node can handle 5k rps and you have 10 nodes. You can handle 50k rps. If you are receiving 40k rps, a good strategy will put each node at 80% capacity. A bad strategy will knock a node out, reducing your total capacity, putting extra pressure on the rest of the system, causing more failures, and more pressure. This is called a thundering herd. At some point, your only option is load shedding. But with a bad LB strategy, you start load shedding much much earlier than you should. This is a bad experience for customers that is avoidable with good LB strategies.
  - ignoramous 3y ago
    Fair. Our solution is to make sure a biased load balancer (like in TFA) isn't sending the workload towards a select few machines while others may be not be as over worked, in as simple way as possible. We run load balancers in a fail open mode. As in, if every backend is excusing itself, then none are excused. But as you point out, load balancing is a hairy beast unto itself.
ignoramous 3y ago
> Load averages can be very useful, but it's problematic to combine them with something like readiness checks. We don't use loadavg. The linked article in my parent comment makes the same point as you are (: