22 ms·

For sufficiently high volume services where loads can be uneven, we had a lot of success with thermostat adaptive balancing. The backend node reports how hot it

For sufficiently high volume services where loads can be uneven, we had a lot of success with thermostat adaptive balancing. The backend node reports how hot it is to the LB and gets a relative weight. If it can take more connections, it raises its relative weight, as it takes on more load, it lowers its relative weight. This worked well on a fleet of a couple thousand servers handling thousands of requests per second each. I can't recall why, but lower volume and few servers caused uneven load, and we went back to least connections for those clusters.

jameshart 3y ago
> lower volume and few servers caused uneven load, and we went back to least connections for those clusters. The more traffic you’re handling, the more the central limit theorem applies as you are summing the behavior of lots of random events drawn from various random distributions, and the more the system behavior regresses to the mean. I think of it as analogous to how when an engine is idling it tends to pop and judder, but as you rev it up the smoother it sounds until at sustained high revs it can make almost a pure note. All the variations are smoothed out and the system runs much more evenly and predictably, but when it drops to idle speed the subtle variation in the fuel air mix and the pressure in the exhaust and inlet every stroke makes it chug and sputter chaotically.
CyberDildonics 3y ago
Why not just look at how much the CPU is being used?
twic 3y ago
To be clear: by "hot", you literally mean physical temperature? That's rather ingenious, i like it! I wonder i you could also measure power consumption.
ignoramous 3y ago
> The backend node reports how hot it is to the LB and gets a relative weight. If it can take more connections, it raises its relative weight, as it takes on more load, it lowers its relative weight. Some load balancers support readiness checks. When a backend is over-worked it can fail readiness to excuse itself from more work. We implement one variation of this by looking at load metrics (cpu, disk, mem, net) [0]. readiness is binary (ok / !ok) and not weighted (ex: no. of conns). The backend, however, is free to load shed (reject already sent requests) as it sees fit. [0] https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html https://www.brendangregg.com/blog/2017-08-08/linux-load-aver...
- sethammons 3y ago
  in our case, the idea was not to shed all load (bool on/off), but to lower load slowly and to smooth traffic. Another confounding factor was that there were multiple services running on a single node and any of the services may be the one needing to push back a bit. We were able to have each service control the load onto the given node this way -- and if all nodes reported they were overloaded, they would all end up with the same weight meaning we kept processing as much as possible. If we just pulled them from the lb, you could crash the entire cluster during a period of overload.
- xyzzy_plugh 3y ago
  Load averages can be very useful, but it's problematic to combine them with something like readiness checks. I must point out the corresponding comment in loadavg.c which is required reading: https://github.com/torvalds/linux/blob/b97d64c722598ffed42ece814a2cb791336c6679/kernel/sched/loadavg.c#L5-L7 https://github.com/torvalds/linux/blob/b97d64c722598ffed42ec... The parent's concept of reporting "hotness" is really powerful. It lets a central authority (the load balancer in this scenario) make an informed decision about where to send traffic: the coolest backend. In the case where all systems are actually quite hot, it probably makes sense in most scenarios to continue sending traffic as best-effort. In the case of readiness checks, nacking a check removes the backend from the pool of eligible new-traffic recipients. At least this is true for systems I've encountered. The solution you propose is problematic as it's possible for all systems to decline traffic resulting in no available backends -- rarely a desirable state -- which the load balancer usually can't do much about. A good system will allow you to decouple the signals (loadavg, connections, latency, CPU, etc.) and the health decision (readiness checks) and the interpretation of that decision (load balancer policy) to provide the best of both these worlds. Now, there are many scenarios where simply excusing yourself works fine -- processing expensive batch jobs, for example -- so this can work nicely, but for typical production traffic scenarios I'd advise against this approach.
  - heavenlyblue 3y ago
    > as it's possible for all systems to decline traffic resulting in no available backends I don't see why this is a problem? This is when we should start rejecting requests on the frontend

jameshart 3y ago

> lower volume and few servers caused uneven load, and we went back to least connections for those clusters. The more traffic you’re handling, the more the central limit theorem applies as you are summing the behavior of lots of random events drawn from various random distributions, and the more the system behavior regresses to the mean. I think of it as analogous to how when an engine is idling it tends to pop and judder, but as you rev it up the smoother it sounds until at sustained high revs it can make almost a pure note. All the variations are smoothed out and the system runs much more evenly and predictably, but when it drops to idle speed the subtle variation in the fuel air mix and the pressure in the exhaust and inlet every stroke makes it chug and sputter chaotically.

CyberDildonics 3y ago

Why not just look at how much the CPU is being used?

twic 3y ago

To be clear: by "hot", you literally mean physical temperature? That's rather ingenious, i like it! I wonder i you could also measure power consumption.

ignoramous 3y ago

> The backend node reports how hot it is to the LB and gets a relative weight. If it can take more connections, it raises its relative weight, as it takes on more load, it lowers its relative weight. Some load balancers support readiness checks. When a backend is over-worked it can fail readiness to excuse itself from more work. We implement one variation of this by looking at load metrics (cpu, disk, mem, net) [0]. readiness is binary (ok / !ok) and not weighted (ex: no. of conns). The backend, however, is free to load shed (reject already sent requests) as it sees fit. [0] https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html https://www.brendangregg.com/blog/2017-08-08/linux-load-aver...

sethammons 3y ago
in our case, the idea was not to shed all load (bool on/off), but to lower load slowly and to smooth traffic. Another confounding factor was that there were multiple services running on a single node and any of the services may be the one needing to push back a bit. We were able to have each service control the load onto the given node this way -- and if all nodes reported they were overloaded, they would all end up with the same weight meaning we kept processing as much as possible. If we just pulled them from the lb, you could crash the entire cluster during a period of overload.
xyzzy_plugh 3y ago
Load averages can be very useful, but it's problematic to combine them with something like readiness checks. I must point out the corresponding comment in loadavg.c which is required reading: https://github.com/torvalds/linux/blob/b97d64c722598ffed42ece814a2cb791336c6679/kernel/sched/loadavg.c#L5-L7 https://github.com/torvalds/linux/blob/b97d64c722598ffed42ec... The parent's concept of reporting "hotness" is really powerful. It lets a central authority (the load balancer in this scenario) make an informed decision about where to send traffic: the coolest backend. In the case where all systems are actually quite hot, it probably makes sense in most scenarios to continue sending traffic as best-effort. In the case of readiness checks, nacking a check removes the backend from the pool of eligible new-traffic recipients. At least this is true for systems I've encountered. The solution you propose is problematic as it's possible for all systems to decline traffic resulting in no available backends -- rarely a desirable state -- which the load balancer usually can't do much about. A good system will allow you to decouple the signals (loadavg, connections, latency, CPU, etc.) and the health decision (readiness checks) and the interpretation of that decision (load balancer policy) to provide the best of both these worlds. Now, there are many scenarios where simply excusing yourself works fine -- processing expensive batch jobs, for example -- so this can work nicely, but for typical production traffic scenarios I'd advise against this approach.
- heavenlyblue 3y ago
  > as it's possible for all systems to decline traffic resulting in no available backends I don't see why this is a problem? This is when we should start rejecting requests on the frontend