3 ms·

If every request took exactly the same amount of cpu time and wall time and same amount of ram and other resources, then random load-balancing is just fine for

If every request took exactly the same amount of cpu time and wall time and same amount of ram and other resources, then random load-balancing is just fine for a sufficiently large number of machines. But there in lies the problem. Every request cost is different. And they have noisy neighbor consequences to other requests on the same machine etc. Hence, any naive load-balancing will lead to hotspots. If every request/response contained a X-Health: header that is understood by the load-balancer to decide how much headroom is available and if every request also had deadlines, then load-balancer can do interesting things with it to avoid hotspots, drive near-uniform utilization, avoid bad tail-latencies and provide near-ideal back pressure to upstream.

ignoramous 3y ago
> If every request took exactly the same amount of cpu time and wall time and same amount of ram and other resources, then random load-balancing is just fine for a sufficiently large number of machines. The article points out random load balancing scales sub-linearly with increase in capacity? If we distribute a set of items randomly across a set of servers (e.g. by hashing, or by randomly selecting a server), the average number of items on each server is num_items / num_servers. It is easy to assume this means each server has close to the same number of items. However, since we are selecting servers at random, they will have different numbers of items, and the imbalance can be important. ... This is a classic balls in bins problem... the summary is that the imbalance depends on the expected number of items per server (that is, num_items / num_servers). This means workload is more balanced with fewer servers, or with more items. This means that dividing a set of items over more servers makes the distribution more unfair, which is a reason we can get worse than linear scaling of a distributed system.*
slashdev 3y ago
The client doesn’t know anything about your request implementation, so sending that via a header doesn’t make sense, unless it’s added by some middleware that does. The load balancer could keep track of how long requests for a specific path tend to take, and what servers are currently handling and use that information to be smarter about routing requests. I wouldn’t be surprised if that already exists in things like haproxy or nginx. There’s issues where people stuff data into the URL, like ids or keywords for SEO. But that’s possible to workaround.

ignoramous 3y ago

> If every request took exactly the same amount of cpu time and wall time and same amount of ram and other resources, then random load-balancing is just fine for a sufficiently large number of machines. The article points out random load balancing scales sub-linearly with increase in capacity? If we distribute a set of items randomly across a set of servers (e.g. by hashing, or by randomly selecting a server), the average number of items on each server is num_items / num_servers. It is easy to assume this means each server has close to the same number of items. However, since we are selecting servers at random, they will have different numbers of items, and the imbalance can be important. ... This is a classic balls in bins problem... the summary is that the imbalance depends on the expected number of items per server (that is, num_items / num_servers). This means workload is more balanced with fewer servers, or with more items. This means that dividing a set of items over more servers makes the distribution more unfair, which is a reason we can get worse than linear scaling of a distributed system.*

slashdev 3y ago

The client doesn’t know anything about your request implementation, so sending that via a header doesn’t make sense, unless it’s added by some middleware that does. The load balancer could keep track of how long requests for a specific path tend to take, and what servers are currently handling and use that information to be smarter about routing requests. I wouldn’t be surprised if that already exists in things like haproxy or nginx. There’s issues where people stuff data into the URL, like ids or keywords for SEO. But that’s possible to workaround.