3 ms·

I have working knowledge of a couple of large routing systems (Yahoo, Facebook) and built a few smaller ones. In general, state is the enemy of scale. You want

by aristus 14y ago

I have working knowledge of a couple of large routing systems (Yahoo, Facebook) and built a few smaller ones. In general, state is the enemy of scale. You want large systems to be simpler and less stateful, not more stateful and "intelligent". One good notion is to embed server statistics into the response, like in an HTTP header. That way the LB can be aware of queue size, queue wait, utilization, etc. But that only works if the knowledge of server health is global to all load balancers, or if some LBs only talk to some servers. If your requests are more or less the same "size", eg they take roughly the same amount of CPU / wall / etc, all of this gets vastly easier.

spaghetti 14y ago
I've never designed large-scale systems however I'm still fascinated. I wonder if LBs could learn over time which requests will probably take a while and which have large probability of being fast? Then the requests could be uniformly distributed amongst the servers. A system like this could have all sorts of knobs to turn. Requests could be partitioned into two groups: "probably fast" and "probably slow". Or three, four, n groups etc. Then the way in which the LBs distribute the requests could be tweaked. For example a ratio of 1/5 slow/fast requests per server. This does require some feedback from the servers to the LBs. However it doesn't have to be fast. The servers could push some (request, time) pairs to some aggregating system at their leisure. Then the response time prediction algorithms used by the LBs are updated at some point. Probably doesn't have to be immediate.
- aristus 14y ago
  You have good instincts here, but 1) In very large systems there are many many load balancers 2) Server "spikes" (eg swapping) can happen very suddenly, so distributing health info is a hard problem 3) Determining a priori which requests are heavy and light is also hard. Think about "SEO urls": they blow up the cardinality. 4) Even assuming perfect global knowledge, you are proposing to solve a variant of the knapsack problem.
aphyr 14y ago
For single-threaded servers, request latency dominates--it's hard to get useful state information out of a dyno, because as soon as it's returned a response, its state has changed. You can add a dedicated stats endpoint to each server, but that requires cooperation from the customer's application. There are... alternate approaches, however.
- aristus 14y ago
  I'm not talking about polling. Presumably the server's response packets flow back through the webserver, unless they are doing DSR. You can piggyback data on that channel in a way that's transparent to the customer.
- Xorlev 14y ago
  IMO, adding feedback mechanisms (e.g. LB polling app or server agent) are among the best ways to create smart, fast LBs without keeping a lot of expensive state. Calculating busyness is another question entirely but it's also a much easier problem to solve than stateful routers.
deleted 14y ago
[deleted]

by aristus 14y ago

spaghetti 14y ago

I've never designed large-scale systems however I'm still fascinated. I wonder if LBs could learn over time which requests will probably take a while and which have large probability of being fast? Then the requests could be uniformly distributed amongst the servers. A system like this could have all sorts of knobs to turn. Requests could be partitioned into two groups: "probably fast" and "probably slow". Or three, four, n groups etc. Then the way in which the LBs distribute the requests could be tweaked. For example a ratio of 1/5 slow/fast requests per server. This does require some feedback from the servers to the LBs. However it doesn't have to be fast. The servers could push some (request, time) pairs to some aggregating system at their leisure. Then the response time prediction algorithms used by the LBs are updated at some point. Probably doesn't have to be immediate.

aristus 14y ago
You have good instincts here, but 1) In very large systems there are many many load balancers 2) Server "spikes" (eg swapping) can happen very suddenly, so distributing health info is a hard problem 3) Determining a priori which requests are heavy and light is also hard. Think about "SEO urls": they blow up the cardinality. 4) Even assuming perfect global knowledge, you are proposing to solve a variant of the knapsack problem.

aphyr 14y ago

For single-threaded servers, request latency dominates--it's hard to get useful state information out of a dyno, because as soon as it's returned a response, its state has changed. You can add a dedicated stats endpoint to each server, but that requires cooperation from the customer's application. There are... alternate approaches, however.

aristus 14y ago
I'm not talking about polling. Presumably the server's response packets flow back through the webserver, unless they are doing DSR. You can piggyback data on that channel in a way that's transparent to the customer.
Xorlev 14y ago
IMO, adding feedback mechanisms (e.g. LB polling app or server agent) are among the best ways to create smart, fast LBs without keeping a lot of expensive state. Calculating busyness is another question entirely but it's also a much easier problem to solve than stateful routers.

deleted 14y ago

[deleted]