4 ms·
Now you have another problem: the load balancer needs to know accurately when the backend is ready. This can be prone to race conditions, too, especially for se
by puzzle 8y ago
Now you have another problem: the load balancer needs to know accurately when the backend is ready. This can be prone to race conditions, too, especially for services that have very short response times, unless you manage to keep the LB/RPC protocol in lockstep with the garbage collector's behaviour. E.g. you might have to extend the LB/RPC protocol for responses in-band, to include a bit that says "those were the results for the request and, by the way, don't send me new work until I signal you again", which also assumes a permanent connection between the two ends, etc.
It is doable, but it benefits greatly if you have complete control over the whole stack. It gets harder and harder as more languages are thrown into the mix.
- ASalazarMX 8y agoCome on, we can solve that by adding a message queue manager in between. Whenever in doubt, just throw more layers at the problem.
- adrianratnapala 8y agoYes! Especially when fixing latency problems.
- zokier 8y agoIf the OOBGC pauses are anywhere near predictable, then maybe even a super stupid simple solution would work: just have the load balancer assume that backend is busy a fixed amount of time after each request. Even if you massively underestimate the busy time, it shouldn't hurt the latencies. Overestimating on the other hand leads to under-utilization, so it would make sense to set the busy time value to some small percentile of observed oobgc times.
- twic 8y agoYou absolutely have to extend the protocol - or rather, at least one protocol in the stack - to communicate the status in-band. It would be a nightmare otherwise. With HTTP/2 you could have a separate control channel carrying a text/event-stream payload, with messages indicating when the server was ready. With HTTP/1.x, you could abuse HTTP trailers to indicate readiness - the load balancer could read the response until the end of the body, send that to the client, then wait until the server sends a trailer to say it's ready for more work. Or the load balancer could follow every real request with a request to /pleaseBlockUntilYouAreReadyForMoreWork, and only send another real request once it got a reponse to that. With RPC protocols that aren't HTTP, and don't have a flexible presentation layer where this would fit, then you would probably have to add explicit messages to the protocol, which would be a shame.