4 ms·
I find this an interesting contrast to Go's move towards a "request-oriented collector": https://docs.google.com/document/d/1gCsFxXamW8RRvOe5hECz98Ftk-tcRRJcDF
by twic 8y ago
I find this an interesting contrast to Go's move towards a "request-oriented collector":
https://docs.google.com/document/d/1gCsFxXamW8RRvOe5hECz98Ftk-tcRRJcDFANj2VwCB0/edit https://docs.google.com/document/d/1gCsFxXamW8RRvOe5hECz98Ft...
Perhaps a key difference is this point from the Github document:
"Since the OOBGC runs the GC after the response is finished, it can cause the web worker to take longer in order to be ready to process the next incoming request. This means that clients can suffer from latency due to queuing wait times."
It seems like a little bit more intelligence in the load balancer that sits in front of the application servers would go a long way here. Could it only route requests to servers which have finished garbage collecting and are ready to serve again?
- vidarh 8y agoOne option is a reversal: Make the worker request a request. Slow, unavailable or failed workers will in effect automatically back off.
- jashmatthews 8y agoPhusion Passenger has some intelligent routing to balance requests so you don't end up with one process idle and one process serving two requests using 2 threads.
- vidarh 8y agoLots of systems do, but that's requiring lots of logic in the load balancer. Reversal makes this an emergent property of the system.
- tybit 8y agoI’ve been wondering whether this approach would be viable for a while now. Are there any systems that do this? I know solo’s Gloo reverses the flow using NATS but that’s all I’ve seen use this approach.
- vidarh 8y agoI'm not sure. Mongrel2 uses ZeroMQ to talk to the backends to get it automatic handling of disconnecting backends, but I'm not sure if it's actually waiting for the backends to be ready before sending the next request.
- puzzle 8y agoNow you have another problem: the load balancer needs to know accurately when the backend is ready. This can be prone to race conditions, too, especially for services that have very short response times, unless you manage to keep the LB/RPC protocol in lockstep with the garbage collector's behaviour. E.g. you might have to extend the LB/RPC protocol for responses in-band, to include a bit that says "those were the results for the request and, by the way, don't send me new work until I signal you again", which also assumes a permanent connection between the two ends, etc. It is doable, but it benefits greatly if you have complete control over the whole stack. It gets harder and harder as more languages are thrown into the mix.
- ASalazarMX 8y agoCome on, we can solve that by adding a message queue manager in between. Whenever in doubt, just throw more layers at the problem.
- adrianratnapala 8y agoYes! Especially when fixing latency problems.
- zokier 8y agoIf the OOBGC pauses are anywhere near predictable, then maybe even a super stupid simple solution would work: just have the load balancer assume that backend is busy a fixed amount of time after each request. Even if you massively underestimate the busy time, it shouldn't hurt the latencies. Overestimating on the other hand leads to under-utilization, so it would make sense to set the busy time value to some small percentile of observed oobgc times.
- twic 8y agoYou absolutely have to extend the protocol - or rather, at least one protocol in the stack - to communicate the status in-band. It would be a nightmare otherwise. With HTTP/2 you could have a separate control channel carrying a text/event-stream payload, with messages indicating when the server was ready. With HTTP/1.x, you could abuse HTTP trailers to indicate readiness - the load balancer could read the response until the end of the body, send that to the client, then wait until the server sends a trailer to say it's ready for more work. Or the load balancer could follow every real request with a request to /pleaseBlockUntilYouAreReadyForMoreWork, and only send another real request once it got a reponse to that. With RPC protocols that aren't HTTP, and don't have a flexible presentation layer where this would fit, then you would probably have to add explicit messages to the protocol, which would be a shame.
- jashmatthews 8y agoGo's "request oriented collector" proposal is a generational GC, and Ruby has had a generational GC since 2.1. Generational GC is about minimizing GC overhead by avoiding re-marking the entire heap. OOBC is very different. It's trading an increased overhead from collecting more often than necessary for a lower latency experienced by users.
- twic 8y agoThe request-oriented collector is not just a generational GC by another name. The key point: This write barrier design allows ROC to efficiently reclaim unpublished objects when a goroutine exits simply by resetting the current sweep pointer back to the initial sweep pointer. If you use a disposable goroutine for a task, then when it exits, any memory it allocated but did not publish to other goroutines can be reclaimed without doing any further work. It's very much based on the idea that if you can put off collection to the end of a task, you can do it more efficiently, like OOBGC, but approached from a different angle. I have no idea how well it actually works in practice. I've wanted something similar in Java for years, though!
- jashmatthews 8y agoThis is still generational GC but with a young generation heap per coroutine. I've been working on multi-heap GC for MRI Ruby. Compared to traditional concurrent generational GC, skipping the marking when a coroutine terminates is not really an advantage because there will be no live objects to mark and marking would immediately terminate anyway. After that, you just reset the nursery pointer and you're done.
- smials 8y agoAnd contrast again with Erlang's approach, where each process (actor) keeps its own heap, which can just be thrown away once the request handling process is done. Simple and clean, and requires no proofs, barriers or other juju... :)
- jashmatthews 8y agoIt does have a lot of overhead in copying, though. Erlang has lower throughput than Ruby in a lot of cases. It's somewhat mitigated by the reference counted global heap for passing large messages by reference but multi-threaded reference counting is extremely hard to do fast and AFAIK Erlang doesn't really try and do anything too complex.
- derefr 8y agoErlang doesn’t usually need to do anything fancy with shared binary refcounts across processes, though, since in most workloads you’ve got a small number of processes sharing the binary (e.g. one transport + one request handler in HTTP), and the refcount in the shared-binaries table only needs to be synchronously updated when a process gets its first/loses its last reference. Shared-binary refcounts are process-grained. That is: each Erlang process will allocate a single local handle to hold a reference to a given shared heap binary, and then all the slices of that shared binary in that process are pointers to that single process-arena-local handle. So you only actually have to update the refcount cross-thread when the entire binary goes out of scope, which will only happen once per request (either when all process-local references to it are found to be dead during a GC sweep on a long-lived process; or, more often, when the process’s arena is deallocated at process exit.)
- TheSwordsman 8y agoI thought this Request-Oriented Collector idea was abandoned? I seem to remember it being talked about in the Go community, where this hypothesis was tested and the results were sub-par.
- jonhohle 8y agoA Least Connections strategy effectively does this without needing to know why slow servers are slow (be it GC, lemon host, networking issues, etc) or needing the server to communicate anything back to the load balancer. Hosts which are more performant will naturally get more traffic.
- compsciphd 8y agoI was thinking of this, if the backend handler would say "take me out of rotation for Xms as I'm about to do a garbage collection" and the load balancer would see this and act on it. for http I'd imagine it can be a header. For https, I can imagine it might be harder.