3 ms·

Why not model GEN-2 GC frequency, then do the following: 1. When a gen-2 is likely to occur (looks like 3 vertical lines on your chart), tell the load balancer

by dlss 15y ago

Why not model GEN-2 GC frequency, then do the following: 1. When a gen-2 is likely to occur (looks like 3 vertical lines on your chart), tell the load balancer to stop forwarding requests. 2. Let any existing connections finish. 3. Run the GC manually. 4. Ask the load balancer to start sending requests again. This seems cleaner than marring what sounded like already complex code with additional cognitive burdens.

kmontrose 15y ago
While this possible using the profiling interfaces, it's not really much simpler IMO. What happens if more than 1 server needs to GC at the same time (or overlapping times)? You definitely don't want to pull everything out of rotation, so you'll need some coordination, and how do you determine how many servers are allowed to be pulled out? We know we can limp along on about 1/3 of our servers, but it's not a stellar end user experience; at some point you're not solving the problem, you're spreading the pain. There are some minor annoyance this would introduce too. We use "server in/out of rotation" as an at a glance "things are getting ill" check, if a web server has been pulled (due to slow responses to requests) it's almost invariably something that needs fixing right now. We'd lose that if pulling servers out of rotation was normal operating procedure. This would also be a royal pain in terms of local and dev tier testing, since you've basically made the load balancer and frequent GCs a pre-requisite for thorough testing. Of course, this remains one of the options when dealing with GC stalls; just think what Marc went with here made more sense.
ww520 15y ago
Actually the load balancer approach can be further simplified. Just have a shorter health check on the server, like 3 seconds. When the server is in long GC pause, its health check request from the load balancer will fail. The load balancer automatically takes it off the pool and no traffic reaches it. After GC completed, the server's health check succeeds. Load balancer adds it back to pool automatically. It's just a simpler solution to have a group of servers with enough capacity accounting for the random GC pauses. Having short health check enables failing fast which is generally better for the users overall - their requests don't stuck in overloaded servers for a long time (GC or otherwise caused).

by dlss 15y ago

kmontrose 15y ago

While this possible using the profiling interfaces, it's not really much simpler IMO. What happens if more than 1 server needs to GC at the same time (or overlapping times)? You definitely don't want to pull everything out of rotation, so you'll need some coordination, and how do you determine how many servers are allowed to be pulled out? We know we can limp along on about 1/3 of our servers, but it's not a stellar end user experience; at some point you're not solving the problem, you're spreading the pain. There are some minor annoyance this would introduce too. We use "server in/out of rotation" as an at a glance "things are getting ill" check, if a web server has been pulled (due to slow responses to requests) it's almost invariably something that needs fixing right now. We'd lose that if pulling servers out of rotation was normal operating procedure. This would also be a royal pain in terms of local and dev tier testing, since you've basically made the load balancer and frequent GCs a pre-requisite for thorough testing. Of course, this remains one of the options when dealing with GC stalls; just think what Marc went with here made more sense.

ww520 15y ago

Actually the load balancer approach can be further simplified. Just have a shorter health check on the server, like 3 seconds. When the server is in long GC pause, its health check request from the load balancer will fail. The load balancer automatically takes it off the pool and no traffic reaches it. After GC completed, the server's health check succeeds. Load balancer adds it back to pool automatically. It's just a simpler solution to have a group of servers with enough capacity accounting for the random GC pauses. Having short health check enables failing fast which is generally better for the users overall - their requests don't stuck in overloaded servers for a long time (GC or otherwise caused).