3 ms·

Side question : what’s the current best practice for ensuring that a server ( node or anything) isn’t currently processing some important information before you

by bsaul 7y ago

Side question : what’s the current best practice for ensuring that a server ( node or anything) isn’t currently processing some important information before you shut it down ? Is it a mix of waiting for request handlers to terminate upon receiving a sigterm then end the current process (and timeouting after a while) ? Does kubernetes handles those kind of things (waiting for a given process to stop before trashing the vm) or is there another layer or tool to do so ?

benburleson 7y ago
Remove the instance from whatever routes requests to it, without killing it -- wait some predefined amount of time -- kill it.
otterley 7y ago
I've built services that can gracefully shut down, and my typical design is as follows: First, maintain an active request counter that you increment when you accept a request and decrement when the request has been fully delivered (including output buffer drained). Second, implement a TERM signal handler. The handler shuts down the listening socket (so no new connections will be accepted), starts a countdown timer, and waits for the active-request counter to reach 0. When either the countdown timer expires or the active-request counter reaches 0, exit.
bjacokes 7y ago
When we start processing a request, we take the JS Promise that represents the result of that operation and put it into an array in our internal "server state" object. When the server gets a shutdown request, it basically awaits Promise.all(this.inFlightRequests) before exiting. Depending on the LB/routing layer on top of your service, you might need to do this in a loop in case work comes in after you do your first await. In other languages you can join a thread pool, wait on a wait group, or use whatever other synchronization tools are provided. Note that there's an ECS_CONTAINER_STOP_TIMEOUT parameter in ECS that sets a hard upper limit on how long containers have to exit before getting SIGKILLed, and it defaults to 30 seconds. If you want to allow requests to drain for longer than that, you'll need to update that parameter. (For services with fast request processing time, the draining process is often a lot simpler. You can just route traffic away from the server, then wait a short period of time before telling the process to exit. It's not quite as precise, but works well for many use cases and requires no bookkeeping.)
rohansingh 7y ago
While graceful shutdown is important, I think the higher priority should be ensuring that you can gracefully recover. Because eventually, something is going to die while a request is in-flight. So your batch processing needs to be able to recover from the "somebody tripped on the power cord" scenario.

by bsaul 7y ago

benburleson 7y ago

Remove the instance from whatever routes requests to it, without killing it -- wait some predefined amount of time -- kill it.

otterley 7y ago

I've built services that can gracefully shut down, and my typical design is as follows: First, maintain an active request counter that you increment when you accept a request and decrement when the request has been fully delivered (including output buffer drained). Second, implement a TERM signal handler. The handler shuts down the listening socket (so no new connections will be accepted), starts a countdown timer, and waits for the active-request counter to reach 0. When either the countdown timer expires or the active-request counter reaches 0, exit.

bjacokes 7y ago

When we start processing a request, we take the JS Promise that represents the result of that operation and put it into an array in our internal "server state" object. When the server gets a shutdown request, it basically awaits Promise.all(this.inFlightRequests) before exiting. Depending on the LB/routing layer on top of your service, you might need to do this in a loop in case work comes in after you do your first await. In other languages you can join a thread pool, wait on a wait group, or use whatever other synchronization tools are provided. Note that there's an ECS_CONTAINER_STOP_TIMEOUT parameter in ECS that sets a hard upper limit on how long containers have to exit before getting SIGKILLed, and it defaults to 30 seconds. If you want to allow requests to drain for longer than that, you'll need to update that parameter. (For services with fast request processing time, the draining process is often a lot simpler. You can just route traffic away from the server, then wait a short period of time before telling the process to exit. It's not quite as precise, but works well for many use cases and requires no bookkeeping.)

rohansingh 7y ago

While graceful shutdown is important, I think the higher priority should be ensuring that you can gracefully recover. Because eventually, something is going to die while a request is in-flight. So your batch processing needs to be able to recover from the "somebody tripped on the power cord" scenario.