7 ms·
Graceful Shutdown in Go: Practical Patterns
- wbl 1y agoIf a distribute system relies on clients gracefully exiting to work the system will eventually break badly.
- smcleod 1y agoWay back when, in physical land - I used STONITH for that! https://smcleod.net/2015/07/delayed-serial-stonith/ https://smcleod.net/2015/07/delayed-serial-stonith/
- XorNot 1y agoThere's valid reasons to want the typical exit not to look like a catastrophic one even if that's a recoverable situation. That my application went down from sig int makes a big difference compared to kill. Blue-Green migrations for example require a graceful exit behavior.
- shoo 1y ago> Blue-Green migrations for example require a graceful exit behavior. it may not always be necessary. e.g. if you are deploying a new version of a stateless backend service, and there is a load balancer forwarding traffic to current version and new version backends, the load balancer could be responsible for cutting over, allowing in flight requests to be processed by the current version backends while only forwarding new requests to the new backends. then the old backends could be ungracefully terminated once the LB says they are not processing any requests.
- ikiris 1y agoThere's a big gap between graceful shutdown to be nice to clients / workflows, and clients relying on it to work.
- Thaxll 1y agoNo one said that.
- Rhapso 1y agoAnd i believe that so much that I don't even consider graceful shutdown in design. Components should be able to safely (and even frequently) hard-crash and so long as a critical percentage of the system is WAI then it shouldn't meaningfully impact the overall system. The only way to make sure a system can handle components hard crashing, is if hard crashing is a normal thing that happens all the time. All glory to the chaos monkey!
- eknkc 1y agoYeah. However, I do not need to pull the plug to shut things down even if the software was designed to tolerate that. In a second thought though, maybe I do. That might be the only way to ensure the assumption is true. Like the Netflix's chaos monkey thing a couple years ago.
- icedchai 1y agoRelying on graceful exit and supporting it are two different things. You want to support it so you can stop serving clients without giving them nasty 5xx errors.
- evil-olive 1y agoanother factor to consider is that if you have a typical Prometheus `/metrics` endpoint that gets scraped every N seconds, there's a period in between the "final" scrape and the actual process exit where any recorded metrics won't get propagated. this may give you a false impression about whether there are any errors occurring during the shutdown sequence. it's also possible, if you're not careful, to lose the last few seconds of logs from when your service is shutting down. for example, if you write to a log file that is watched by a sidecar process such as Promtail or Vector, and on startup the service truncates and starts writing to that same path, you've got a race condition that can cause you to lose logs from the shutdown.
- tmpz22 1y agoIs it me or are observability stacks kind of ridiculous. Logs, metrics, and traces, each with their own databases, sidecars, visualization stacks. Language-specific integration libraries written by whoever felt like it. MASSIVE cloud bills. Then after you go through all that effort most of the data is utterly ignored and rarely are the business insights much better then the trailer park version ssh'ing into a box and greping a log file to find the error output. Like we put so much effort into this ecosystem but I don't think it has paid us back with any significant increase in uptime, performance, or ergonomics.
- nkraft11 1y agoI can say that going from a place that had all of that observability tooling set up to one that was at the "ssh'ing into a box and greping a log" stage, you best believe I missed company A immensely. Even knowing which box to ssh into, which log file to grep, and which magic words to search far was nigh impossible if you weren't the dev that set up the machine and wrote the bug in the first place.
- MortyWaves 1y agoI completely agree with you but I also think, like many aspects of "tech" certain segments of it have been monopolised and turned into profit generators for certain organisations. DevOps, Agile/Scrum, Observability, Kubernetes, are all examples of this. This dilutes the good and helpful stuff with marketing bullshit. Grafana seemingly inventing new time series databases and engines every few months is absolutely painful to try keep up to date with in order to make informed decisions. So much so I've started using rrdtool/smokeping again.
- gchamonlive 1y agoThis is one of the things I think Elixir is really smart in handling. I'm not very experienced in it, but it seems to me that having your processes designed around tiny VM processes that are meant to panic, quit and get respawned eliminates the need to have to intentionally create graceful shutdown routines, because this is already embedded in the application architecture.
- cle 1y agoHow does that eliminate the need for the graceful shutdown the author discusses?
- fredrikholm 1y agoIn the same way that GC eliminates the need for manual memory management. Sometimes it's not enough and you have to 'do it by hand', but generally if you're working in a system that has GC, freeing memory is not something that you think of often. The BEAM is designed for building distributed, fault tolerant systems in the sense that these type of concerns are first class objects, as compared to having them as external libraries (eg. Kafka) or completely outside of the system (eg. Kubernetes). The three points the author lists in the beginning of the article are already built in and their behavior are described rather than implemented, which is what I think OP meant with not having to 'intentionally create graceful shutdown routines'.
- joaohaas 1y agoI really don't see how what you are describing has anything to do with the graceful shutdown strategies/tips mentioned in the post. - Some applications want to instantly terminate upon receiving kill sigs, others want to handle them, OP shows how to handle them - In the case of HTTP servers, you want to stop listening for new requests, but finish handling current ones under a timer. TBF, OPs post actually handles that badly with a time.Sleep when there's a running connection, instead of using a sync.WaitGroup like most applications would want to do - Regardless if the application is GCd or not, you probably want to still manually close connections, so you can handle any possible errors (a lot of connections stuff flushes data on close)
- deathanatos 1y ago> After updating the readiness probe to indicate the pod is no longer ready, wait a few seconds to give the system time to stop sending new requests. > The exact wait time depends on your readiness probe configuration A terminating pod is not ready by definition. The service will also mark the endpoint as terminating (and as not ready). This occurs on the transition into Terminating; you don't have to fail a readiness check to cause it. (I don't know about the ordering of the SIGTERM & the various updates to the objects such as Pod.status or the endpoint slice; there might be a small window after SIGTERM where you could still get a connection, but it isn't the large "until we fail a readiness check" TFA implies.) (And as someone who manages clusters, honestly that infintesimal window probably doesn't matter. Just stop accepting new connections, gracefully close existing ones, and terminate reasonably fast. But I feel like half of the apps I work with fall into either a bucket of "handle SIGTERM & take forever to terminate" or "fail to handle SIGTERM (and take forever to terminate)".
- giancarlostoro 1y agoI had a coworker that would always say, if your program cannot cleanly handle ctrl c and a few other commands to close it, then its written poorly.
- amelius 1y agoCtrl-C is reserved for copy into the clipboard ... Stopping the program instead is highly counter-intuitive and will result in angry users.
- moooo99 1y agoHave you really never cancelled a program in a terminal session?
- tgv 1y agoI think it was a joke. The style, clearly, almost pedantically stating an annoyance as fact, does suggest that.
- kevin_thibedeau 1y agoDefinitely yanking us around.
- moooo99 1y agoProbably was an r/whoosh moment on my part
- danhau 1y agoYour coworker is correct.
- zdc1 1y agoI've been bitten by the surprising amount of time it takes for Kubernetes to update loadbalancer target IPs in some configurations. For me, 90% of the graceful shutdown battle was just ensuring that traffic was actually being drained before pod termination. Adding a global preStop hook with a 15 second sleep did wonders for our HTTP 503 rates. This creates time between when the loadbalancer deregistration gets kicked off, and when SIGTERM is actually passed to the application, which in turn simplifies a lot of the application-side handling.
- LazyMans 1y agoWe just realized this was a problem too
- rdsubhas 1y agoYes. Prestop sleep is the magic SLO solution for high quality rolling deployments. IMHO, there are two things that kubernetes could improve on: 1. Pods should be removed from Endoints _before_ initiating the shutdown sequence. Like the termination grace, there should be an option for termination delay. 2. PDB should allow an option for recreation _before_ eviction.
- eberkund 1y agoI created a small library for handling graceful shutdowns in my projects: https://github.com/eberkund/graceful https://github.com/eberkund/graceful I find that I typically have a few services that I need to start-up and sometimes they have different mechanisms for start-up and shutdown. Sometimes you need to instantiate an object first, sometimes you have a context you want to cancel, other times you have a "Stop" method to call. I designed the library to help my consolidate this all in one place with a unified API.
- mariusor 1y agoHaha, I had the exact same idea, though my API looks a bit less elegant. Maybe it's because it allows the caller to set-up multiple signals to handle and in which way to do it. https://pkg.go.dev/git.sr.ht/~mariusor/wrapper#example-RegisterSignalHandlers https://pkg.go.dev/git.sr.ht/~mariusor/wrapper#example-Regis...
- pseidemann 1y agoI did something similar as well: https://github.com/pseidemann/finish https://github.com/pseidemann/finish
- cientifico 1y agoWe've adopted Google Wire for some projects at JustWatch, and it's been a game changer. It's surprisingly under the radar, but it helped us eliminate messy shutdown logic in Kubernetes. Wire forces clean dependency injection, so now everything shuts down in order instead... well who knows :-D https://go.dev/blog/wire https://go.dev/blog/wire https://github.com/google/wire https://github.com/google/wire
- liampulles 1y agoI tend to use a waitgroup plus context pattern. Any internal service which needs to wind down for shutdown gets a context which it can listen to in a goroutine to start shutting down, and a waitgroup to indicate that it is finished shutting down. Then the main app goroutine can close the context when it wants to shutdown, and block on the waitgroup until everything is closed.
- mariusor 1y agoIf you look at the article, it presents some additional niceties, like having middleware that is aware of the shutdown - though they didn't detail exactly how the WithCancellation() function is doing that. So if you send a SIG-INT/-TERM signal to the server there's a delay to clean up resources, during which the new requests get served a response that doesn't try to access them and fail in unexpected ways, but a configurable "not in service" error.
- fpoling 1y agoI was hoping the article describe how to perform the application restart without dropping a single incoming connections when a new service instance receives the listening socket from the old instance. It is relatively straightforward to implement under systemd. And nginx has been supporting that for over 20 years. Sadly Kuberenets and Docker have no support for that assuming it is done in load balancer or the reverse proxy.
- joaohaas 1y agoYou're probably looking for Cloudflare's tableflip: https://github.com/cloudflare/tableflip https://github.com/cloudflare/tableflip
- gitroom 1y agohonestly i always end up wrestling with logs and shutdowns too, nothing ever feels simple - feels like every setup needs its own pile of band aids
- karel-3d 1y agoone tiny thing I see quite often: people think that if you do `log.Fatal`, it will still run things in `defer`. It won't! package main import ( "fmt" "log" ) func main() { defer fmt.Println("in defer") log.Fatal("fatal") } this just runs "fatal"... because log.Fatal calls os.Exit, and that closes everything immediately. package main import ( "fmt" "log" ) func main() { defer fmt.Println("in defer") panic("fatal") } This shows both `fatal` and `in defer`
- Savageman 1y agoI wish it would talk about liveness too, I've see several times apps that use the same endpoint for liveness/readiness but it feels wrong.