6 ms·

Gracefully Restarting a Go Program Without Downtime

122 points by aberoham 8y ago

latchkey 8y ago
This is an excellent write up, very easy to follow. Overseer [1] is a pretty good implementation as well. "Commonly, graceful restarts are performed by the active process (dark blue) closing its listeners and passing these matching listening socket files (green) over to a newly started process. This restart causes any foreground process monitoring to incorrectly detect a program crash. overseer attempts to solve this by using a small process to perform this socket file exchange and proxying signals and exit code from the active process." [1] https://github.com/jpillora/overseer https://github.com/jpillora/overseer
kjeetgill 8y ago
Fantastic examples. I appreciate that they allude to the (canonical?) example of how Nginx does zero downtime upgrades using SO_REUSEPORT. I had hard of the trick of sharing the FD with a forked process but it's nice to a working example.
spraak 8y ago
Does anyone have any good articles about the same technique for Node?
crawshaw 8y ago
What a coincidence seeing this topic today. Just yesterday I released some similar code for upgrading Go servers in-place while maintaining sockets: https://github.com/crawshaw/littleboss https://github.com/crawshaw/littleboss
joevandyk 8y ago
Seems simpler to do this at the load balancer level. That way you can use the same strategy for any type of application.
- SteveNuts 8y ago
  What if your Go application IS a load balancer?
  - joevandyk 8y ago
    Then you should use nginx or haproxy ;)
  - ebikelaw 8y ago
    Even in that case you don't need graceful in-place restarts. If you have multiple replicas of the load balancer and ECMP from higher on your stack, all you need is for your balancer process to 1) remove itself as a destination for new flows; 2) stop accepting new connections and complete all existing connections; 3) shut down and restart; 4) add itself back to the set of new flow destinations. If you don't have multiple instances of your load balancer, that's weird.
    - kevin_nisbet 8y ago
      I don't believe this approach would work for a layer 7 load balancer (where the connection is terminated on the host). This is in regards to the original article, which is discussing how to maintain socket state through a process upgrade. When you remove the host from the routing table, the ECMP algorithm will redistribute the traffic to the other hosts still in the pool. For TCP connections, this will generate reset's, and drop any active connections moved between hosts. Also, at times I've not gotten clear answers from router vendors on whether they support a consistent hash in their ECMP implementation, which means that connections through the load balancers not being reloaded may also be rebalanced to other hosts and be affected by the routing table change. Even if we consider going down the stack to a layer 4 load balancer, outside the scope of this article, it may still be desirable to combine approaches and offer hitless reloads (a common case) and not just recover from a node failure (a less common case). The reason is, that you would need state replication of the load balancing tables to support removing the load balancer from the routing tables, and having another load balancer take over it's work while maintaining the connections through the load balancer. This is under the assumption that the service being load balanced requires an uninterrupted connection of course. Disclaimer: I work with the author of the article and contributed to it.
ngrilly 8y ago
Will Tarreau of haproxy has written an insightful post on the same topic: "Truly Seamless Reloads with HAProxy – No More Hacks!" https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks/ https://www.haproxy.com/blog/truly-seamless-reloads-with-hap...
epoxyhockey 8y ago
Another reference project: https://github.com/facebookgo/grace https://github.com/facebookgo/grace
lmb 8y ago
It is possible to pass file descriptors using os/exec, using the ExtraFiles attribute [1]. No need drop down to os.StartProcess. 1: https://golang.org/pkg/os/exec/#Cmd https://golang.org/pkg/os/exec/#Cmd
ebikelaw 8y ago
This kind of thing is foreign to me because I can't imagine writing an application, to run in the cloud (as I always do) where I care if a single instance has some downtime. I always rely on something at a higher level proxying my traffic to the application, and when I need to push my application I just shoot each instance in the face and start another one elsewhere, usually on a different host:port. When I'm using the cheapest cloud capacity I can buy my application is constantly preempted by higher-paying tenants anyway, so I have to write my systems as if any instance will suddenly stop. Once you have that problem you can forget about graceful in-place restarts forever.
- twblalock 8y ago
  > Once you have that problem you can forget about graceful in-place restarts forever. Sure, unless you're the one running the load balancer or the service discovery system that enables other people to just kill their instances without worrying about graceful restarts.
  - ebikelaw 8y ago
    See my other comment on this thread. Even in that case you don't have to care.
    - _wmd 8y ago
      There are tons of applications where rolling upgrade doesn't necessarily work out.. anything that is even remotely latency sensitive or state-heavy, for example a game server or an order matching engine
      - ebikelaw 8y ago
        Every service that Google operates is released with rolling updates and is fully preemptible, including the most highly latency-sensitive services and all of the stateful storage services. Google is an existence proof for the applicability of rolling updates to services of all kinds.
erik_seaberg 8y ago
It's theoretically interesting to hand off a listen socket, but I'd rather take 1/n machines out of service than leave half of my memory unused on every machine. Edit: I also remember hearing some out-there stuff about handing off your IP address and open connection state to another machine so the client doesn't even notice…
z3t4 8y ago
I like to use a simple protocol: 1) Server receives shutdown signal 2) Server tells all clients to disconnect, and also tells the load balancer to not send more traffic 3) Client's connect to another server or waits for the server to restart 4) server exit
- sleepydog 8y ago
  How do you tell the load balancer to stop sending traffic? I've done two ways in previous jobs: - Refuse all new connections. If you can change the program, this can be done by closing the listening socket (I think) , but I just installed an iptables rule. - use the load balancer's API to offline the server(at the time it was F5). I liked the first solution because it was load balancer agnostic, but required retries by the load balancer or client to avoid disruptions. Curious to hear what other people did.
  - kyrra 8y ago
    See the lameduck section of this chapter of the Google SRE book. They do frequent health checks on all running jobs. If a health check fails they stop sending that job traffic. The concept of lameduck is you report yourself as unhealthy but are still available to serve traffic, you wait before all of your clients that were loaded balanced to you before you stop a sending traffic. https://landing.google.com/sre/book/chapters/load-balancing-datacenter.html https://landing.google.com/sre/book/chapters/load-balancing-... Health checks are part of GCPs load balancer. https://cloud.google.com/compute/docs/load-balancing/health-checks https://cloud.google.com/compute/docs/load-balancing/health-...
  - z3t4 8y ago
    I think it's best if the client can handle the load balancing by connecting to a random server, and have some servers built into the source for bootstrapping. The client can know what to do if the connection closes and handle interruptions.
wejick 8y ago
I'm using socketmaster[1] to do the graceful update or just normal service reload. It is basically another process to facilitate socket exchange between 2 process. Just make sure that the service able to shutdown gracefully. However this approach need the binary to be started from socketmaster (as child process). I'm not sure what is the real cons tho, except the fact that it alter SIGHUP behavior - which is fine in this case imo. [1] https://github.com/zimbatm/socketmaster https://github.com/zimbatm/socketmaster
- wejick 8y ago
  https://github.com/paytm/grace https://github.com/paytm/grace The graceful handle on the service side can use this package
berdon 8y ago
I made a blog post a couple years ago about doing this for a mud in Go using the FD passing approach. http://austinhanson.com/slinging-mud-graceful-reboots http://austinhanson.com/slinging-mud-graceful-reboots
gonyea 8y ago
“modern systems software” Lol, no. Modern, professional software development uses a load balancer to manage a canary/red-black deployment strategy. Zero downtime deployments to a live server, outside of telecom/finance, is a sign of an amateurish team.

122 points by aberoham 8y ago

This is an excellent write up, very easy to follow. Overseer [1] is a pretty good implementation as well. "Commonly, graceful restarts are performed by the active process (dark blue) closing its listeners and passing these matching listening socket files (green) over to a newly started process. This restart causes any foreground process monitoring to incorrectly detect a program crash. overseer attempts to solve this by using a small process to perform this socket file exchange and proxying signals and exit code from the active process." [1] https://github.com/jpillora/overseer https://github.com/jpillora/overseer

kjeetgill 8y ago

Fantastic examples. I appreciate that they allude to the (canonical?) example of how Nginx does zero downtime upgrades using SO_REUSEPORT. I had hard of the trick of sharing the FD with a forked process but it's nice to a working example.

spraak 8y ago

Does anyone have any good articles about the same technique for Node?

crawshaw 8y ago

What a coincidence seeing this topic today. Just yesterday I released some similar code for upgrading Go servers in-place while maintaining sockets: https://github.com/crawshaw/littleboss https://github.com/crawshaw/littleboss

joevandyk 8y ago

Seems simpler to do this at the load balancer level. That way you can use the same strategy for any type of application.

SteveNuts 8y ago
What if your Go application IS a load balancer?
- joevandyk 8y ago
  Then you should use nginx or haproxy ;)
- ebikelaw 8y ago
  Even in that case you don't need graceful in-place restarts. If you have multiple replicas of the load balancer and ECMP from higher on your stack, all you need is for your balancer process to 1) remove itself as a destination for new flows; 2) stop accepting new connections and complete all existing connections; 3) shut down and restart; 4) add itself back to the set of new flow destinations. If you don't have multiple instances of your load balancer, that's weird.
  - kevin_nisbet 8y ago
    I don't believe this approach would work for a layer 7 load balancer (where the connection is terminated on the host). This is in regards to the original article, which is discussing how to maintain socket state through a process upgrade. When you remove the host from the routing table, the ECMP algorithm will redistribute the traffic to the other hosts still in the pool. For TCP connections, this will generate reset's, and drop any active connections moved between hosts. Also, at times I've not gotten clear answers from router vendors on whether they support a consistent hash in their ECMP implementation, which means that connections through the load balancers not being reloaded may also be rebalanced to other hosts and be affected by the routing table change. Even if we consider going down the stack to a layer 4 load balancer, outside the scope of this article, it may still be desirable to combine approaches and offer hitless reloads (a common case) and not just recover from a node failure (a less common case). The reason is, that you would need state replication of the load balancing tables to support removing the load balancer from the routing tables, and having another load balancer take over it's work while maintaining the connections through the load balancer. This is under the assumption that the service being load balanced requires an uninterrupted connection of course. Disclaimer: I work with the author of the article and contributed to it.

ngrilly 8y ago

Will Tarreau of haproxy has written an insightful post on the same topic: "Truly Seamless Reloads with HAProxy – No More Hacks!" https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks/ https://www.haproxy.com/blog/truly-seamless-reloads-with-hap...

epoxyhockey 8y ago

Another reference project: https://github.com/facebookgo/grace https://github.com/facebookgo/grace

lmb 8y ago

It is possible to pass file descriptors using os/exec, using the ExtraFiles attribute [1]. No need drop down to os.StartProcess. 1: https://golang.org/pkg/os/exec/#Cmd https://golang.org/pkg/os/exec/#Cmd

ebikelaw 8y ago

This kind of thing is foreign to me because I can't imagine writing an application, to run in the cloud (as I always do) where I care if a single instance has some downtime. I always rely on something at a higher level proxying my traffic to the application, and when I need to push my application I just shoot each instance in the face and start another one elsewhere, usually on a different host:port. When I'm using the cheapest cloud capacity I can buy my application is constantly preempted by higher-paying tenants anyway, so I have to write my systems as if any instance will suddenly stop. Once you have that problem you can forget about graceful in-place restarts forever.

twblalock 8y ago
> Once you have that problem you can forget about graceful in-place restarts forever. Sure, unless you're the one running the load balancer or the service discovery system that enables other people to just kill their instances without worrying about graceful restarts.
- ebikelaw 8y ago
  See my other comment on this thread. Even in that case you don't have to care.
  - _wmd 8y ago
    There are tons of applications where rolling upgrade doesn't necessarily work out.. anything that is even remotely latency sensitive or state-heavy, for example a game server or an order matching engine
    - ebikelaw 8y ago
      Every service that Google operates is released with rolling updates and is fully preemptible, including the most highly latency-sensitive services and all of the stateful storage services. Google is an existence proof for the applicability of rolling updates to services of all kinds.

erik_seaberg 8y ago

It's theoretically interesting to hand off a listen socket, but I'd rather take 1/n machines out of service than leave half of my memory unused on every machine. Edit: I also remember hearing some out-there stuff about handing off your IP address and open connection state to another machine so the client doesn't even notice…

z3t4 8y ago

I like to use a simple protocol: 1) Server receives shutdown signal 2) Server tells all clients to disconnect, and also tells the load balancer to not send more traffic 3) Client's connect to another server or waits for the server to restart 4) server exit

sleepydog 8y ago
How do you tell the load balancer to stop sending traffic? I've done two ways in previous jobs: - Refuse all new connections. If you can change the program, this can be done by closing the listening socket (I think) , but I just installed an iptables rule. - use the load balancer's API to offline the server(at the time it was F5). I liked the first solution because it was load balancer agnostic, but required retries by the load balancer or client to avoid disruptions. Curious to hear what other people did.
- kyrra 8y ago
  See the lameduck section of this chapter of the Google SRE book. They do frequent health checks on all running jobs. If a health check fails they stop sending that job traffic. The concept of lameduck is you report yourself as unhealthy but are still available to serve traffic, you wait before all of your clients that were loaded balanced to you before you stop a sending traffic. https://landing.google.com/sre/book/chapters/load-balancing-datacenter.html https://landing.google.com/sre/book/chapters/load-balancing-... Health checks are part of GCPs load balancer. https://cloud.google.com/compute/docs/load-balancing/health-checks https://cloud.google.com/compute/docs/load-balancing/health-...
- z3t4 8y ago
  I think it's best if the client can handle the load balancing by connecting to a random server, and have some servers built into the source for bootstrapping. The client can know what to do if the connection closes and handle interruptions.

wejick 8y ago

I'm using socketmaster[1] to do the graceful update or just normal service reload. It is basically another process to facilitate socket exchange between 2 process. Just make sure that the service able to shutdown gracefully. However this approach need the binary to be started from socketmaster (as child process). I'm not sure what is the real cons tho, except the fact that it alter SIGHUP behavior - which is fine in this case imo. [1] https://github.com/zimbatm/socketmaster https://github.com/zimbatm/socketmaster

wejick 8y ago
https://github.com/paytm/grace https://github.com/paytm/grace The graceful handle on the service side can use this package

berdon 8y ago

I made a blog post a couple years ago about doing this for a mud in Go using the FD passing approach. http://austinhanson.com/slinging-mud-graceful-reboots http://austinhanson.com/slinging-mud-graceful-reboots

gonyea 8y ago

“modern systems software” Lol, no. Modern, professional software development uses a load balancer to manage a canary/red-black deployment strategy. Zero downtime deployments to a live server, outside of telecom/finance, is a sign of an amateurish team.