4 ms·

How does one handle zero downtime deployments with single-file golang binaries? I remember I tried this setup some time ago and I couldn't successfully manage c

by danwee 4y ago

How does one handle zero downtime deployments with single-file golang binaries? I remember I tried this setup some time ago and I couldn't successfully manage cleanly to accomplish no downtime when deploying a new version of my service. The reason was mainly port reuse. I couldn't have the old and the new version of my service running on the same port... so I started to hack together something and it became dirty pretty quickly. I'm talking about deployment of new version of service on the same machine/server as the old version was running.

iamjackg 4y ago
If you really don't want to use different ports you can handle it with Docker. Since each container has its own IP, they can all expose the same port. Otherwise, for non-containerized deployments you'll have to resort to two different ports. In either case, you will need a reverse proxy like Traefik/Nginx in front to smartly "balance" incoming requests to the two instances of the service.
deleted 4y ago
[deleted]
hnarn 4y ago
This doesn’t sound Go-specific, if you use something like haproxy targeting multiple nodes you can take them down one by one to perform a rolling upgrade.
dividuum 4y ago
Can't help with how to implement this, but just to be sure: You should be able to use the same port in multiple instances if you bind those with SO_REUSEPORT. A quick search points to https://github.com/libp2p/go-reuseport https://github.com/libp2p/go-reuseport for an implementation. Now you just need a mechanism to drain the old process.
- joncfoo 4y ago
  Rough psuedocode to do this with the built-in http.Server where startServer(..) would use the reuseport library to create the listener so multiple servers can listen within the same process: func reloadConfig(config) { if newServer, err := startServer(config); err != nil { // gracefully shutdown previous server // no new connections will go to old server oldServer.Shutdown(...) oldServer = newServer } }
Edd314159 4y ago
I guess this is a problem inherent not just in a single-file go app, but in any deployment where the whole stack is contained within a single process. The post says the process starts up quick enough that the process being temporarily unavailable isn't noticeable - but what if the process _doesn't come back_? It's also impossible to do blue/green deployments this way. It's clearly not a solution suitable to large-scale deployments. The simplicity has its trade-offs.
- InitialBP 4y ago
  If you want to do deployments with single-file apps or other "whole stack in a single process" type of deployments there are other options to do it with zero-downtime. One good option would be to spin up a second server/instance/container, run binary on new system, ensure it's good, once comfortable then swap DNS entry to the new system.
klodolph 4y ago
Some of this is solved by using e.g. systemd, depending on your needs. > I couldn't have the old and the new version of my service running on the same port... You can, actually! You just can’t open the port twice by default. So one or both of the processes needs to inherit the port from a parent process, get passed the port over a socket (Unix sockets can transmit file descriptors), or use SO_REUSEADDR. There are some libraries that abstract this, and some of this is provided by tools like systemd. Some of this is probably going to have to be done in your application—like, once your new version starts, the old version should stop accepting new connections and finish the requests it has already started.
- mattbillenstein 4y ago
  So can I just start another process with SO_REUSEADDR and gracefully shutdown the old process? The master/worker thing that nginx / gunicorn et al do is pretty neat, but relies on signals; so seems pretty messy and error prone to write yourself.
  - klodolph 4y ago
    You may have to start both processes with SO_REUSEADDR, I don’t remember the exact semantics. People have a healthy skepticism of signals from the C days, but if we’re talking about Go, you’d just call signal.Notify. Any way of signaling your app to shut down works, though. https://pkg.go.dev/os/signal@go1.20.2#Notify https://pkg.go.dev/os/signal@go1.20.2#Notify
- joncfoo 4y ago
  > Some of this is probably going to have to be done in your application... FWICT tableflip does exactly this: https://github.com/cloudflare/tableflip https://github.com/cloudflare/tableflip
marcosdumay 4y ago
You can always share ports. But the one way to do no downtime deployments is to have more than one server.
bojanz 4y ago
Socket activation via systemd[0] is an option, assuming you are fine with certain requests taking a longer time to complete (if they arrive while the service is being restarted). Otherwise using a proxy in front of your app is your best bet (which has other benefits too, as you can offload TLS and request logging/instrumentation). - https://github.com/bojanz/httpx#systemd-setup https://github.com/bojanz/httpx#systemd-setup
stasmo 4y ago
This is where the simplicity of single-file golang deployments falls short. Just make sure you’re not slowly recreating bad, homebrew versions of all of the nice things that Kubernetes does in an attempt to turn a simple deployment into a production ready deployment.
adql 4y ago
Same way you do with any other app not specifically designed for it; you start 2 copies of it and put loadbalancer in front of it. I did that via some systemd voodoo But TECHNICALLY to do that in one without external proxy you'd need to figure out how to set SO_REUSEPORT for the web socket handler, then start the second one before the first. Haven't actually tried it but someone apparently did: https://iximiuz.com/en/posts/go-net-http-setsockopt-example/ https://iximiuz.com/en/posts/go-net-http-setsockopt-example/ You'd still have any ongoing connections cut unless you unbind socket and then finish any existing connection, which would be pretty hard with default http server. I just put HAProxy instance on my VPS that does all of that, including only allowing traffic once app says "yes I am ok" in healthcheck. Then the app can have "shutting down" phase, where it reports "I am down" on healthcheck but still finished any active connections to the client.
SkyPuncher 4y ago
I've always run services behind a proxy. Spin up a new server with the code (works for any type of deployment). Validate it's up. Switch the proxy from the old to new server.
tinglymintyfrsh 4y ago
You don't. You can sort of emulate it with services that stateless, load-balanced, and L7 proxied. If you want stateful zero downtime deployments, use Elixir or Erlang that has the ability to live migrate data from one version of code to the next.

by danwee 4y ago

iamjackg 4y ago

If you really don't want to use different ports you can handle it with Docker. Since each container has its own IP, they can all expose the same port. Otherwise, for non-containerized deployments you'll have to resort to two different ports. In either case, you will need a reverse proxy like Traefik/Nginx in front to smartly "balance" incoming requests to the two instances of the service.

deleted 4y ago

[deleted]

hnarn 4y ago

This doesn’t sound Go-specific, if you use something like haproxy targeting multiple nodes you can take them down one by one to perform a rolling upgrade.

dividuum 4y ago

Can't help with how to implement this, but just to be sure: You should be able to use the same port in multiple instances if you bind those with SO_REUSEPORT. A quick search points to https://github.com/libp2p/go-reuseport https://github.com/libp2p/go-reuseport for an implementation. Now you just need a mechanism to drain the old process.

joncfoo 4y ago
Rough psuedocode to do this with the built-in http.Server where startServer(..) would use the reuseport library to create the listener so multiple servers can listen within the same process: func reloadConfig(config) { if newServer, err := startServer(config); err != nil { // gracefully shutdown previous server // no new connections will go to old server oldServer.Shutdown(...) oldServer = newServer } }

Edd314159 4y ago

I guess this is a problem inherent not just in a single-file go app, but in any deployment where the whole stack is contained within a single process. The post says the process starts up quick enough that the process being temporarily unavailable isn't noticeable - but what if the process _doesn't come back_? It's also impossible to do blue/green deployments this way. It's clearly not a solution suitable to large-scale deployments. The simplicity has its trade-offs.

InitialBP 4y ago
If you want to do deployments with single-file apps or other "whole stack in a single process" type of deployments there are other options to do it with zero-downtime. One good option would be to spin up a second server/instance/container, run binary on new system, ensure it's good, once comfortable then swap DNS entry to the new system.

klodolph 4y ago

Some of this is solved by using e.g. systemd, depending on your needs. > I couldn't have the old and the new version of my service running on the same port... You can, actually! You just can’t open the port twice by default. So one or both of the processes needs to inherit the port from a parent process, get passed the port over a socket (Unix sockets can transmit file descriptors), or use SO_REUSEADDR. There are some libraries that abstract this, and some of this is provided by tools like systemd. Some of this is probably going to have to be done in your application—like, once your new version starts, the old version should stop accepting new connections and finish the requests it has already started.

mattbillenstein 4y ago
So can I just start another process with SO_REUSEADDR and gracefully shutdown the old process? The master/worker thing that nginx / gunicorn et al do is pretty neat, but relies on signals; so seems pretty messy and error prone to write yourself.
- klodolph 4y ago
  You may have to start both processes with SO_REUSEADDR, I don’t remember the exact semantics. People have a healthy skepticism of signals from the C days, but if we’re talking about Go, you’d just call signal.Notify. Any way of signaling your app to shut down works, though. https://pkg.go.dev/os/signal@go1.20.2#Notify https://pkg.go.dev/os/signal@go1.20.2#Notify
joncfoo 4y ago
> Some of this is probably going to have to be done in your application... FWICT tableflip does exactly this: https://github.com/cloudflare/tableflip https://github.com/cloudflare/tableflip

marcosdumay 4y ago

You can always share ports. But the one way to do no downtime deployments is to have more than one server.

bojanz 4y ago

Socket activation via systemd[0] is an option, assuming you are fine with certain requests taking a longer time to complete (if they arrive while the service is being restarted). Otherwise using a proxy in front of your app is your best bet (which has other benefits too, as you can offload TLS and request logging/instrumentation). - https://github.com/bojanz/httpx#systemd-setup https://github.com/bojanz/httpx#systemd-setup

stasmo 4y ago

This is where the simplicity of single-file golang deployments falls short. Just make sure you’re not slowly recreating bad, homebrew versions of all of the nice things that Kubernetes does in an attempt to turn a simple deployment into a production ready deployment.

adql 4y ago

Same way you do with any other app not specifically designed for it; you start 2 copies of it and put loadbalancer in front of it. I did that via some systemd voodoo But TECHNICALLY to do that in one without external proxy you'd need to figure out how to set SO_REUSEPORT for the web socket handler, then start the second one before the first. Haven't actually tried it but someone apparently did: https://iximiuz.com/en/posts/go-net-http-setsockopt-example/ https://iximiuz.com/en/posts/go-net-http-setsockopt-example/ You'd still have any ongoing connections cut unless you unbind socket and then finish any existing connection, which would be pretty hard with default http server. I just put HAProxy instance on my VPS that does all of that, including only allowing traffic once app says "yes I am ok" in healthcheck. Then the app can have "shutting down" phase, where it reports "I am down" on healthcheck but still finished any active connections to the client.

SkyPuncher 4y ago

I've always run services behind a proxy. Spin up a new server with the code (works for any type of deployment). Validate it's up. Switch the proxy from the old to new server.

tinglymintyfrsh 4y ago

You don't. You can sort of emulate it with services that stateless, load-balanced, and L7 proxied. If you want stateful zero downtime deployments, use Elixir or Erlang that has the ability to live migrate data from one version of code to the next.