7 ms·

"NGINX’s binary upgrade process achieves the holy grail of high-availability; you can upgrade the software on the fly, without any dropped connections, downtime

"NGINX’s binary upgrade process achieves the holy grail of high-availability; you can upgrade the software on the fly, without any dropped connections, downtime or interruption in service." Is this really true? I remember seeing an article[1] recently on using an iptables hack to prevent dropping connections when reloading haproxy. Does nginx actually provide zero-downtime configuration reloads? [1] https://medium.com/@Drew_Stokes/actual-zero-downtime-with-haproxy-18318578fde6 https://medium.com/@Drew_Stokes/actual-zero-downtime-with-ha...

MCRed 11y ago
If it's the holy grail, then erlang has had it for a couple decades... and whether it actually works in nginx I don't know, but it does work in erlang. The way it's handled is that the process where the previous socket was connected remains in place until it terminates but all new sockets connect to the new code.
- nodesocket 11y ago
  "The way it's handled is that the process where the previous socket was connected remains in place until it terminates but all new sockets connect to the new code." That would imply that two separate processes are bound to the same port right? I thought that was not possible.
  - SEJeff 11y ago
    Things have changed :) https://lwn.net/Articles/542629/ https://lwn.net/Articles/542629/
    - bkirkbri 11y ago
      If only the JVM could take advantage of this.
      - SEJeff 11y ago
        Just about bet you could do that via some grotesque JNI work, but I wouldn't wish JNI on really anyone.
  - ericmoritz 11y ago
    No, usually there's an accept process spawning working processes and passing the connection off to it. Erlang can have two versions of a module present in memory at a time. At the moment of a code upgrade, the processes working with existing connections continue to use the old code while newly accepted connections are passed to the version of the module that was loaded in between connection accepts.
  - ericmoritz 11y ago
    Keep in mind that Erlang processes != System processes. Erlang processes are preemptively scheduled microthreads that are scheduled by the Erlang VM.
  - rb12345 11y ago
    Presumably the original server process uses exec() to start the new process, which inherits the open socket handles, and the descriptor is passed as an argument to the new process. (e.g. http://stackoverflow.com/questions/14351147/perl-passing-an-open-socket-across-fork-exec http://stackoverflow.com/questions/14351147/perl-passing-an-...)
  - noselasd 11y ago
    While there are certain socket options to share a port - two processes can share a port easily in another elegant way; just call fork(). One way of doing such upgrades is then for the existing code to fork() then exec() the new version. The existing old code closes the original server socket, which leaves accepting new requests to the new code. When all in-flight requests served by the old code is finished, that process can exit.
- Blackthorn 11y ago
  This is known to me as a "lame duck" mode, and I thought it was a standard thing to do!
Denzel 11y ago
Yes, nginx allows you to upgrade the binary and reload the configuration without any downtime: http://wiki.nginx.org/CommandLine#Upgrading_To_a_New_Binary_On_The_Fly http://wiki.nginx.org/CommandLine#Upgrading_To_a_New_Binary_...
- yeukhon 11y ago
  Does this work when you update with package manager?
  - LukeShu 11y ago
    Yes.
ploxiln 11y ago
I didn't want to be a negative nancy in the comments for that haproxy article... but that is a ridiculous ugly hack. It's really a lot easier to achieve real and robust zero-downtime upgrades for a simple unix process. Remember, fork()ed and exec()ed processes inherit file descriptors (except those marked CLOEXEC), including the listen() fd. Pending connections will queue in the kernel until userspace calls accept() on the listening fd. So one simple model is to stop calling accept(), cleanly/quickly finish up current connections, set an environment variable to tell the future instance that the listening fd X is already open, and exec your own binary again. A more complicated one is to fork, have the (identical) child just finish the current connections, while the parent execs itself similar to above. (The client connection fds should be marked CLOEXEC in this case.) With a more complicated service with more moving parts, libraries, threads, getting the above to work out is more complicated. But that's basically how you want to do it.
- mortehu 11y ago
  I program all my server programs to have a --listen-fd=N command line option. That way, the process that starts the server is responsible for creating the listening socket, and it can create multiple server instances using the same socket. The servers can then be programmed to handle SIGUSR1 by no longer calling accept, and terminating when the last client exits. This way, the amount of code required in each server program is tiny; less than 15 lines, with no calls to fork or execve. systemd already supports creating listening sockets, but its mode of operation is more similar to inetd. I think supporting something like what my proprietary launcher program does would require very little changes. Another advantage of passing the listening socket like this is that the server process can be in a private network namespace, without requiring any type of NAT setup or port mapping.
0x0 11y ago
The irc client irssi has something like this in /upgrade where it spawns a new binary but passing along active socket connections and their associated irc room states, I believe
- lorenzhs 11y ago
  WeeChat definitely does this. I don't know whether irssi supports it as well. Too bad WeeChat can't keep TLS connections open on /upgrade though
  - zobzu 11y ago
    yeah weechat is one of the rare C apps to actually preserve state on upgrade (passes fds, saves state, load in new process, kill old process) its not that hard to do if the program is made for this, even in C, its pretty hard to add when it isnt tho since you need a compat. layer. Sometimes the new program simply has import functions for the old memory layout.
gregham 11y ago
Yes, it's true. NGINX does zero-downtime configuration reloads and binary upgrades since 2004 without any dirty hacks.
gull 11y ago
The holy grail of high-availability isn't upgrading stateless software. It's upgrading stateful ones. Like upgrading when a data structure changes between versions. The HTTP protocol nginx serves is stateless and by comparison far simpler. Same goes for Erlang. It offers nothing more than simple function replacement, and that's not enough to handle data structure changes either.
- vardump 11y ago
  Synchronize to a checkpoint. Serialize all state. Run data structure upgrading code. Deserialize upgraded state. Of course possible to do in-place, but I'd imagine serialization makes testing it easier. Not to mention sending error reports if something goes wrong. Having all that state in a bug report could help a bit!
- toast0 11y ago
  In Erlang, if you need a new data structure for your state, you can check if your state is old, and upgrade it and continue on. In a gen_server, you might have something like handle_call(Request, From, State) when is_record(State, state) -> handle_call(Request, From, upgrade_state(State); handle_call(Request, From, State) when is_record(State, state2) -> ... (you'll want to do something similar on handle_cast and handle_info if you use those). You have to do a little work, but I don't see how you avoid that?
  - gull 11y ago
    That's not guaranteed to be safe. Having new code check if your state is old and upgrading isn't enough. You also need to check old code doesn't process new state. That becomes harder under concurrency. http://en.wikipedia.org/wiki/Dynamic_software_updating#Update_safety http://en.wikipedia.org/wiki/Dynamic_software_updating#Updat... Erlang doesn't offer this check. "Old code may still be evaluated because of processes lingering in the old code." http://www.erlang.org/doc/reference_manual/code_loading.html#id86424 http://www.erlang.org/doc/reference_manual/code_loading.html...
    - serialx 11y ago

MCRed 11y ago

If it's the holy grail, then erlang has had it for a couple decades... and whether it actually works in nginx I don't know, but it does work in erlang. The way it's handled is that the process where the previous socket was connected remains in place until it terminates but all new sockets connect to the new code.

nodesocket 11y ago
"The way it's handled is that the process where the previous socket was connected remains in place until it terminates but all new sockets connect to the new code." That would imply that two separate processes are bound to the same port right? I thought that was not possible.
- SEJeff 11y ago
  Things have changed :) https://lwn.net/Articles/542629/ https://lwn.net/Articles/542629/
  - bkirkbri 11y ago
    If only the JVM could take advantage of this.
    - SEJeff 11y ago
      Just about bet you could do that via some grotesque JNI work, but I wouldn't wish JNI on really anyone.
- ericmoritz 11y ago
  No, usually there's an accept process spawning working processes and passing the connection off to it. Erlang can have two versions of a module present in memory at a time. At the moment of a code upgrade, the processes working with existing connections continue to use the old code while newly accepted connections are passed to the version of the module that was loaded in between connection accepts.
- ericmoritz 11y ago
  Keep in mind that Erlang processes != System processes. Erlang processes are preemptively scheduled microthreads that are scheduled by the Erlang VM.
- rb12345 11y ago
  Presumably the original server process uses exec() to start the new process, which inherits the open socket handles, and the descriptor is passed as an argument to the new process. (e.g. http://stackoverflow.com/questions/14351147/perl-passing-an-open-socket-across-fork-exec http://stackoverflow.com/questions/14351147/perl-passing-an-...)
- noselasd 11y ago
  While there are certain socket options to share a port - two processes can share a port easily in another elegant way; just call fork(). One way of doing such upgrades is then for the existing code to fork() then exec() the new version. The existing old code closes the original server socket, which leaves accepting new requests to the new code. When all in-flight requests served by the old code is finished, that process can exit.
Blackthorn 11y ago
This is known to me as a "lame duck" mode, and I thought it was a standard thing to do!

Denzel 11y ago

Yes, nginx allows you to upgrade the binary and reload the configuration without any downtime: http://wiki.nginx.org/CommandLine#Upgrading_To_a_New_Binary_On_The_Fly http://wiki.nginx.org/CommandLine#Upgrading_To_a_New_Binary_...

yeukhon 11y ago
Does this work when you update with package manager?
- LukeShu 11y ago
  Yes.

ploxiln 11y ago

I didn't want to be a negative nancy in the comments for that haproxy article... but that is a ridiculous ugly hack. It's really a lot easier to achieve real and robust zero-downtime upgrades for a simple unix process. Remember, fork()ed and exec()ed processes inherit file descriptors (except those marked CLOEXEC), including the listen() fd. Pending connections will queue in the kernel until userspace calls accept() on the listening fd. So one simple model is to stop calling accept(), cleanly/quickly finish up current connections, set an environment variable to tell the future instance that the listening fd X is already open, and exec your own binary again. A more complicated one is to fork, have the (identical) child just finish the current connections, while the parent execs itself similar to above. (The client connection fds should be marked CLOEXEC in this case.) With a more complicated service with more moving parts, libraries, threads, getting the above to work out is more complicated. But that's basically how you want to do it.

mortehu 11y ago
I program all my server programs to have a --listen-fd=N command line option. That way, the process that starts the server is responsible for creating the listening socket, and it can create multiple server instances using the same socket. The servers can then be programmed to handle SIGUSR1 by no longer calling accept, and terminating when the last client exits. This way, the amount of code required in each server program is tiny; less than 15 lines, with no calls to fork or execve. systemd already supports creating listening sockets, but its mode of operation is more similar to inetd. I think supporting something like what my proprietary launcher program does would require very little changes. Another advantage of passing the listening socket like this is that the server process can be in a private network namespace, without requiring any type of NAT setup or port mapping.

0x0 11y ago

The irc client irssi has something like this in /upgrade where it spawns a new binary but passing along active socket connections and their associated irc room states, I believe

lorenzhs 11y ago
WeeChat definitely does this. I don't know whether irssi supports it as well. Too bad WeeChat can't keep TLS connections open on /upgrade though
- zobzu 11y ago
  yeah weechat is one of the rare C apps to actually preserve state on upgrade (passes fds, saves state, load in new process, kill old process) its not that hard to do if the program is made for this, even in C, its pretty hard to add when it isnt tho since you need a compat. layer. Sometimes the new program simply has import functions for the old memory layout.

gregham 11y ago

Yes, it's true. NGINX does zero-downtime configuration reloads and binary upgrades since 2004 without any dirty hacks.

gull 11y ago

The holy grail of high-availability isn't upgrading stateless software. It's upgrading stateful ones. Like upgrading when a data structure changes between versions. The HTTP protocol nginx serves is stateless and by comparison far simpler. Same goes for Erlang. It offers nothing more than simple function replacement, and that's not enough to handle data structure changes either.

vardump 11y ago
Synchronize to a checkpoint. Serialize all state. Run data structure upgrading code. Deserialize upgraded state. Of course possible to do in-place, but I'd imagine serialization makes testing it easier. Not to mention sending error reports if something goes wrong. Having all that state in a bug report could help a bit!
toast0 11y ago
In Erlang, if you need a new data structure for your state, you can check if your state is old, and upgrade it and continue on. In a gen_server, you might have something like handle_call(Request, From, State) when is_record(State, state) -> handle_call(Request, From, upgrade_state(State); handle_call(Request, From, State) when is_record(State, state2) -> ... (you'll want to do something similar on handle_cast and handle_info if you use those). You have to do a little work, but I don't see how you avoid that?
- gull 11y ago
  That's not guaranteed to be safe. Having new code check if your state is old and upgrading isn't enough. You also need to check old code doesn't process new state. That becomes harder under concurrency. http://en.wikipedia.org/wiki/Dynamic_software_updating#Update_safety http://en.wikipedia.org/wiki/Dynamic_software_updating#Updat... Erlang doesn't offer this check. "Old code may still be evaluated because of processes lingering in the old code." http://www.erlang.org/doc/reference_manual/code_loading.html#id86424 http://www.erlang.org/doc/reference_manual/code_loading.html...
  - serialx 11y ago