3 ms·

To simplify this process further, you can run both the new and old gunicorn process on the same port using the SO_REUSEPORT feature in Linux 3.9. This way you d

To simplify this process further, you can run both the new and old gunicorn process on the same port using the SO_REUSEPORT feature in Linux 3.9. This way you don't need to listen on a new port and update nginx config (potentially once for each upstream web server). I submitted a patch to gunicorn just this week about this: https://github.com/benoitc/gunicorn/issues/598 https://github.com/benoitc/gunicorn/issues/598. Ideally I'd love to see all server software that listens on a port have a SO_REUSEPORT option for hot-swapping. This feature makes operations so much simpler.

abrookewood 11y ago
But if both processes are listening on the same port, how would he check the health of only the new process?? How do you determine which process answers each incoming connection?
- andy_ppp 11y ago
  Will the old processes not be successfully killed and the new processes just take over? Healthwise supervisor would be the one checking the gunicorn processes? Looking at this article is just seems to make ports operate either with any new server initialisation just automatically takes over the port (or maybe they become effectively load balanced between all running processes, it's hard to tell conclusively)... http://freeprogrammersblog.vhex.net/post/linux-39-introdued-new-way-of-writing-socket-servers/2 http://freeprogrammersblog.vhex.net/post/linux-39-introdued-...
- smilliken 11y ago
  You can keep both the new and old process running concurrently as long as you like, and they'll both be responding to requests. You can run your health checks at this time (perhaps distinguishing the process with a header). When you're confident, send a signal to the old process to gracefully exit.
geofft 11y ago
I'm not sure if I believe that SO_REUSEPORT is the right thing here. While both servers are running, aren't they both answering requests, depending on which one gets scheduled first? Can you work around this by instructing the first server to stop calling accept() before launching the second server? (But keep the first server's listening socket open, in case the second server doesn't start, so you can tell it to start calling accept() again.)
- smilliken 11y ago
  Yes, that's exactly it. First both processes are accepting (at the whim of the scheduler), then the old stops accepting, and then the old exits when it's finished all outstanding requests.
  - geofft 11y ago
    Why do you want both processes to accept? That sounds like a bad thing, since you can't test to make sure the new server came up right. You have no way to make sure you're hitting the new process, until you tell the old to stop accepting. And you want to be able to recover if the new process doesn't work. What I'd suggest is to add a "stop accepting" and "start accepting" signal, maybe a URL route that checks that it came from localhost or something and sets/clears a flag. If the flag is set, the mainloop skips all calls to accept(). So the full set up is, first the old process stops accepting, but keeps the listening socket open, so new connections queue in the kernel. Then you start the new process and make sure it works. Then you shut down the old process. If the new process doesn't work (e.g., accepts connections and returns 500s or something), then you've lost some requests, but the old process is still around. So you can signal it to start accepting again. Maybe this is overengineering, but the case I'm worried about is where something in the state of the old process is important (it has an open database connection, it has a module loaded into memory that got corrupted/deleted on disk, etc.). So if you shut down the old process before you test the new one, you have no guarantee you can get the old process back up if you need it. You may have lost the node entirely.

abrookewood 11y ago

But if both processes are listening on the same port, how would he check the health of only the new process?? How do you determine which process answers each incoming connection?

andy_ppp 11y ago
Will the old processes not be successfully killed and the new processes just take over? Healthwise supervisor would be the one checking the gunicorn processes? Looking at this article is just seems to make ports operate either with any new server initialisation just automatically takes over the port (or maybe they become effectively load balanced between all running processes, it's hard to tell conclusively)... http://freeprogrammersblog.vhex.net/post/linux-39-introdued-new-way-of-writing-socket-servers/2 http://freeprogrammersblog.vhex.net/post/linux-39-introdued-...
smilliken 11y ago
You can keep both the new and old process running concurrently as long as you like, and they'll both be responding to requests. You can run your health checks at this time (perhaps distinguishing the process with a header). When you're confident, send a signal to the old process to gracefully exit.

geofft 11y ago

I'm not sure if I believe that SO_REUSEPORT is the right thing here. While both servers are running, aren't they both answering requests, depending on which one gets scheduled first? Can you work around this by instructing the first server to stop calling accept() before launching the second server? (But keep the first server's listening socket open, in case the second server doesn't start, so you can tell it to start calling accept() again.)

smilliken 11y ago
Yes, that's exactly it. First both processes are accepting (at the whim of the scheduler), then the old stops accepting, and then the old exits when it's finished all outstanding requests.
- geofft 11y ago
  Why do you want both processes to accept? That sounds like a bad thing, since you can't test to make sure the new server came up right. You have no way to make sure you're hitting the new process, until you tell the old to stop accepting. And you want to be able to recover if the new process doesn't work. What I'd suggest is to add a "stop accepting" and "start accepting" signal, maybe a URL route that checks that it came from localhost or something and sets/clears a flag. If the flag is set, the mainloop skips all calls to accept(). So the full set up is, first the old process stops accepting, but keeps the listening socket open, so new connections queue in the kernel. Then you start the new process and make sure it works. Then you shut down the old process. If the new process doesn't work (e.g., accepts connections and returns 500s or something), then you've lost some requests, but the old process is still around. So you can signal it to start accepting again. Maybe this is overengineering, but the case I'm worried about is where something in the state of the old process is important (it has an open database connection, it has a module loaded into memory that got corrupted/deleted on disk, etc.). So if you shut down the old process before you test the new one, you have no guarantee you can get the old process back up if you need it. You may have lost the node entirely.