5 ms·

We did this for Caddy 1 too [1]. It was really cool. I am not sure how many people used this feature, so I haven't implemented it for Caddy 2 yet, and in the ~t

by mholt 5y ago

We did this for Caddy 1 too [1]. It was really cool. I am not sure how many people used this feature, so I haven't implemented it for Caddy 2 yet, and in the ~two years that Caddy 2 has been released, I've only had the request once. It's a bit tricky/tedious to do properly, but I'm willing to bring it over to Caddy 2 with a sufficient sponsorship. [1]: https://github.com/caddyserver/caddy/blob/v1/upgrade.go https://github.com/caddyserver/caddy/blob/v1/upgrade.go

JesseObrien 5y ago
Can you explain any of the technical details around this perchance? I'm super curious. I know that SO_REUSEPORT[1] exists but is that the only little trick to make this work? From what I've read with SO_REUSEPORT it can open up that port to hijacking by rogue processes, so is that fine to rely on? [1] https://lwn.net/Articles/542629/ https://lwn.net/Articles/542629/
- duskwuff 5y ago
  You don't even need that. If the old server process exec()s the new one, it can pass on its file descriptors -- including the listening socket -- when that happens.
  - tyingq 5y ago
    You could also be fancy and pass open sockets over a unix domain socket with sendmsg().
    - xyzzy_plugh 5y ago
      This is the best way as it avoids any sort of session/parenting issues which are not always easy to solve portably as a parent.
  - mholt 5y ago
    Yep, we don't use SO_REUSEPORT. We just pass it from the old process to the new one.
- tyingq 5y ago
  >it can open up that port to hijacking by rogue processes That seems relevant if the process is using a non-privileged port that's >= 1024. If we're talking about privileged ports (<= 1023), though, only another root process could hijack that, and those can already hijack you many other ways.
  - cmeacham98 5y ago
    What about processes that aren't root but hold CAP_NET_BIND_SERVICE?
    - tyingq 5y ago
      Sure, should have mentioned that, and perhaps namespaces too.
- fragmede 5y ago
  If an attacker is already running rogue processes on your box, the minor details surrounding SO_REUSEPORT is the least of your worries. An attacker could just restart nginx, and won't care about lost requests.
- khc 5y ago
  poked around a bit of that from a previous job, here's what I remember: 1. there's a control process and worker processes 2. on upgrade, control process launches new worker processes from the new binary 3. requests are drained from old worker processes 4. most of the time nginx request handlers allocate from a per-request allocation pool, so requests mostly don't share memory 5. for the cases where there are global states, there's a separate shared memory pool that you need to allocate from (which is kind of hard to work if you are not using built-in nginx primitives)
eliaspro 5y ago
I'm torn on this feature. In the one hand, an application should never be able to replace itself with "random code" to be executed. I want my systems to be immutable. I want my services to be run with the smallest set of privileges required. On the other hand, it encourages "consumer level" users to keep their software up-to-date, even when it wasn't installed from a distribution's repository etc. So I think in general it's a good feature to have, as advanced users/distributions will restrict what a service/process is able to to anyways and won't have any downsides of not using this feature. It should be optional, that's all!
- mholt 5y ago
  > an application should never be able to replace itself with "random code" to be executed. To clarify: it doesn't, nor has it ever worked that way. You have to be the one to do that (or someone with privileges to write to that file on disk). Most production setups don't give Caddy that permission. And you have to trigger the upgrade too.
  - eliaspro 5y ago
    Ok, this explains a lot and also clearly shows I never used this feature :) I always assumed it would be the Caddy process itself taking care of downloading the update & replacing the binary, before restarting it.
    - mholt 5y ago
      Caddy 2 can do that at least, but it's something you have to command it to do. :)
- blibble 5y ago
  if you can log into the machine and replace the nginx executable you are probably capable of running it too
zimbatm 5y ago
If Caddy were to support systemd socket activation, this self-restart dance is not necessary as the parent process (systemd) is holding the socket for you. And for other systems, they can use https://github.com/zimbatm/socketmaster https://github.com/zimbatm/socketmaster instead. I believe this to be more elegant and robust than the nginx approach as there is no PID re-parenting issues. But I suspect that most Caddy deployments are done via docker, and that requires a whole container restart anyways.
- mholt 5y ago
  Good point, and I'm not sure which deployment method is more popular. In general I am personally not a fan of Docker due to added complexities (often unnecessary for static binaries like Caddy) and technical limitations such as this. All my Caddy deployments use systemd (which I don't love either, sigh).
  - zimbatm 5y ago
    Technically, it should be possible to pass a file descriptor to a docker process, but I haven't seen that before. I don't like all parts of systemd but I think the socket activation is pretty elegant. The decoupling also allows to bind on port 80 and 443 (because systemd runs as root), and then still have Caddy run as a user process. I think it would unlock nice things for Caddy.
- tyingq 5y ago
  It's kind of fun to watch things go out of fashion and back in. We used to use inetd, mostly because memory was expensive, so it could spawn a service only when a request came in, then the spawned process would exit and give the memory back to the os. Then someone decided tcpd should sit between inetd and servers, for security and logging. Then, every service just ran as it's own daemon. Now I'm occasionally seeing posts like this reviving inetd.
  - zimbatm 5y ago
    It's the same but a bit different. Inetd is listening to a port and then for each new connection, spawning a new process, binding stdin/stout to the socket pair. The main issue was that it could lead to system resource exhaustion pretty easily if too many connections were being opened and there were no good ways to control that. With systemd, the listening socket is only bound by systemd and passed to the service. The service itself is responsible for accepting and handling the new connections. So it has the control on the rate of new connections, and can also more easily share memory. The main advantage of that approach is that as soon as systemd binds the socket, new connections won't be rejected by the system and will be on hold until the service accept() them. So no connection gets dropped, even during a restart. The service itself is still responsible for gracefully shutting down existing connections on SIGTERM.
latchkey 5y ago
Someone did this for golang. It isn't perfect, but works for some basic use cases... https://github.com/jpillora/overseer https://github.com/jpillora/overseer

by mholt 5y ago

JesseObrien 5y ago

Can you explain any of the technical details around this perchance? I'm super curious. I know that SO_REUSEPORT[1] exists but is that the only little trick to make this work? From what I've read with SO_REUSEPORT it can open up that port to hijacking by rogue processes, so is that fine to rely on? [1] https://lwn.net/Articles/542629/ https://lwn.net/Articles/542629/

duskwuff 5y ago
You don't even need that. If the old server process exec()s the new one, it can pass on its file descriptors -- including the listening socket -- when that happens.
- tyingq 5y ago
  You could also be fancy and pass open sockets over a unix domain socket with sendmsg().
  - xyzzy_plugh 5y ago
    This is the best way as it avoids any sort of session/parenting issues which are not always easy to solve portably as a parent.
- mholt 5y ago
  Yep, we don't use SO_REUSEPORT. We just pass it from the old process to the new one.
tyingq 5y ago
>it can open up that port to hijacking by rogue processes That seems relevant if the process is using a non-privileged port that's >= 1024. If we're talking about privileged ports (<= 1023), though, only another root process could hijack that, and those can already hijack you many other ways.
- cmeacham98 5y ago
  What about processes that aren't root but hold CAP_NET_BIND_SERVICE?
  - tyingq 5y ago
    Sure, should have mentioned that, and perhaps namespaces too.
fragmede 5y ago
If an attacker is already running rogue processes on your box, the minor details surrounding SO_REUSEPORT is the least of your worries. An attacker could just restart nginx, and won't care about lost requests.
khc 5y ago
poked around a bit of that from a previous job, here's what I remember: 1. there's a control process and worker processes 2. on upgrade, control process launches new worker processes from the new binary 3. requests are drained from old worker processes 4. most of the time nginx request handlers allocate from a per-request allocation pool, so requests mostly don't share memory 5. for the cases where there are global states, there's a separate shared memory pool that you need to allocate from (which is kind of hard to work if you are not using built-in nginx primitives)

eliaspro 5y ago

I'm torn on this feature. In the one hand, an application should never be able to replace itself with "random code" to be executed. I want my systems to be immutable. I want my services to be run with the smallest set of privileges required. On the other hand, it encourages "consumer level" users to keep their software up-to-date, even when it wasn't installed from a distribution's repository etc. So I think in general it's a good feature to have, as advanced users/distributions will restrict what a service/process is able to to anyways and won't have any downsides of not using this feature. It should be optional, that's all!

mholt 5y ago
> an application should never be able to replace itself with "random code" to be executed. To clarify: it doesn't, nor has it ever worked that way. You have to be the one to do that (or someone with privileges to write to that file on disk). Most production setups don't give Caddy that permission. And you have to trigger the upgrade too.
- eliaspro 5y ago
  Ok, this explains a lot and also clearly shows I never used this feature :) I always assumed it would be the Caddy process itself taking care of downloading the update & replacing the binary, before restarting it.
  - mholt 5y ago
    Caddy 2 can do that at least, but it's something you have to command it to do. :)
blibble 5y ago
if you can log into the machine and replace the nginx executable you are probably capable of running it too

zimbatm 5y ago

If Caddy were to support systemd socket activation, this self-restart dance is not necessary as the parent process (systemd) is holding the socket for you. And for other systems, they can use https://github.com/zimbatm/socketmaster https://github.com/zimbatm/socketmaster instead. I believe this to be more elegant and robust than the nginx approach as there is no PID re-parenting issues. But I suspect that most Caddy deployments are done via docker, and that requires a whole container restart anyways.

mholt 5y ago
Good point, and I'm not sure which deployment method is more popular. In general I am personally not a fan of Docker due to added complexities (often unnecessary for static binaries like Caddy) and technical limitations such as this. All my Caddy deployments use systemd (which I don't love either, sigh).
- zimbatm 5y ago
  Technically, it should be possible to pass a file descriptor to a docker process, but I haven't seen that before. I don't like all parts of systemd but I think the socket activation is pretty elegant. The decoupling also allows to bind on port 80 and 443 (because systemd runs as root), and then still have Caddy run as a user process. I think it would unlock nice things for Caddy.
tyingq 5y ago
It's kind of fun to watch things go out of fashion and back in. We used to use inetd, mostly because memory was expensive, so it could spawn a service only when a request came in, then the spawned process would exit and give the memory back to the os. Then someone decided tcpd should sit between inetd and servers, for security and logging. Then, every service just ran as it's own daemon. Now I'm occasionally seeing posts like this reviving inetd.
- zimbatm 5y ago
  It's the same but a bit different. Inetd is listening to a port and then for each new connection, spawning a new process, binding stdin/stout to the socket pair. The main issue was that it could lead to system resource exhaustion pretty easily if too many connections were being opened and there were no good ways to control that. With systemd, the listening socket is only bound by systemd and passed to the service. The service itself is responsible for accepting and handling the new connections. So it has the control on the rate of new connections, and can also more easily share memory. The main advantage of that approach is that as soon as systemd binds the socket, new connections won't be rejected by the system and will be on hold until the service accept() them. So no connection gets dropped, even during a restart. The service itself is still responsible for gracefully shutting down existing connections on SIGTERM.

latchkey 5y ago

Someone did this for golang. It isn't perfect, but works for some basic use cases... https://github.com/jpillora/overseer https://github.com/jpillora/overseer