3 ms·

> there is some small downtime when I deploy new versions of things Some time ago I was looking for an easier way to fix this. It seemed to me that a good way

by remram 5y ago

> there is some small downtime when I deploy new versions of things Some time ago I was looking for an easier way to fix this. It seemed to me that a good way would be to have the reverse proxy (e.g. nginx or similar) hold the requests into the app restarts. Fit the user, this would mean a ~10s hiccup rather than 504 errors. I didn't find an easy way to do it with nginx though and was sort of disappointed. Maybe other reverse proxies make this easier? Or maybe there is a stand-alone tool to do this? [edit: one app I host can only be run single-instance as it keeps some state in-process. It can outsource it to Redis, but that seems overkill if it's only needed during upgrades, 10s/week or so]

anderspitman 5y ago
This would be a cool feature for reverse proxies to have.
- remram 5y ago
  Seems HAProxy can do it, though it's manual, not automatic when the app is unavailable during reload: https://serverfault.com/a/450983/187237 https://serverfault.com/a/450983/187237
southerntofu 5y ago
> [edit: one app I host can only be run single-instance as it keeps some state in-process. It can outsource it to Redis, but that seems overkill if it's only needed during upgrades, 10s/week or so] Not exactly the same, but couldn't you serve a 425 Too Early with a meta refresh of 10s to achieve exactly the same goal? So change your nginx config to serve this "updating, your browser will refresh automatically" page, reload nginx, update the app, revert nginx config and reload nginx. Would that not address your needs, albeit in a more convoluted way?
- remram 5y ago
  That might work for HTML pages, API routes would need a different configuration. Then every app consuming the API needs to be updated to support the 425 code gracefully. In addition, I have to manually do this switch in nginx before and after the update... I would rather a system that holds requests for 20s max if the backend refuses connections.
  - southerntofu 5y ago
    > I would rather a system that holds requests for 20s max if the backend refuses connections. I think that's what nginx does by default, isn't it? Try again until "gateway timeout". The problem in this case is if you want graceful shutdown you need your backend to stop accepting new connections without stopping to process the existing ones, then update the app. If some client connections are long-lived, that's a hard problem and that's why in that case running a second instance makes sense if you can do it. > Then every app consuming the API needs to be updated to support the 425 code gracefully. That's indeed a problem, but your client should probably support some forms of server "failure" modes, including a "try again soon" type of reply (or interpreting 5XX as such). > I have to manually do this switch in nginx before and after the update... This can be fully automated. Just use different config files in sites-enabled. Then you can mv or ln to enable one or the other and run nginx -s reload. The whole process would be a few lines of shell script.
    - remram 5y ago
      This is not my experience with nginx, I am seeing requests eventually time out. Then I am seeing 504 errors even though the backend is up, seemingly like it takes some time for nginx to notice. This might very well be a configuration error on my part though. > your client should probably support some forms of server "failure" modes, including a "try again soon" type of reply Currently the "try again soon" is passed on directly to the user. For a lot of endpoints, the client is a web browser for which this is not possible (short of turning the whole thing into a PWA... just for the ~10s a week the app is restarting...)

by remram 5y ago

anderspitman 5y ago

This would be a cool feature for reverse proxies to have.

remram 5y ago
Seems HAProxy can do it, though it's manual, not automatic when the app is unavailable during reload: https://serverfault.com/a/450983/187237 https://serverfault.com/a/450983/187237

southerntofu 5y ago

> [edit: one app I host can only be run single-instance as it keeps some state in-process. It can outsource it to Redis, but that seems overkill if it's only needed during upgrades, 10s/week or so] Not exactly the same, but couldn't you serve a 425 Too Early with a meta refresh of 10s to achieve exactly the same goal? So change your nginx config to serve this "updating, your browser will refresh automatically" page, reload nginx, update the app, revert nginx config and reload nginx. Would that not address your needs, albeit in a more convoluted way?

remram 5y ago
That might work for HTML pages, API routes would need a different configuration. Then every app consuming the API needs to be updated to support the 425 code gracefully. In addition, I have to manually do this switch in nginx before and after the update... I would rather a system that holds requests for 20s max if the backend refuses connections.
- southerntofu 5y ago
  > I would rather a system that holds requests for 20s max if the backend refuses connections. I think that's what nginx does by default, isn't it? Try again until "gateway timeout". The problem in this case is if you want graceful shutdown you need your backend to stop accepting new connections without stopping to process the existing ones, then update the app. If some client connections are long-lived, that's a hard problem and that's why in that case running a second instance makes sense if you can do it. > Then every app consuming the API needs to be updated to support the 425 code gracefully. That's indeed a problem, but your client should probably support some forms of server "failure" modes, including a "try again soon" type of reply (or interpreting 5XX as such). > I have to manually do this switch in nginx before and after the update... This can be fully automated. Just use different config files in sites-enabled. Then you can mv or ln to enable one or the other and run nginx -s reload. The whole process would be a few lines of shell script.
  - remram 5y ago
    This is not my experience with nginx, I am seeing requests eventually time out. Then I am seeing 504 errors even though the backend is up, seemingly like it takes some time for nginx to notice. This might very well be a configuration error on my part though. > your client should probably support some forms of server "failure" modes, including a "try again soon" type of reply Currently the "try again soon" is passed on directly to the user. For a lot of endpoints, the client is a web browser for which this is not possible (short of turning the whole thing into a PWA... just for the ~10s a week the app is restarting...)