4 ms·

Thanks for this article, it's great to see people caring for their server (does it have a name?) and not defaulting to the serverless craze. Here's a few though

Thanks for this article, it's great to see people caring for their server (does it have a name?) and not defaulting to the serverless craze. Here's a few thoughts :) > there is some small downtime when I deploy new versions of things since I don't have any load balancing or rolling deployments It's entirely possible to achieve, depending on your stack. `nginx -s reload` will reload the entire config without killing existing connections or inducing undue downtime. So if you can start a second instance of your "webapp" on a separate port/socket (or folder for PHP) and point nginx to it there shouldn't be any downtime involved. > for users that are geographically far away latency can be high That's true, but counter-intuitively, i found unless you're serving huge content (think video or multiple MB pages) it's not a problem. CDN can actually make it worse on a bad connection, because it takes additional roundtrips to resolve the CDN's domain and fetch stuff from there while i already have a connection established to your site. As someone who regularly uses really poor xDSL (from the other side of the atlantic ocean) i have a better experience with sites without a CDN that fit in under 1MB (or even better < 200KB) with as little requests as possible (for the clients that don't support HTTP2). > CloudFlare (...) That may become necessary if I ever have trouble with DDOS attacks I've personally found OVH to be more than capable and willing to deal with DDOS for their customers. OVH has been previously posted on HN for dealing with huge DDOS. That is of course if you have proper caching and you don't have an easy venue for a remote attacker to induce huge CPU/RAM load. For example, Plausible-like analytics can be such an attack vector because every request is logged in a database; something like GoAccess [0] is more resilient, and no logs is even lighter on resources. [0] https://goaccess.io/ https://goaccess.io/

remram 5y ago
> there is some small downtime when I deploy new versions of things Some time ago I was looking for an easier way to fix this. It seemed to me that a good way would be to have the reverse proxy (e.g. nginx or similar) hold the requests into the app restarts. Fit the user, this would mean a ~10s hiccup rather than 504 errors. I didn't find an easy way to do it with nginx though and was sort of disappointed. Maybe other reverse proxies make this easier? Or maybe there is a stand-alone tool to do this? [edit: one app I host can only be run single-instance as it keeps some state in-process. It can outsource it to Redis, but that seems overkill if it's only needed during upgrades, 10s/week or so]
- anderspitman 5y ago
  This would be a cool feature for reverse proxies to have.
  - remram 5y ago
    Seems HAProxy can do it, though it's manual, not automatic when the app is unavailable during reload: https://serverfault.com/a/450983/187237 https://serverfault.com/a/450983/187237
- southerntofu 5y ago
  > [edit: one app I host can only be run single-instance as it keeps some state in-process. It can outsource it to Redis, but that seems overkill if it's only needed during upgrades, 10s/week or so] Not exactly the same, but couldn't you serve a 425 Too Early with a meta refresh of 10s to achieve exactly the same goal? So change your nginx config to serve this "updating, your browser will refresh automatically" page, reload nginx, update the app, revert nginx config and reload nginx. Would that not address your needs, albeit in a more convoluted way?
  - remram 5y ago
    That might work for HTML pages, API routes would need a different configuration. Then every app consuming the API needs to be updated to support the 425 code gracefully. In addition, I have to manually do this switch in nginx before and after the update... I would rather a system that holds requests for 20s max if the backend refuses connections.
    - southerntofu 5y ago
      > I would rather a system that holds requests for 20s max if the backend refuses connections. I think that's what nginx does by default, isn't it? Try again until "gateway timeout". The problem in this case is if you want graceful shutdown you need your backend to stop accepting new connections without stopping to process the existing ones, then update the app. If some client connections are long-lived, that's a hard problem and that's why in that case running a second instance makes sense if you can do it. > Then every app consuming the API needs to be updated to support the 425 code gracefully. That's indeed a problem, but your client should probably support some forms of server "failure" modes, including a "try again soon" type of reply (or interpreting 5XX as such). > I have to manually do this switch in nginx before and after the update... This can be fully automated. Just use different config files in sites-enabled. Then you can mv or ln to enable one or the other and run nginx -s reload. The whole process would be a few lines of shell script.

remram 5y ago

> there is some small downtime when I deploy new versions of things Some time ago I was looking for an easier way to fix this. It seemed to me that a good way would be to have the reverse proxy (e.g. nginx or similar) hold the requests into the app restarts. Fit the user, this would mean a ~10s hiccup rather than 504 errors. I didn't find an easy way to do it with nginx though and was sort of disappointed. Maybe other reverse proxies make this easier? Or maybe there is a stand-alone tool to do this? [edit: one app I host can only be run single-instance as it keeps some state in-process. It can outsource it to Redis, but that seems overkill if it's only needed during upgrades, 10s/week or so]

anderspitman 5y ago
This would be a cool feature for reverse proxies to have.
- remram 5y ago
  Seems HAProxy can do it, though it's manual, not automatic when the app is unavailable during reload: https://serverfault.com/a/450983/187237 https://serverfault.com/a/450983/187237
southerntofu 5y ago
> [edit: one app I host can only be run single-instance as it keeps some state in-process. It can outsource it to Redis, but that seems overkill if it's only needed during upgrades, 10s/week or so] Not exactly the same, but couldn't you serve a 425 Too Early with a meta refresh of 10s to achieve exactly the same goal? So change your nginx config to serve this "updating, your browser will refresh automatically" page, reload nginx, update the app, revert nginx config and reload nginx. Would that not address your needs, albeit in a more convoluted way?
- remram 5y ago
  That might work for HTML pages, API routes would need a different configuration. Then every app consuming the API needs to be updated to support the 425 code gracefully. In addition, I have to manually do this switch in nginx before and after the update... I would rather a system that holds requests for 20s max if the backend refuses connections.
  - southerntofu 5y ago
    > I would rather a system that holds requests for 20s max if the backend refuses connections. I think that's what nginx does by default, isn't it? Try again until "gateway timeout". The problem in this case is if you want graceful shutdown you need your backend to stop accepting new connections without stopping to process the existing ones, then update the app. If some client connections are long-lived, that's a hard problem and that's why in that case running a second instance makes sense if you can do it. > Then every app consuming the API needs to be updated to support the 425 code gracefully. That's indeed a problem, but your client should probably support some forms of server "failure" modes, including a "try again soon" type of reply (or interpreting 5XX as such). > I have to manually do this switch in nginx before and after the update... This can be fully automated. Just use different config files in sites-enabled. Then you can mv or ln to enable one or the other and run nginx -s reload. The whole process would be a few lines of shell script.