3 ms·

Can somebody speak to why dynamic upstreams included in a file paired with `sudo service nginx reload` for prod deploys stopped scaling?

SaveTheRbtz 6y ago
It is easy enough for simple cases (and we used it for quite a while, until we moved to using Lua for that.) For more complex scenarios you will have new `server` blocks, certificates, tls tickets, log files / syslog endpoints, so the automation will end up interacting not with just a single dynamic upstream file but with rather large amount of system interfaces. Control-plane ends up being distributed between config generation, filesystem state, service configuration (e.g. syslog.) On a more practical note, each nginx `reload` will double the number of workers, almost doubling memory consumption and significantly increasing CPU usage (need to re-establish all TCP connections, re-do TLS handshake, etc.) So there is only that many reloads that you can do in an hour.
- DarkWiiPlayer 6y ago
  nginx is not well suited for constantly reconfiguring your infrastructure on very hot servers. This is a problem when you expose such infrastructure configurations to users (think cloudflare), but otherwise you can just mitigate this problem by having a sane deployment strategy.
nielsole 6y ago
Nginx configuration is bound to its workers. When you reload nginx new workers are created(and start responding to new connections) and the old ones are drained. The draining finishes when the last connection is finished or a timeout is reached. In OSS nginx every upstream change requires a configuration reload. If you have lots of upstream changes and don't want to terminate connections prematurely, this can quickly require lots of RAM as you have many workers. Stock nginx worker is around 150Mb, but issues with openresty integration(they mention lua usage) can bloat this to > 1GB.

SaveTheRbtz 6y ago

It is easy enough for simple cases (and we used it for quite a while, until we moved to using Lua for that.) For more complex scenarios you will have new `server` blocks, certificates, tls tickets, log files / syslog endpoints, so the automation will end up interacting not with just a single dynamic upstream file but with rather large amount of system interfaces. Control-plane ends up being distributed between config generation, filesystem state, service configuration (e.g. syslog.) On a more practical note, each nginx `reload` will double the number of workers, almost doubling memory consumption and significantly increasing CPU usage (need to re-establish all TCP connections, re-do TLS handshake, etc.) So there is only that many reloads that you can do in an hour.

DarkWiiPlayer 6y ago
nginx is not well suited for constantly reconfiguring your infrastructure on very hot servers. This is a problem when you expose such infrastructure configurations to users (think cloudflare), but otherwise you can just mitigate this problem by having a sane deployment strategy.

nielsole 6y ago

Nginx configuration is bound to its workers. When you reload nginx new workers are created(and start responding to new connections) and the old ones are drained. The draining finishes when the last connection is finished or a timeout is reached. In OSS nginx every upstream change requires a configuration reload. If you have lots of upstream changes and don't want to terminate connections prematurely, this can quickly require lots of RAM as you have many workers. Stock nginx worker is around 150Mb, but issues with openresty integration(they mention lua usage) can bloat this to > 1GB.