4 ms·

Does anyone have a solution to this problem with nginx I have had few times over the years? http://www.howtoforge.com/forums/showthread.php?p=296263#post296263

by pavs 14y ago

Does anyone have a solution to this problem with nginx I have had few times over the years? http://www.howtoforge.com/forums/showthread.php?p=296263#post296263 http://www.howtoforge.com/forums/showthread.php?p=296263#pos... Please note that I am not 100% sure if its nginx related.

mattparlane 14y ago
I see you're running FPM -- I had the same problem, and it turned out to be FPM's worker auto-spawning system. I set it to pm = static and set the number of workers to 50, which is the most I ever need, plus some headroom. Give that a go.
- pavs 14y ago
  Thanks I will give it a shot when I go home, I can only ssh to my server from home computer. You are right the pm is set to dynamic last time I checked, its the default. I will update you on this after observing my server for 24 hours.
- pavs 13y ago
  Just wanted to update you on this. I switched to another hosting provider (for other reasons) so I ended up working on the server from scratch. The pm is still dynamic because I want to see if I can reproduce the problem again, and if it does I can always try static. Thanks.
1SaltwaterC 14y ago
I can provide you some pointers as we made it go away. Some data from one of our production virtual hosts: cat access.log.1 | wc -l 1054423 # no static objects served here cat error.log.1 | grep timed For PHP-FPM we use static pm and unix domain sockets. This virtual host if fairly busy with some slow (~200 ms) requests, therefore it uses 96 processes per pool. listen.backlog = -1 in php-fpm.ini for letting the kernel decide the size of the actual connection backlog. UDS is getting filled faster than TCP and nginx starts responding with 502's. Throw net.core.somaxconn = 65535 somewhere in /etc/sysctl.d for increasing the actual backlog since even if you specify a high listen.backlog value, the actual value is truncated to SOMAXCONN. Couple of years ago I wrote an article about stuff like this: http://www.saltwaterc.eu/nginx-php-fpm-for-high-loaded-websites.html http://www.saltwaterc.eu/nginx-php-fpm-for-high-loaded-websi... (shameless plug, I know, but you may get some useful info). As a side note, I am curious how the backend persistency for nginx is playing. Our production still uses the same config since 2011 as it isn't broken, but it may be more efficient.
- pavs 13y ago
  I tried your suggestions listen backlog and somaxconn, are they dependent on unix socket or do I get benefit from it either way? Thanks.
  - 1SaltwaterC 13y ago
    Both TCP and UDS depend on it. UDS uses an API that follows the BSD sockets standard. However, TCP makes sense when you use nginx as load balancer between multiple PHP-FPM backends. A really tricky setup if you ask me. For UDS, besides lower latency, the namespace is cleaner (filesystem paths vs numeric ports - much easier to automate the configuration), and (at least under Linux) they follow the filesystem ACL. For example, under my setup, only nginx's user is allowed to read / write to the PHP-FPM sockets.

by pavs 14y ago

mattparlane 14y ago

I see you're running FPM -- I had the same problem, and it turned out to be FPM's worker auto-spawning system. I set it to pm = static and set the number of workers to 50, which is the most I ever need, plus some headroom. Give that a go.

pavs 14y ago
Thanks I will give it a shot when I go home, I can only ssh to my server from home computer. You are right the pm is set to dynamic last time I checked, its the default. I will update you on this after observing my server for 24 hours.
pavs 13y ago
Just wanted to update you on this. I switched to another hosting provider (for other reasons) so I ended up working on the server from scratch. The pm is still dynamic because I want to see if I can reproduce the problem again, and if it does I can always try static. Thanks.

1SaltwaterC 14y ago

I can provide you some pointers as we made it go away. Some data from one of our production virtual hosts: cat access.log.1 | wc -l 1054423 # no static objects served here cat error.log.1 | grep timed For PHP-FPM we use static pm and unix domain sockets. This virtual host if fairly busy with some slow (~200 ms) requests, therefore it uses 96 processes per pool. listen.backlog = -1 in php-fpm.ini for letting the kernel decide the size of the actual connection backlog. UDS is getting filled faster than TCP and nginx starts responding with 502's. Throw net.core.somaxconn = 65535 somewhere in /etc/sysctl.d for increasing the actual backlog since even if you specify a high listen.backlog value, the actual value is truncated to SOMAXCONN. Couple of years ago I wrote an article about stuff like this: http://www.saltwaterc.eu/nginx-php-fpm-for-high-loaded-websites.html http://www.saltwaterc.eu/nginx-php-fpm-for-high-loaded-websi... (shameless plug, I know, but you may get some useful info). As a side note, I am curious how the backend persistency for nginx is playing. Our production still uses the same config since 2011 as it isn't broken, but it may be more efficient.

pavs 13y ago
I tried your suggestions listen backlog and somaxconn, are they dependent on unix socket or do I get benefit from it either way? Thanks.
- 1SaltwaterC 13y ago
  Both TCP and UDS depend on it. UDS uses an API that follows the BSD sockets standard. However, TCP makes sense when you use nginx as load balancer between multiple PHP-FPM backends. A really tricky setup if you ask me. For UDS, besides lower latency, the namespace is cleaner (filesystem paths vs numeric ports - much easier to automate the configuration), and (at least under Linux) they follow the filesystem ACL. For example, under my setup, only nginx's user is allowed to read / write to the PHP-FPM sockets.