3 ms·

Thanks for the post! Is there anything you could suggest looking into to improve nginx processing of high volume large POST bodies (5KB, tens of thousands per

Thanks for the post! Is there anything you could suggest looking into to improve nginx processing of high volume large POST bodies (5KB, tens of thousands per second)? I'm using kernel 4.13 with BBR congestion control on 20Gbps network and seem to hit this weird bottleneck when it does not matter how many nginx processes I have, it works similarly terrible on both 16-core and 64-core servers. (Of course irq/process affinity is in place which makes me think it's nginx issue.)

slackingoff2017 9y ago
At that speed it might not be Nginx. Check out Cloudflare's other blog posts too. https://blog.cloudflare.com/how-to-receive-a-million-packets/ https://blog.cloudflare.com/how-to-receive-a-million-packets... And thanks OP. I'm sure thousands of companies do these kinds of optimizations but CloudFlare is one of the only to bother sharing.
spydum 9y ago
Not an nginx user here, but I do know sometimes certain modules in Apache will add significant latency on POST payloads by waiting for the entire POST payload to complete before sending to the backend (in a reverse proxy setup). The idea is that if the backend fails, it can retry on the next node. For large payloads this sucks for latency.. No idea if nginx has this problem.
- edwhitesell 9y ago
  This is a configurable option in nginx. You can have it wait until the entire request is recevied before passing to the upstream (backend) or have it stream as data arrives. I seem to recall there is still a manageable buffer when streaming too, though it's been a few years since I looked at that in detail.
tyingq 9y ago
You mention 5KB...is there some chance that it exceeds 8KB? There's a setting called client_body_buffersize. (http://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size http://nginx.org/en/docs/http/ngx_http_core_module.html#clie...) It defaults to 8k on x86-64. If you exceed that setting, it writes the body to a temporary file, which slows things down. Maybe try bumping that up to 16k? Also see the other related settings on that page. Edit: Maybe also test out switching from urlencoded to multipart for the post? For some types of data, urlencoded can bloat the size quite a lot.

slackingoff2017 9y ago

At that speed it might not be Nginx. Check out Cloudflare's other blog posts too. https://blog.cloudflare.com/how-to-receive-a-million-packets/ https://blog.cloudflare.com/how-to-receive-a-million-packets... And thanks OP. I'm sure thousands of companies do these kinds of optimizations but CloudFlare is one of the only to bother sharing.

spydum 9y ago

Not an nginx user here, but I do know sometimes certain modules in Apache will add significant latency on POST payloads by waiting for the entire POST payload to complete before sending to the backend (in a reverse proxy setup). The idea is that if the backend fails, it can retry on the next node. For large payloads this sucks for latency.. No idea if nginx has this problem.

edwhitesell 9y ago
This is a configurable option in nginx. You can have it wait until the entire request is recevied before passing to the upstream (backend) or have it stream as data arrives. I seem to recall there is still a manageable buffer when streaming too, though it's been a few years since I looked at that in detail.

tyingq 9y ago

You mention 5KB...is there some chance that it exceeds 8KB? There's a setting called client_body_buffersize. (http://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size http://nginx.org/en/docs/http/ngx_http_core_module.html#clie...) It defaults to 8k on x86-64. If you exceed that setting, it writes the body to a temporary file, which slows things down. Maybe try bumping that up to 16k? Also see the other related settings on that page. Edit: Maybe also test out switching from urlencoded to multipart for the post? For some types of data, urlencoded can bloat the size quite a lot.