3 ms·

HTTP is specifically the case where Nagle will probably hit most often. If you treat TCP just like file stream (it's a stream after all) and you're implementin

HTTP is specifically the case where Nagle will probably hit most often. If you treat TCP just like file stream (it's a stream after all) and you're implementing, say, a webserver, the most straightforward way to implement a response is to: - figure out what was requested - build headers - send headers - send the file body especially if you intend to send the file body using sendfile. but this pattern is broken by Nagle - because there's no return traffic between the two sends in HTTP and the headers often won't fill out a full TCP packet, you'll trigger the wait.

lmilcin 6y ago
The solution I have implemented in my case is to move the problem to the application by designing the writes a little bit better. Instead of writing into dumb stream the application writes buffers of data or stream of data but each time indicating if there is more data that will immediately follow. In any case, TCP_NODELAY is enabled and all writes are immediately flushed unless the application clearly indicates it has a queue of data that will immediately follow.
nly 6y ago
In your example sending the file body will fill the remainder of the packet containing the headers and trigger a packet send. The client waiting for your response isn't going to lose out. In fact, they'll benefit from having file body data in that first packet. Your example is when you want Nagles algorithm to apply. This exact use case is mentioned on tcp(7) under the TCP_CORK option, which is like a more explicit, application controlled, version of Nagle. Nagles algorithm sucks for HTTP clients who happen to send big HTTP requests. If the HTTP headers being sent are >1400 bytes or so, then the tail end of the request will wait 1 RTT for the ACK of the first part. This will be particularly bad if the HTTP server has delayed-ACKs enabled (TCP_QUICKACK is not on). The server TCP stack has no idea that the HTTP server application hasn't yet got a full request and can't yet respond, so will delay the ACK of the first 1400 request bytes in the hope that it will. The delayed ACK delay is probably small compared to the RTT, so Nagle still dominates
derefr 6y ago
True, but also kind of irrelevant, in that HTTP1.x is a protocol designed under the constraint that there are going to be an unbounded number of application-level proxies/gateways between the client and the server, where some of them might be under the client’s control (e.g. Tor), some under the server’s (e.g. Varnish), and some neither (e.g. a corporate or ISP WAF.) And because of that, modern web servers don’t really do NODELAY; rather the opposite — they fully buffer HTTP requests and responses into what are essentially single giant datagrams. Put your Express app behind Nginx, and Nginx will buffer the entire request to feed it to your app as one TCP jumbo-frame (if possible); and then buffer your response to feed it back down the line to the client as one TCP jumbo-frame, or a burst of such packets. This happens not just with HEAD responses, but with full-bodied GETs/POSTs/PUTs, too. The only time modern web servers attempt to break your buffer up and schedule sends at different times, is when you tell them that you’re doing a `Transfer-Encoding: Chunked` response; or when your upstream is HTTP/2 and sends down an HTTP/2 stream non-terminal body-frame; or when your upstream is speaking websockets or WebRTC. Otherwise you get this “hyper-Nagling.” (Why do they do it? Because these servers want to achieve sub-worker-thread concurrency, and system calls like write(2) are blocking, getting in the way of that worker doing anything else. So web-server worker threads want to do as few system calls as possible, which means batching up your writes so they can do one context-switch for one big write(2) or writev(2). For these servers, sendfile(2) can actually be an anti-pattern, compared to mmap(2)ing the relevant static files so that they can just be passed as a pointer to writev(2) along with the headers buffer. That’s one blocking context-switch rather than two!)

lmilcin 6y ago

The solution I have implemented in my case is to move the problem to the application by designing the writes a little bit better. Instead of writing into dumb stream the application writes buffers of data or stream of data but each time indicating if there is more data that will immediately follow. In any case, TCP_NODELAY is enabled and all writes are immediately flushed unless the application clearly indicates it has a queue of data that will immediately follow.

nly 6y ago

In your example sending the file body will fill the remainder of the packet containing the headers and trigger a packet send. The client waiting for your response isn't going to lose out. In fact, they'll benefit from having file body data in that first packet. Your example is when you want Nagles algorithm to apply. This exact use case is mentioned on tcp(7) under the TCP_CORK option, which is like a more explicit, application controlled, version of Nagle. Nagles algorithm sucks for HTTP clients who happen to send big HTTP requests. If the HTTP headers being sent are >1400 bytes or so, then the tail end of the request will wait 1 RTT for the ACK of the first part. This will be particularly bad if the HTTP server has delayed-ACKs enabled (TCP_QUICKACK is not on). The server TCP stack has no idea that the HTTP server application hasn't yet got a full request and can't yet respond, so will delay the ACK of the first 1400 request bytes in the hope that it will. The delayed ACK delay is probably small compared to the RTT, so Nagle still dominates

derefr 6y ago

True, but also kind of irrelevant, in that HTTP1.x is a protocol designed under the constraint that there are going to be an unbounded number of application-level proxies/gateways between the client and the server, where some of them might be under the client’s control (e.g. Tor), some under the server’s (e.g. Varnish), and some neither (e.g. a corporate or ISP WAF.) And because of that, modern web servers don’t really do NODELAY; rather the opposite — they fully buffer HTTP requests and responses into what are essentially single giant datagrams. Put your Express app behind Nginx, and Nginx will buffer the entire request to feed it to your app as one TCP jumbo-frame (if possible); and then buffer your response to feed it back down the line to the client as one TCP jumbo-frame, or a burst of such packets. This happens not just with HEAD responses, but with full-bodied GETs/POSTs/PUTs, too. The only time modern web servers attempt to break your buffer up and schedule sends at different times, is when you tell them that you’re doing a `Transfer-Encoding: Chunked` response; or when your upstream is HTTP/2 and sends down an HTTP/2 stream non-terminal body-frame; or when your upstream is speaking websockets or WebRTC. Otherwise you get this “hyper-Nagling.” (Why do they do it? Because these servers want to achieve sub-worker-thread concurrency, and system calls like write(2) are blocking, getting in the way of that worker doing anything else. So web-server worker threads want to do as few system calls as possible, which means batching up your writes so they can do one context-switch for one big write(2) or writev(2). For these servers, sendfile(2) can actually be an anti-pattern, compared to mmap(2)ing the relevant static files so that they can just be passed as a pointer to writev(2) along with the headers buffer. That’s one blocking context-switch rather than two!)