4 ms·

Yeah but Nagle's Algorithm and Delayed ACKs interaction is what causes the 200ms. Servers tend to enable Nagle's algorithm by default. Clients tend to enabled

Yeah but Nagle's Algorithm and Delayed ACKs interaction is what causes the 200ms. Servers tend to enable Nagle's algorithm by default. Clients tend to enabled Delayed ACK by default, and then you get this horrible interaction all because they're trying to be more efficient but stalling eachother. I think Go's behavior is the right default because you can't control every server. But if Nagle's was off by default on servers then we wouldn't need to disabled Delayed ACKs on clients.

pclmulqdq 4y ago
Clients having delayed acks has a very good reason: those ACKs cost data, and clients tend to have much higher download bandwidth than upload bandwidth. Really, clients should probably be delaying acks and nagling packets, while servers should probably be doing neither.
- charleslmunger 4y ago
  Clients should not be nagling unless the connection is emitting tiny bytes at high frequency. But that's a very odd thing to do, and in most/all cases there's some reasonable buffering occuring higher up in the stack that the nagle's algorithm will only add overhead to. Making things worse are tcp-within-tcp things like http/2. Nagle's algorithm works great for things like telnet but should not be applied as a default to general purpose networking.
  - avianlyric 4y ago
    Why would Nagles algorithm add delay to “reasonable buffering up the stack”? Assuming that buffering is resulting in writes to the TCP stack greater than the packet size, Nagles algorithm won’t add any delay. The only place where Nagles algorithm adds delay is when your doing many tiny writes to a socket, which is exactly the situation you believe Nagles should be applied to.
- avianlyric 4y ago
  The size of an ACK is minuscule (40 bytes) compared to any reasonable packet size (usually around 1400 bytes). In most client situations where you have high down bandwidth, but limited up, that suggests the vast majority of data is heading towards the client, and client isn’t sending much outbound. In which case your client may end up delaying every ACK to maximum timeout, simply because it doesn’t often send reply data in response to a server response. HTTP is clear example of this. Client issues a request to the server, server replies. Client accepts rely, but never sends any further data to the server. In this case, delaying the client ACK is just a waste of time.
Terretta 4y ago
Part of OPs point is 'most clients' do not have an ideal congestionless/lossless network between them and, well, anything.
- fragmede 4y ago
  Most clients that OP deals with, anyway. If your code runs exclusively in a data center, like the kind I suspect Google has, then the situation is probably reversed.
  - vitus 4y ago
    If you run all of your code in one datacenter, and it never talks to the outside world, sure. That is a fairly rare usage pattern for production systems at Google, though. Just like anyone else, we have packet drops and congestion within our backbone. We like to tell ourselves that the above is less frequent in our network than the wider internet, but it still exists.
    - oefrha 4y ago
      If your DC-DC links are regularly as noisy as shitty apartment WiFi routers competing for air time on a narrow band, fix your DC links.
  - mmis1000 4y ago
    Consider the rising of mobile device. The devices that don't have a good internet is probably everywhere now. It's no longer like 10 years ago. You either have good internet or don't have internet. The devices that have shitty network grow a lot compare to the past.
    - Karrot_Kream 4y ago
      Almost every application I've written atop a TCP socket batches up writes into a buffer and then flushes out the buffer. I'd be curious to see how often this doesn't happen.
dccoolgai 4y ago
"Be conservative in what you send and liberal in what you accept" I would cite Postels Law: Nagle's is the "conservative send" side. An ACK is a signal of acceptance, and should be issued more liberally (even though it's also sent, I guess).

pclmulqdq 4y ago

Clients having delayed acks has a very good reason: those ACKs cost data, and clients tend to have much higher download bandwidth than upload bandwidth. Really, clients should probably be delaying acks and nagling packets, while servers should probably be doing neither.

charleslmunger 4y ago
Clients should not be nagling unless the connection is emitting tiny bytes at high frequency. But that's a very odd thing to do, and in most/all cases there's some reasonable buffering occuring higher up in the stack that the nagle's algorithm will only add overhead to. Making things worse are tcp-within-tcp things like http/2. Nagle's algorithm works great for things like telnet but should not be applied as a default to general purpose networking.
- avianlyric 4y ago
  Why would Nagles algorithm add delay to “reasonable buffering up the stack”? Assuming that buffering is resulting in writes to the TCP stack greater than the packet size, Nagles algorithm won’t add any delay. The only place where Nagles algorithm adds delay is when your doing many tiny writes to a socket, which is exactly the situation you believe Nagles should be applied to.
avianlyric 4y ago
The size of an ACK is minuscule (40 bytes) compared to any reasonable packet size (usually around 1400 bytes). In most client situations where you have high down bandwidth, but limited up, that suggests the vast majority of data is heading towards the client, and client isn’t sending much outbound. In which case your client may end up delaying every ACK to maximum timeout, simply because it doesn’t often send reply data in response to a server response. HTTP is clear example of this. Client issues a request to the server, server replies. Client accepts rely, but never sends any further data to the server. In this case, delaying the client ACK is just a waste of time.

Terretta 4y ago

Part of OPs point is 'most clients' do not have an ideal congestionless/lossless network between them and, well, anything.

fragmede 4y ago
Most clients that OP deals with, anyway. If your code runs exclusively in a data center, like the kind I suspect Google has, then the situation is probably reversed.
- vitus 4y ago
  If you run all of your code in one datacenter, and it never talks to the outside world, sure. That is a fairly rare usage pattern for production systems at Google, though. Just like anyone else, we have packet drops and congestion within our backbone. We like to tell ourselves that the above is less frequent in our network than the wider internet, but it still exists.
  - oefrha 4y ago
    If your DC-DC links are regularly as noisy as shitty apartment WiFi routers competing for air time on a narrow band, fix your DC links.
- mmis1000 4y ago
  Consider the rising of mobile device. The devices that don't have a good internet is probably everywhere now. It's no longer like 10 years ago. You either have good internet or don't have internet. The devices that have shitty network grow a lot compare to the past.
  - Karrot_Kream 4y ago
    Almost every application I've written atop a TCP socket batches up writes into a buffer and then flushes out the buffer. I'd be curious to see how often this doesn't happen.

dccoolgai 4y ago

"Be conservative in what you send and liberal in what you accept" I would cite Postels Law: Nagle's is the "conservative send" side. An ACK is a signal of acceptance, and should be issued more liberally (even though it's also sent, I guess).