3 ms·

By "expose it to the application", I mean in the flow-control sense. That send() (or select/epoll/etc) will block until there is room in the TCP window, rather

by Peaker 14y ago

By "expose it to the application", I mean in the flow-control sense. That send() (or select/epoll/etc) will block until there is room in the TCP window, rather than in the pre-determined buffer which will always be too small or too big. The way it works now, the kernel is basically forcing applications to have buffer-bloat (fill a 0.5MB socket buffer), or auto-detect RTT or manually select a proper buffer size. All are bad options. Also, in high-latency-high-bandwidth situations, the default 0.5MB buffer will simply fail to make use of the available bandwidth, so increasing the buffer size does not defeat the purpose. Latency spikes are a different situation, there are cases of constant high latency (e.g: inter-continental 1Gbps links).

natermer 14y ago
> Also, in high-latency-high-bandwidth situations, the default 0.5MB buffer will simply fail to make use of the available bandwidth, so increasing the buffer size does not defeat the purpose. To find out how much buffer is required you multiply the latency it takes to process the packet by the maximum rate. Say it takes 0.2 msecs to send a packet out on the network then a 1Gbps connection would need a buffer of 0.03 MB or so. So that 0.5MB should be sufficient to saturate a 10Gbps link. It seems that if you are trying to avoid blocking in a application when writing to a socket you should take a different strategy. > Latency spikes are a different situation, there are cases of constant high latency (e.g: inter-continental 1Gbps links). Latency spikes in terms of this discussion can be caused by situations were you have a low speed network connected to a high speed network with a large buffer between them. In this situation the server (or whatever) will write to the buffer on the bridge/router at a maximum amount rate allowed by the network. What should happen is that the router should block causing packet loss which will cause the TCP congestion algorithms to kick in and slow down the rate at which the packets are sent. What actually happens is that the massive buffers fill up and when you have multiple connections competing for buffer space you will all of a sudden hit a brick wall. So what comes up with high latency spikes, weird packet loss patterns, and such things. Small buffers means high performance...

by Peaker 14y ago

natermer 14y ago

> Also, in high-latency-high-bandwidth situations, the default 0.5MB buffer will simply fail to make use of the available bandwidth, so increasing the buffer size does not defeat the purpose. To find out how much buffer is required you multiply the latency it takes to process the packet by the maximum rate. Say it takes 0.2 msecs to send a packet out on the network then a 1Gbps connection would need a buffer of 0.03 MB or so. So that 0.5MB should be sufficient to saturate a 10Gbps link. It seems that if you are trying to avoid blocking in a application when writing to a socket you should take a different strategy. > Latency spikes are a different situation, there are cases of constant high latency (e.g: inter-continental 1Gbps links). Latency spikes in terms of this discussion can be caused by situations were you have a low speed network connected to a high speed network with a large buffer between them. In this situation the server (or whatever) will write to the buffer on the bridge/router at a maximum amount rate allowed by the network. What should happen is that the router should block causing packet loss which will cause the TCP congestion algorithms to kick in and slow down the rate at which the packets are sent. What actually happens is that the massive buffers fill up and when you have multiple connections competing for buffer space you will all of a sudden hit a brick wall. So what comes up with high latency spikes, weird packet loss patterns, and such things. Small buffers means high performance...