The TCP protocol needs to be able buffer one round trip time (RTT) worth of data at both sender and receiver in the kernel. It needs one RTT of data at the rec

The TCP protocol needs to be able buffer one round trip time (RTT) worth of data at both sender and receiver in the kernel. It needs one RTT of data at the receiver to cope with waiting for a retransmission of a lost packet, or just so it doesn't drop incoming packets when the receiving app doesn't read immediately. The same amount of data must be buffered at the sender because it may need to be retransmitted if a packet was lost. So, 6MiB of buffering with a fairly typical 100ms RTT gives you 6*8/0.1 Mbit/s = 480Mbit/s. In practice you may need more buffering than this, but that's the maximum rate you can achieve. Now, if you've got an overbuffered router at the bottleneck, as is common on home broadband links, you can build a queue there. It's not really all that unusual to see queues build to 1 second, especially on ADSL. If you've a 1 second RTT, the maximum rate you can support with 6MiB of buffering at both sender and receiver drops to 48Mbit/s. It seems to me that 6MiB is a fairly reasonable default value, though if you're running a really busy server, you might benefit from reducing this, especially if you're mostly serving small files.