3 ms·

> Setting the rmem sysctl to only 2MiB is not recommended as it could affect the performance of high throughput, high latency connections So what I took away f

by _wmd 11y ago

> Setting the rmem sysctl to only 2MiB is not recommended as it could affect the performance of high throughput, high latency connections So what I took away from this article is that they'd set some absurdly high receive buffer, which predictably caused something to break, and never quite got around to explaining how they arrived at the original number. That buffer is for data the kernel has received but has not yet been read() by userspace. Seriously, 5MB of RAM dedicated to each socket in the kernel? Assuming these are Nginx proxy connections, why did you choose to tweak a kernel tunable rather than increasing the corresponding userspace application buffer sizes instead? edit: so the answer seems to be that Nginx is wholly reliant on the kernel to handle buffering, which is completely asinine to me, but perhaps there is good rationale here I've somehow missed. The take home becomes "our app stack is so broken we managed to break additional extreme-battle-tested components in a bid to to get it to function sensibly"

creshal 11y ago
> Seriously, 5MB of RAM dedicated to each socket in the kernel? Debian 8 seems to have a default of 6 MiB. So, the "absurdly high receive buffer" might not be cloudflare's fault.
- mhandley 11y ago
  The TCP protocol needs to be able buffer one round trip time (RTT) worth of data at both sender and receiver in the kernel. It needs one RTT of data at the receiver to cope with waiting for a retransmission of a lost packet, or just so it doesn't drop incoming packets when the receiving app doesn't read immediately. The same amount of data must be buffered at the sender because it may need to be retransmitted if a packet was lost. So, 6MiB of buffering with a fairly typical 100ms RTT gives you 6*8/0.1 Mbit/s = 480Mbit/s. In practice you may need more buffering than this, but that's the maximum rate you can achieve. Now, if you've got an overbuffered router at the bottleneck, as is common on home broadband links, you can build a queue there. It's not really all that unusual to see queues build to 1 second, especially on ADSL. If you've a 1 second RTT, the maximum rate you can support with 6MiB of buffering at both sender and receiver drops to 48Mbit/s. It seems to me that 6MiB is a fairly reasonable default value, though if you're running a really busy server, you might benefit from reducing this, especially if you're mostly serving small files.
noselasd 11y ago
Note that these buffers are not only relevant for unread data of the user space application, they also reflect the TCP window size - this means the sending side knows how much room the receiver has at all times , and can throw that much data at it without receiving a ACK - so there's a fine dance between network speed , latency, and this buffer size.
anewhnaccount 11y ago
Well one rationale could be that if nginx did buffering too, you're essentially doing buffering twice. Doing something once is often faster than doing it twice.

by _wmd 11y ago

creshal 11y ago

> Seriously, 5MB of RAM dedicated to each socket in the kernel? Debian 8 seems to have a default of 6 MiB. So, the "absurdly high receive buffer" might not be cloudflare's fault.

mhandley 11y ago
The TCP protocol needs to be able buffer one round trip time (RTT) worth of data at both sender and receiver in the kernel. It needs one RTT of data at the receiver to cope with waiting for a retransmission of a lost packet, or just so it doesn't drop incoming packets when the receiving app doesn't read immediately. The same amount of data must be buffered at the sender because it may need to be retransmitted if a packet was lost. So, 6MiB of buffering with a fairly typical 100ms RTT gives you 6*8/0.1 Mbit/s = 480Mbit/s. In practice you may need more buffering than this, but that's the maximum rate you can achieve. Now, if you've got an overbuffered router at the bottleneck, as is common on home broadband links, you can build a queue there. It's not really all that unusual to see queues build to 1 second, especially on ADSL. If you've a 1 second RTT, the maximum rate you can support with 6MiB of buffering at both sender and receiver drops to 48Mbit/s. It seems to me that 6MiB is a fairly reasonable default value, though if you're running a really busy server, you might benefit from reducing this, especially if you're mostly serving small files.

noselasd 11y ago

Note that these buffers are not only relevant for unread data of the user space application, they also reflect the TCP window size - this means the sending side knows how much room the receiver has at all times , and can throw that much data at it without receiving a ACK - so there's a fine dance between network speed , latency, and this buffer size.

anewhnaccount 11y ago

Well one rationale could be that if nginx did buffering too, you're essentially doing buffering twice. Doing something once is often faster than doing it twice.