4 ms·

Hack initcwnd for faster page loading

44 points by sajal83 15y ago

wazoox 15y ago
What I'm wondering is if that could have some influence on other protocols, too, like NFS. I'm going to do some testing.
- sajal83 15y ago
  I think NFS usually uses UDP by default. Even when using TCP, I think increasing the initcwnd/initrwnd will only benefit the first few transfers after mounting. Would be curious to what you find out.
  - wazoox 15y ago
    I did some initial TCP tests running netperf on a single GigE line, and there are some tenuous gains. I'll do NFS over TCP tests with 10 GigE or IB soon to get some meaningful numbers.
ck2 15y ago
We went through this last week on HN but this article is far more detailed. Thanks.
necro 15y ago
Great writeup. Have been hacking initcwd to way back when it was just hardcoded in the kernel. Other parameters not mentioned that control congestion: 1. net.ipv4.tcp_slow_start_after_idle By default this is set to 1, which means, in a longer keep-alive connection, your connection will revert back to slow start (whatever you ended up setting it as per article) after "idle" (idle is 2 seconds last time i checked). Of course in a keep-alive connection you may have opened up your window from slow start to max, and you probably dont want it do default back to initcwd on subsequent requests. Especially applies on static file/cdn type servers where you probably are running longer keep-alives. You can just set this in sysctl. 2. the last time i checked (years ago) if you run conntrack and turn off tcp_no_metrics_save, you get the benefit of the window opening up to the last remembered window size from your previous connection. The concepts on increasing initcwd are particularly advantageous in the mobile arena where the rtt is usually much higher.
- sajal83 15y ago
  I am somewhat skeptical about the benefits to mobile. the RTT is higher, so lower round-trips is good, but isint the risk of congestion higher in mobile? Say someone with a poor signal?
  - necro 15y ago
    Perhaps this then... if ($network=='at&t') { $initcwd=1; } else { $initcwd=10; } I think a reasonable increase in initcwd for mobile makes sense. I imagine most mobile pages are smaller too. The way I look at it, I want to increase initcwd to fit the page size in it.
  - saurik 15y ago
    Good direction, but bad conclusion: "poor signal" is not the same thing as "congestion". This fact is what causes TCP over any form of wireless (including high-speed WiFi) to have been a really bad decision (and one that academic researchers the world over love trying to solve ;P). The reason is that TCP makes an assumption (almost entirely true in the case of physical cable) that "packet loss" is always caused by "congestion" (due to the router simply throwing away packets it cannot handle), and so if you are losing packets you need to slow down your rate of sending, to help more of them get through. However, if the reason you are losing packets is "poor signal", then you are going to get a pretty much constant rate of packet loss: if 10% of your airwaves are "damaged", 10% of your packets are going to be lost whether you are sending them at 100k/s or 1b/s (as your choice of "when to send" is entirely random from the perspective of the interference: your device is not modeling the underlying issue so it can try to "dodge the raindrops"; people work on this, though ;P). The result is that when your signal is bad, even if you are currently getting 90k/s out after 10% loss, your standard TCP stack goes "oh shit" and starts dialing down its transmit window to "virtually nothing", leaving you unable to use that connection to get any data through anymore; even just opening a new connection when this happens (which may require restarting the app) tends to help tremendously. (The solutions, for the record, all involve some sort of mechanism that allows explicit bookkeeping of either packet loss or congestion. However, this explicit bookkeeping requires various players that are not the device to be "in on it", and therefore makes it really hard to find the right balance of "who needs to install better routers to solve the problem", given the wide deployment of existing TCP... I mean, even getting IPv6 out there has been a tough sell, replacing IP entirely is a pipe-dream right now.)
perssontm 15y ago
This is really interesting, but it seems like its the same information floating around on different blogs. If I didnt know better I'd assume they copy each other regarding this. All lack some real and more thorough testing, they include that simple example, perhaps its enough, but its a bit light on the facts to get me to try it. Would be interesting to know how this works in virtual environments, can I set this inside a VPS, or would the main kernel in the dom0(xen-host-stuff) override this? Are there any sideeffects?
- sajal83 15y ago
  id be posting some stats soon about the RWIN of traffic to our site. i.e. what % of users have a high enough RWIN to benefit from it. The setting works as expected on EC2 instances, i don't see why this would not work on dom0 as well. It doesn't need a kernel recompile, its just settings for your networking stack.

44 points by sajal83 15y ago

What I'm wondering is if that could have some influence on other protocols, too, like NFS. I'm going to do some testing.

sajal83 15y ago
I think NFS usually uses UDP by default. Even when using TCP, I think increasing the initcwnd/initrwnd will only benefit the first few transfers after mounting. Would be curious to what you find out.
- wazoox 15y ago
  I did some initial TCP tests running netperf on a single GigE line, and there are some tenuous gains. I'll do NFS over TCP tests with 10 GigE or IB soon to get some meaningful numbers.

ck2 15y ago

We went through this last week on HN but this article is far more detailed. Thanks.

necro 15y ago

Great writeup. Have been hacking initcwd to way back when it was just hardcoded in the kernel. Other parameters not mentioned that control congestion: 1. net.ipv4.tcp_slow_start_after_idle By default this is set to 1, which means, in a longer keep-alive connection, your connection will revert back to slow start (whatever you ended up setting it as per article) after "idle" (idle is 2 seconds last time i checked). Of course in a keep-alive connection you may have opened up your window from slow start to max, and you probably dont want it do default back to initcwd on subsequent requests. Especially applies on static file/cdn type servers where you probably are running longer keep-alives. You can just set this in sysctl. 2. the last time i checked (years ago) if you run conntrack and turn off tcp_no_metrics_save, you get the benefit of the window opening up to the last remembered window size from your previous connection. The concepts on increasing initcwd are particularly advantageous in the mobile arena where the rtt is usually much higher.

sajal83 15y ago
I am somewhat skeptical about the benefits to mobile. the RTT is higher, so lower round-trips is good, but isint the risk of congestion higher in mobile? Say someone with a poor signal?
- necro 15y ago
  Perhaps this then... if ($network=='at&t') { $initcwd=1; } else { $initcwd=10; } I think a reasonable increase in initcwd for mobile makes sense. I imagine most mobile pages are smaller too. The way I look at it, I want to increase initcwd to fit the page size in it.
- saurik 15y ago
  Good direction, but bad conclusion: "poor signal" is not the same thing as "congestion". This fact is what causes TCP over any form of wireless (including high-speed WiFi) to have been a really bad decision (and one that academic researchers the world over love trying to solve ;P). The reason is that TCP makes an assumption (almost entirely true in the case of physical cable) that "packet loss" is always caused by "congestion" (due to the router simply throwing away packets it cannot handle), and so if you are losing packets you need to slow down your rate of sending, to help more of them get through. However, if the reason you are losing packets is "poor signal", then you are going to get a pretty much constant rate of packet loss: if 10% of your airwaves are "damaged", 10% of your packets are going to be lost whether you are sending them at 100k/s or 1b/s (as your choice of "when to send" is entirely random from the perspective of the interference: your device is not modeling the underlying issue so it can try to "dodge the raindrops"; people work on this, though ;P). The result is that when your signal is bad, even if you are currently getting 90k/s out after 10% loss, your standard TCP stack goes "oh shit" and starts dialing down its transmit window to "virtually nothing", leaving you unable to use that connection to get any data through anymore; even just opening a new connection when this happens (which may require restarting the app) tends to help tremendously. (The solutions, for the record, all involve some sort of mechanism that allows explicit bookkeeping of either packet loss or congestion. However, this explicit bookkeeping requires various players that are not the device to be "in on it", and therefore makes it really hard to find the right balance of "who needs to install better routers to solve the problem", given the wide deployment of existing TCP... I mean, even getting IPv6 out there has been a tough sell, replacing IP entirely is a pipe-dream right now.)

perssontm 15y ago

This is really interesting, but it seems like its the same information floating around on different blogs. If I didnt know better I'd assume they copy each other regarding this. All lack some real and more thorough testing, they include that simple example, perhaps its enough, but its a bit light on the facts to get me to try it. Would be interesting to know how this works in virtual environments, can I set this inside a VPS, or would the main kernel in the dom0(xen-host-stuff) override this? Are there any sideeffects?

sajal83 15y ago
id be posting some stats soon about the RWIN of traffic to our site. i.e. what % of users have a high enough RWIN to benefit from it. The setting works as expected on EC2 instances, i don't see why this would not work on dom0 as well. It doesn't need a kernel recompile, its just settings for your networking stack.