4 ms·

SPDY performance on mobile networks

32 points by Garbage 14y ago

chrisacky 14y ago
Incidentally, for anyone interested in support for nginx, it should hopefully be available by the end of this month. https://twitter.com/#!/nginxorg/status/192301063934705665 https://twitter.com/#!/nginxorg/status/192301063934705665
jsnell 14y ago
Measuring the speed of web page loading on mobile networks is something that we wrestle with weekly (we make a TCP accelerator product for mobile operators), so I always love reading about new approaches and results. Unfortunately this particular one seems to have particularly dodgy methodology. Most importantly, they did in fact not measure anything on mobile networks or even simulations of mobile networks. Instead it was a just a crudely throttled wired connection, which is a poor approximation for how a 3G network behaves. The right way to deal with changing network conditions is to actually do lots of measurements. This is particularly problematic since one effect of parallel HTTP connections is to isolate the congestion control effect of a random network event to just one of the 6 connections to a server. Second, they are not fetching the actual web pages, but getting all the copies from a local cache. This totally removes the point of testing against multiple web sites. In the real world there's huge variability in things like web servers and TCP stacks / options. Also, if I understand the replay tool correctly, it's not accounting for the network latency on the inet side of the test setup at all. I'm not completely sure whether this is benefiting SPDY or HTTP though (on one hand SPDY suffers from the latency at the start since it can't accelerate the single flow as quickly during slow start, while HTTP suffers from the latency). But it certainly removes the very common effect of there being a one or two outlier resources being loaded from slow servers. Removing DNS overhead completely is a bit strange too, and would have the effect of magnifying any relative speedups / slowdowns.
- jgrahamc 14y ago
  I would be interested in talking to you about 3G TCP performance and CloudFlare. If you are interested my email address is in my profile.
  - mp3geek 14y ago
    When is cloudflare pushing out spdy for SSL users? Or can I beta test it? :)
    - jgrahamc 14y ago
      CloudFlare is built on nginx at the front end. When nginx supports SPDY (soon!) we'll being rolling it out. We've been in very close contact with the nginx guys to make sure that SPDY happens.
- zokier 14y ago
  > Second, they are not fetching the actual web pages, but getting all the copies from a local cache. This totally removes the point of testing against multiple web sites. Disagree. Different websites have different structures of content. eg some sites may rely heavily on external resources, while another has everything inlined, etc.
  - jsnell 14y ago
    Fair enough. Let's just say that this method removes some major sources of real-world variation between sites.
- mdwelsh 14y ago
  Hi Juho, I'm the guy who did this study at Google. I agree with your criticisms that the measurement setup does not reflect real network conditions in all cases. However, we have done a tremendous amount of measurement on real mobile networks as well, and settled on this approach after a lot of iterating and tuning to come up with what (we believe to be) a fairly realistic setup that - most importantly - achieves repeatable results. This was a deliberate tradeoff. We have measured mobile networks all over the world and observe that the network variation across space and time is incredibly high. It's pretty much meaningless to talk about "real mobile network conditions", since the conditions you measure in an office in Seattle are so vastly different than what you get on a train in Finland or a highrise in New York. The simple throttling we did does, in fact, accurately reflect HTTP and SPDY load times for those specific network conditions, which were meant to be representative of major carriers in the US. You're right that we did not account for network latency to the origin server, but I would not expect that to skew the results in favor of either protocol, since the last mile is the slow link in this case. The DNS overhead was an artifact of the test setup, but HTTP and SPDY should have the same number of required DNS lookups in both cases. Feel free to drop me a line (my contact info is on the original article)!
zokier 14y ago
I don't see the point of using a phone for measurements if the phone is tethered to computer. Also more interesting research would be measuring performance improvement vs degradation of network.
- mdwelsh 14y ago
  We wanted to use a real phone to accurately capture the performance of the phone hardware, browser (Chrome), and OS (Android). For example, JavaScript runs much more slowly on most mobile platforms than it does on desktop browsers, and we wanted to ensure our results were not skewed by that.

32 points by Garbage 14y ago

Incidentally, for anyone interested in support for nginx, it should hopefully be available by the end of this month. https://twitter.com/#!/nginxorg/status/192301063934705665 https://twitter.com/#!/nginxorg/status/192301063934705665

jsnell 14y ago

Measuring the speed of web page loading on mobile networks is something that we wrestle with weekly (we make a TCP accelerator product for mobile operators), so I always love reading about new approaches and results. Unfortunately this particular one seems to have particularly dodgy methodology. Most importantly, they did in fact not measure anything on mobile networks or even simulations of mobile networks. Instead it was a just a crudely throttled wired connection, which is a poor approximation for how a 3G network behaves. The right way to deal with changing network conditions is to actually do lots of measurements. This is particularly problematic since one effect of parallel HTTP connections is to isolate the congestion control effect of a random network event to just one of the 6 connections to a server. Second, they are not fetching the actual web pages, but getting all the copies from a local cache. This totally removes the point of testing against multiple web sites. In the real world there's huge variability in things like web servers and TCP stacks / options. Also, if I understand the replay tool correctly, it's not accounting for the network latency on the inet side of the test setup at all. I'm not completely sure whether this is benefiting SPDY or HTTP though (on one hand SPDY suffers from the latency at the start since it can't accelerate the single flow as quickly during slow start, while HTTP suffers from the latency). But it certainly removes the very common effect of there being a one or two outlier resources being loaded from slow servers. Removing DNS overhead completely is a bit strange too, and would have the effect of magnifying any relative speedups / slowdowns.

jgrahamc 14y ago
I would be interested in talking to you about 3G TCP performance and CloudFlare. If you are interested my email address is in my profile.
- mp3geek 14y ago
  When is cloudflare pushing out spdy for SSL users? Or can I beta test it? :)
  - jgrahamc 14y ago
    CloudFlare is built on nginx at the front end. When nginx supports SPDY (soon!) we'll being rolling it out. We've been in very close contact with the nginx guys to make sure that SPDY happens.
zokier 14y ago
> Second, they are not fetching the actual web pages, but getting all the copies from a local cache. This totally removes the point of testing against multiple web sites. Disagree. Different websites have different structures of content. eg some sites may rely heavily on external resources, while another has everything inlined, etc.
- jsnell 14y ago
  Fair enough. Let's just say that this method removes some major sources of real-world variation between sites.
mdwelsh 14y ago
Hi Juho, I'm the guy who did this study at Google. I agree with your criticisms that the measurement setup does not reflect real network conditions in all cases. However, we have done a tremendous amount of measurement on real mobile networks as well, and settled on this approach after a lot of iterating and tuning to come up with what (we believe to be) a fairly realistic setup that - most importantly - achieves repeatable results. This was a deliberate tradeoff. We have measured mobile networks all over the world and observe that the network variation across space and time is incredibly high. It's pretty much meaningless to talk about "real mobile network conditions", since the conditions you measure in an office in Seattle are so vastly different than what you get on a train in Finland or a highrise in New York. The simple throttling we did does, in fact, accurately reflect HTTP and SPDY load times for those specific network conditions, which were meant to be representative of major carriers in the US. You're right that we did not account for network latency to the origin server, but I would not expect that to skew the results in favor of either protocol, since the last mile is the slow link in this case. The DNS overhead was an artifact of the test setup, but HTTP and SPDY should have the same number of required DNS lookups in both cases. Feel free to drop me a line (my contact info is on the original article)!

zokier 14y ago

I don't see the point of using a phone for measurements if the phone is tethered to computer. Also more interesting research would be measuring performance improvement vs degradation of network.

mdwelsh 14y ago
We wanted to use a real phone to accurately capture the performance of the phone hardware, browser (Chrome), and OS (Android). For example, JavaScript runs much more slowly on most mobile platforms than it does on desktop browsers, and we wanted to ensure our results were not skewed by that.