5 ms·

Measuring QUIC vs TCP on mobile and desktop

68 points by josephscott 9y ago

minikites 9y ago
>Google has the largest QUIC deployment, and has reported that more than 85% of requests from Chrome browsers to Google servers (about 90% of Chrome bytes received) are now using QUIC, which accounts for 7% of Internet traffic! >Taking a closer look at the congestion window changes (Figure 6), we found that while both protocols use Cubic, QUIC increases its window more aggressively (both in terms of slope, and in terms of more frequent window size increases). As a result, QUIC is able to grab available bandwidth faster than TCP does, leaving TCP unable to acquire its fair share of the bandwidth. First AMP and now QUIC. As I see it, Google is emerging as the biggest enemy of both the web and the underlying internet.
- matthewmacleod 9y ago
  I'm not sure it's all that bad. AMP is awful, particularly because of the way in which Google abuses it, but QUIC is less of a problem – it's on an IETF standards track with a couple of implementations.
- halfteatree 9y ago
  > First AMP and now QUIC. Um, pretty sure QUIC was in the pipeline a few years longer than AMP. Besides, it's literally a protocol in the network stack -- would you say IPv6 is an enemy of the web and underlying internet? Please stop the needless fear mongering.
  - minikites 9y ago
    I pasted that initial quote ("Google has the largest QUIC deployment...") to point out that Google has a significant advantage in owning the most popular web browser and also most of the top-tier web properties in Google search, Gmail, and YouTube. Because they control the whole chain from browser to content, they can release things that make those experiences better for Chrome users (using QUIC) at the expense of non-Chrome users (who get worse TCP performance until everyone else completely adopts QUIC). Then, toss up some banner ads for non-Chrome users who visit YouTube saying, "YouTube too slow? Switch to Chrome!" >would you say IPv6 is an enemy of the web and underlying internet? If IPv6 made IPv4 performance worse and was pushed by one company who controls both the browser and the most popular web content, yes, I would.
    - wmf 9y ago
      Everything you say is true, yet... the Google approach works and the others don't. If our choice is free and mostly-open protocols from Google or using HTTP/1.1 + TLS 1.1 + TCP w/ CUBIC + IPv4 w/ NAT forever, Google doesn't sound too bad. This is related to the AV1 vs. MPEG discussion from the other day where people pointed out that we basically have a a choice between free codecs developed by Google/Netflix or codecs that suck you dry from legal fees.
  - gsich 9y ago
    Do you even know the OSI-model? Yes QUIC is a protocol, but not on the same layer as IPv6. So the comparison is useless.
    - halfteatree 9y ago
      It is one layer above IPv6, but what difference does that make? It's an improvement over an existing protocol to solve real problems. I don't think anyone can really know the OSI model. I know I frequently use HTTP as a Transport Layer protocol myself, so you got me there. ¯\_(ツ)_/¯
- trevyn 9y ago
  This has nothing to do with Google's politics. TCP is broken for modern networks, and the faster we ditch it for a protocol capable of reliable pipes with low latency and high throughput, the better off we'll all be.
  - minikites 9y ago
    Not all networks are modern, what about the unreliable networks in most of the developing world?
    - trevyn 9y ago
      QUIC still works better than TCP on these networks, but the benefits are not as dramatic.
      - tybit 9y ago
        That doesn't seem to be supported by the article? "While it outperforms TCP in a wide range of scenarios, we found that it underperforms in the presence of packet reordering and on resource-constrained mobile devices."
        trevyn 9y ago
        1) "We shared this result with a QUIC engineer, who subsequently informed us that the QUIC team is experimenting with dynamic threshold and time-based solutions to avoid falsely inferring loss in the presence of reordering." 2) "We find that, similar to the desktop environment, in mobile QUIC outperforms TCP in most cases; however, its advantages diminish across the board... The reason for this behavior is that QUIC runs in a userspace process, whereas TCP runs in the kernel." (i.e. putting it in the kernel should help.) https://conferences.sigcomm.org/imc/2017/papers/imc17-final39.pdf https://conferences.sigcomm.org/imc/2017/papers/imc17-final3...
_wmd 9y ago
Am I understanding things correctly? Because QUIC ramps up its bandwidth estimate more aggressively, it fundamentally competes unfairly with TCP? Is that an inherent property or something that can be fixed? Definitely it's too late to fix it on the TCP side Nice to know 7% of Internet traffic isn't playing fair with the rest just so one company's content loads a few ms faster! Really don't know how I feel about TCP in general losing out to QUIC if it saw much wider deployment outside Google And finally, it's incredibly disappointing to read these results from a third party, rather than having a balanced perspective as part of the original marketing
- trevyn 9y ago
  QUIC fundamentally competes on a different level because TCP is broken for modern networks. From a network usage standpoint, there isn't anything "unfair" about it -- QUIC runs atop UDP, which is also a perfectly acceptable and publicly available Internet protocol. And Chrome will happily use QUIC with non-Google hosts, the real problem is that production-quality QUIC server software is currently rare outside of Google.
  - _wmd 9y ago
    One of the major downsides to UDP is that it lacks any form of congestion control, so it's hard to justify QUIC's behaviour due to what it's built atop :) Meanwhile we're not talking about a small slice of bandwidth spent on SIP video calls playing unfairly, we're talking about 70% or more of all Internet traffic suddenly stampeding over every pre-existing protocol, they're the floodgates I'm worried about
    - trevyn 9y ago
      If this was actually a problem, it would be trivial for network operators to prioritize TCP packets over UDP packets. QUIC builds modern congestion control on top of UDP.
      - solotronics 9y ago
mholt 9y ago
If you want to try QUIC yourself, you can use Caddy, which has experimental QUIC support (https://caddyserver.com/docs/cli#quic https://caddyserver.com/docs/cli#quic) powered by quic-go. (However, QUIC versions are extremely transient and has very limited support in Chrome; the next release of Caddy will bring it back up-to-date again.)
newman314 9y ago
I'm still waiting for QUIC support for nginx.
twouhm 9y ago
I wonder how QUIC compares to KCP[0]. [0]: https://github.com/skywind3000/kcp/blob/master/README.en.md https://github.com/skywind3000/kcp/blob/master/README.en.md
deleted 9y ago
[deleted]
fulafel 9y ago
What's the current positioning of QUIC by Google? It predates HTTP/2, and Google hasn't been making noises about it lately. Is it being slowly phased out or just in a "let it be" status for now in Chrome?
- apenwarr 9y ago
  It's being actively discussed at the IETF. Here's a doc that's last updated 2018-01-28: https://datatracker.ietf.org/doc/draft-ietf-quic-transport/ https://datatracker.ietf.org/doc/draft-ietf-quic-transport/
- trevyn 9y ago
  QUIC replaces TCP and is on IETF standards track. HTTP/2 replaces HTTP/1.1 (and is a direct descendent of SPDY, which has origins ~2009). HTTP/2 on TCP gets you some of the benefits that QUIC provides (multiplexing, reduced roundtrips), but HTTP/2 on QUIC is the best of both worlds. IMO, there's not much noise about QUIC mainly because most people don't care enough about latency to care, and the current lack of production-quality QUIC server software makes caring more complicated than "just enable it", so low demand all around. This should pick up when it gets closer to a ratified standard.
apenwarr 9y ago
This is a pretty disappointing article. It's really comparing a particular implementation (or maybe multiple implementations) of TCP congestion control, implemented in kernel space, with a particular implementation of QUIC congestion control, implemented in user space. The article points out that both the QUIC and TCP implementations they tested are using CUBIC congestion control, but that's not enough information, because the article also points out that QUIC is using "more aggressive parameters." It's tough to say which parameters are better, but what's unsaid is that a TCP implementation could change their parameters and get the same congestion control results as QUIC. The supposed poor performance of QUIC on resource-limited mobile devices is, as they point out, because it's a user space implementation that is thus more expensive. If QUIC becomes popular, I assume there will be kernel implementations that are as resource efficient as TCP. Meanwhile, it's a lot easier to to experiments (such as tuning CUBIC parameters!) when you don't have to reboot to install a new version. It's also quite common for various TCP congestion controllers to completely fail to saturate busy links, because of various limitations. If that happens, it might be that QUIC is able to fill the empty space, thereby taking "more than its fair share" only because TCP wasn't going to use that share anyway. Since the article doesn't even say which TCP implementation it's comparing against, and doesn't say what happens when the TCP sessions are competing only amongst themselves with no QUIC present, it's hard to say what's going on. The funny thing about all this is that, if they use the same congestion control (which seems to be the intention given that they are both using CUBIC and Google is separately trying to fix congestion control via BBR[1]), they should both be about equally fair. The performance benefits of QUIC are not even congestion control related! [1] https://queue.acm.org/detail.cfm?id=3022184 https://queue.acm.org/detail.cfm?id=3022184 [Disclaimer: I've worked with some of the people who wrote BBR and QUIC, so I'm biased.]
- chemag 9y ago
  +1 to this. QUIC allows using CUBIC BBR [1], so a comparison based on the exact parameters used is actually comparing the exact parameters used. The performance effects of QUIC implementing congestion control in userland are more interesting. OTOH, QUIC allows deploying new features to users (through cronet) in an efficient way. TCP does not. [1] https://chromium.googlesource.com/chromium/src/net/+/master/quic/core/congestion_control/bbr_sender.cc https://chromium.googlesource.com/chromium/src/net/+/master/... > [Disclaimer: I've worked with some of the people who wrote BBR and QUIC, so I'm biased.] Ditto

68 points by josephscott 9y ago

>Google has the largest QUIC deployment, and has reported that more than 85% of requests from Chrome browsers to Google servers (about 90% of Chrome bytes received) are now using QUIC, which accounts for 7% of Internet traffic! >Taking a closer look at the congestion window changes (Figure 6), we found that while both protocols use Cubic, QUIC increases its window more aggressively (both in terms of slope, and in terms of more frequent window size increases). As a result, QUIC is able to grab available bandwidth faster than TCP does, leaving TCP unable to acquire its fair share of the bandwidth. First AMP and now QUIC. As I see it, Google is emerging as the biggest enemy of both the web and the underlying internet.

matthewmacleod 9y ago
I'm not sure it's all that bad. AMP is awful, particularly because of the way in which Google abuses it, but QUIC is less of a problem – it's on an IETF standards track with a couple of implementations.
halfteatree 9y ago
> First AMP and now QUIC. Um, pretty sure QUIC was in the pipeline a few years longer than AMP. Besides, it's literally a protocol in the network stack -- would you say IPv6 is an enemy of the web and underlying internet? Please stop the needless fear mongering.
- minikites 9y ago
  I pasted that initial quote ("Google has the largest QUIC deployment...") to point out that Google has a significant advantage in owning the most popular web browser and also most of the top-tier web properties in Google search, Gmail, and YouTube. Because they control the whole chain from browser to content, they can release things that make those experiences better for Chrome users (using QUIC) at the expense of non-Chrome users (who get worse TCP performance until everyone else completely adopts QUIC). Then, toss up some banner ads for non-Chrome users who visit YouTube saying, "YouTube too slow? Switch to Chrome!" >would you say IPv6 is an enemy of the web and underlying internet? If IPv6 made IPv4 performance worse and was pushed by one company who controls both the browser and the most popular web content, yes, I would.
  - wmf 9y ago
    Everything you say is true, yet... the Google approach works and the others don't. If our choice is free and mostly-open protocols from Google or using HTTP/1.1 + TLS 1.1 + TCP w/ CUBIC + IPv4 w/ NAT forever, Google doesn't sound too bad. This is related to the AV1 vs. MPEG discussion from the other day where people pointed out that we basically have a a choice between free codecs developed by Google/Netflix or codecs that suck you dry from legal fees.
- gsich 9y ago
  Do you even know the OSI-model? Yes QUIC is a protocol, but not on the same layer as IPv6. So the comparison is useless.
  - halfteatree 9y ago
    It is one layer above IPv6, but what difference does that make? It's an improvement over an existing protocol to solve real problems. I don't think anyone can really know the OSI model. I know I frequently use HTTP as a Transport Layer protocol myself, so you got me there. ¯\_(ツ)_/¯
trevyn 9y ago
This has nothing to do with Google's politics. TCP is broken for modern networks, and the faster we ditch it for a protocol capable of reliable pipes with low latency and high throughput, the better off we'll all be.
- minikites 9y ago
  Not all networks are modern, what about the unreliable networks in most of the developing world?
  - trevyn 9y ago
    QUIC still works better than TCP on these networks, but the benefits are not as dramatic.
    - tybit 9y ago
      That doesn't seem to be supported by the article? "While it outperforms TCP in a wide range of scenarios, we found that it underperforms in the presence of packet reordering and on resource-constrained mobile devices."
      - trevyn 9y ago
        1) "We shared this result with a QUIC engineer, who subsequently informed us that the QUIC team is experimenting with dynamic threshold and time-based solutions to avoid falsely inferring loss in the presence of reordering." 2) "We find that, similar to the desktop environment, in mobile QUIC outperforms TCP in most cases; however, its advantages diminish across the board... The reason for this behavior is that QUIC runs in a userspace process, whereas TCP runs in the kernel." (i.e. putting it in the kernel should help.) https://conferences.sigcomm.org/imc/2017/papers/imc17-final39.pdf https://conferences.sigcomm.org/imc/2017/papers/imc17-final3...

_wmd 9y ago

Am I understanding things correctly? Because QUIC ramps up its bandwidth estimate more aggressively, it fundamentally competes unfairly with TCP? Is that an inherent property or something that can be fixed? Definitely it's too late to fix it on the TCP side Nice to know 7% of Internet traffic isn't playing fair with the rest just so one company's content loads a few ms faster! Really don't know how I feel about TCP in general losing out to QUIC if it saw much wider deployment outside Google And finally, it's incredibly disappointing to read these results from a third party, rather than having a balanced perspective as part of the original marketing

trevyn 9y ago
QUIC fundamentally competes on a different level because TCP is broken for modern networks. From a network usage standpoint, there isn't anything "unfair" about it -- QUIC runs atop UDP, which is also a perfectly acceptable and publicly available Internet protocol. And Chrome will happily use QUIC with non-Google hosts, the real problem is that production-quality QUIC server software is currently rare outside of Google.
- _wmd 9y ago
  One of the major downsides to UDP is that it lacks any form of congestion control, so it's hard to justify QUIC's behaviour due to what it's built atop :) Meanwhile we're not talking about a small slice of bandwidth spent on SIP video calls playing unfairly, we're talking about 70% or more of all Internet traffic suddenly stampeding over every pre-existing protocol, they're the floodgates I'm worried about
  - trevyn 9y ago
    If this was actually a problem, it would be trivial for network operators to prioritize TCP packets over UDP packets. QUIC builds modern congestion control on top of UDP.
    - solotronics 9y ago

mholt 9y ago

If you want to try QUIC yourself, you can use Caddy, which has experimental QUIC support (https://caddyserver.com/docs/cli#quic https://caddyserver.com/docs/cli#quic) powered by quic-go. (However, QUIC versions are extremely transient and has very limited support in Chrome; the next release of Caddy will bring it back up-to-date again.)

newman314 9y ago

I'm still waiting for QUIC support for nginx.

twouhm 9y ago

I wonder how QUIC compares to KCP[0]. [0]: https://github.com/skywind3000/kcp/blob/master/README.en.md https://github.com/skywind3000/kcp/blob/master/README.en.md

deleted 9y ago

[deleted]

fulafel 9y ago

What's the current positioning of QUIC by Google? It predates HTTP/2, and Google hasn't been making noises about it lately. Is it being slowly phased out or just in a "let it be" status for now in Chrome?

apenwarr 9y ago
It's being actively discussed at the IETF. Here's a doc that's last updated 2018-01-28: https://datatracker.ietf.org/doc/draft-ietf-quic-transport/ https://datatracker.ietf.org/doc/draft-ietf-quic-transport/
trevyn 9y ago
QUIC replaces TCP and is on IETF standards track. HTTP/2 replaces HTTP/1.1 (and is a direct descendent of SPDY, which has origins ~2009). HTTP/2 on TCP gets you some of the benefits that QUIC provides (multiplexing, reduced roundtrips), but HTTP/2 on QUIC is the best of both worlds. IMO, there's not much noise about QUIC mainly because most people don't care enough about latency to care, and the current lack of production-quality QUIC server software makes caring more complicated than "just enable it", so low demand all around. This should pick up when it gets closer to a ratified standard.

apenwarr 9y ago

This is a pretty disappointing article. It's really comparing a particular implementation (or maybe multiple implementations) of TCP congestion control, implemented in kernel space, with a particular implementation of QUIC congestion control, implemented in user space. The article points out that both the QUIC and TCP implementations they tested are using CUBIC congestion control, but that's not enough information, because the article also points out that QUIC is using "more aggressive parameters." It's tough to say which parameters are better, but what's unsaid is that a TCP implementation could change their parameters and get the same congestion control results as QUIC. The supposed poor performance of QUIC on resource-limited mobile devices is, as they point out, because it's a user space implementation that is thus more expensive. If QUIC becomes popular, I assume there will be kernel implementations that are as resource efficient as TCP. Meanwhile, it's a lot easier to to experiments (such as tuning CUBIC parameters!) when you don't have to reboot to install a new version. It's also quite common for various TCP congestion controllers to completely fail to saturate busy links, because of various limitations. If that happens, it might be that QUIC is able to fill the empty space, thereby taking "more than its fair share" only because TCP wasn't going to use that share anyway. Since the article doesn't even say which TCP implementation it's comparing against, and doesn't say what happens when the TCP sessions are competing only amongst themselves with no QUIC present, it's hard to say what's going on. The funny thing about all this is that, if they use the same congestion control (which seems to be the intention given that they are both using CUBIC and Google is separately trying to fix congestion control via BBR[1]), they should both be about equally fair. The performance benefits of QUIC are not even congestion control related! [1] https://queue.acm.org/detail.cfm?id=3022184 https://queue.acm.org/detail.cfm?id=3022184 [Disclaimer: I've worked with some of the people who wrote BBR and QUIC, so I'm biased.]

chemag 9y ago
+1 to this. QUIC allows using CUBIC BBR [1], so a comparison based on the exact parameters used is actually comparing the exact parameters used. The performance effects of QUIC implementing congestion control in userland are more interesting. OTOH, QUIC allows deploying new features to users (through cronet) in an efficient way. TCP does not. [1] https://chromium.googlesource.com/chromium/src/net/+/master/quic/core/congestion_control/bbr_sender.cc https://chromium.googlesource.com/chromium/src/net/+/master/... > [Disclaimer: I've worked with some of the people who wrote BBR and QUIC, so I'm biased.] Ditto