9 ms·

Introducing QUIC support for HTTPS load balancing

170 points by Sami_Lehtinen 8y ago

puzzle 8y ago
HTTP/1.1 to the backend? How about HTTP/2 so people can run gRPC services straight through without translations?
- xfs 8y ago
  A QUIC to h2 adapter is probably very complex to implement in a fully compliant way since both protocols are highly stateful. gRPC has a Cronet transport which again has experimental support for QUIC. I don't think it's reached production level yet...
  - Matthias247 8y ago
    It shouldn't be that complex. On top of both protocols are HTTP semantics. You get a request, which has some headers in both direction and then contains a bidirectional stream. You forward both in each direction. It's not different than building a HTTP/2 (or HTTP/QUIC) to HTTP/1 gateway. On both sides are HTTP APIs. The statefulness is on the lower level, where streams/requests are multiplexed. However that doesn't need to be conveyed or transferred during proxying.
    - xfs 8y ago
      You're basically describing attaching an h1 to h2 adapter behind the existing QUIC to h2 adapter with HTTP/1 as the intermediary protocol, which is a simplification for both sides. Of course that will not be that complex. But the downside is this approach limits both protocols from reaching their full potential by having to operate at a higher level.
  - euyyn 8y ago
    gRPC over QUIC has been in use by some Google apps in production for over a year :)
- jaydj 8y ago
  It's in Alpha. https://groups.google.com/forum/#!msg/grpc-io/bfURoNLojHo/_Go44ZzHBwAJ https://groups.google.com/forum/#!msg/grpc-io/bfURoNLojHo/_G...
- seangrogg 8y ago
  To be fair, the majority of benefits for HTTP/2 and QUIC are for the browser and experienced at the edge of your service. While that's not to say having the option to pipe HTTP/2 to your gRPC servers wouldn't be nice the majority I've seen are using HTTP/1.1 at their application layer and what happens from there is a matter between consenting servers.
  - euyyn 8y ago
    The counterpoint to that would be that, inside Google, it's all gRPC or Stubby.
    - seangrogg 8y ago
      I counter your counter with this neither being an internal tool nor gRPC/Stubby owning even plurality utilization outside of Google.
  - imtringued 8y ago
    If QUIC allows arbitrary length datagrams and retransmits them out of order then it's still going to be useful for video games and other systems with long lived connections. This is how I imagine how a replacement for TCP and UDP should work. But it doesn't look like there are standalone implementations of QUIC so I have no idea what the API actually looks like. There is only google's implementation which is part of chromium.
    - Gazler 8y ago
      There are a few. This page lists the IETF QUIC implementations and gQUIC implementations (which is supported here) at the bottom. https://github.com/quicwg/base-drafts/wiki/Implementations https://github.com/quicwg/base-drafts/wiki/Implementations
  - manigandham 8y ago
    HTTP/2 would still be great for upstreams to ensure HTTPS and make things much more efficient via compression and multiplexing. It makes a difference in higher-scale situations.
dogecoinbase 8y ago
QUIC makes the web faster, particularly for slow connections QUIC is a characteristic Google standard -- dictated, poorly thought through, and destructive for the rest of the web. See e.g. https://blog.apnic.net/2018/01/29/measuring-quic-vs-tcp-mobile-desktop/ https://blog.apnic.net/2018/01/29/measuring-quic-vs-tcp-mobi... "our experiments showed that QUIC always consumes more than half of the bottleneck bandwidth, even as the number of competing TCP flows increases." QUIC improves speed for high-latency connections (at the cost of cohoused TCP flows, as above), but is worse than no change for the reality of poor connectivity on the ground, which has highly variable latency, packet loss and/or duplication, and (sometimes profoundly) asymmetric routing.
- eklitzke 8y ago
  Criticizing QUIC becasue of its interaction with existing TCP flows is tricky, because the most efficient protocol isn't necessarily the one that interoperates best with existing protocols. There have been a lot of iterations on TCP congestion control algorithms that improve bandwidth/latency of TCP connections under the new congestion control algorithms at the detriment of legacy TCP connections. I don't know enough about QUIC to say whether it's good or bad, but the standardization process by which SPDY evolved into HTTP/2 seemed good to me: there was a lot of external feedback that was incorporated and addressed, and the protocol evolution and standardization happened over a reasonable timeline. Do you see a problem with how QUIC is being developed by the IETF?
  - Sylos 8y ago
    As the article that the guy linked to points out, this unfairness is due an aggressive configuration of the CUBIC congestion control algorithm in QUIC. TCP also uses CUBIC. So, you would have just as well already been able to be a dick by configuring your CUBIC in TCP to be more aggressive. Lots of research has gone into an optimal configuration of CUBIC for TCP, under the presumption that everyone will use this implementation, and therefore with the goal of optimal results for everyone. QUIC now falls out of line because early adopters get better results, as they can unfairly compete against the fair TCP implementation. In the long term however, if QUIC is adopted by everyone, this will lead to a worse result for everyone, because this optimal configuration that's been researched under TCP has been thrown out for no good reason. At least I'm not aware of a good reason. Maybe QUIC is actually somewhat less impacted by packets being dropped or something like that and does work better with everyone just trying to squeeze everything through aggressively, but even if that is the case, I cannot imagine it being the case to the degree that this CUBIC configuration has been made more aggressive.
proyb2 8y ago
Quicly is another interesting repo which is developed by the same author who built H2O HTTP server now at Fastly (https://github.com/h2o/h2o/ https://github.com/h2o/h2o/) https://github.com/h2o/quicly https://github.com/h2o/quicly
- wmf 8y ago
  Because QUIC + Fastly = Quicly. Genius.
textmode 8y ago
"But HTTP/2 uses TCP as its transport, so all of its streams can be blocked when a single TCP packet is lost-a problem called head-of-line blocking." But see https://datatracker.ietf.org/wg/httpbis/charter/ https://datatracker.ietf.org/wg/httpbis/charter/ "It is expected that HTTP/2.0 will: ... Address the "head of line blocking" problem in HTTP." Dislaimer: I am a longtime (satisfied) HTTP pipelining user for retrieving multiple resources from single domain over single connection and a CurveCP user since 2011 on own network. Never had any problems with HOL blocking that I am aware of, but I am only a dumb end user and wish to learn what I might be missing.
- charleslmunger 8y ago
  Different kind of head of line blocking. In HTTP it refers to a technique where a client sends multiple requests without waiting for their respective responses. However, the server has to send the responses in order - so if the client makes request A and request B, the response for B can't be delivered until A's response is delivered. So if A's response is large, B will be delayed. In HTTP/2, it's all multiplexed - so the server can interleave responses.
- euyyn 8y ago
  My understanding is that was head of line blocking "at the HTTP level": When multiple HTTP/1.1 requests shared a TCP connection, the responses had to come in order, so if the first response was slow, all others were blocked behind it. The authors instead talk of the analogous problem at the TCP level, which I don't remember enough of TCP to comment on.
- zeronone 8y ago
  It is about TCP Head of line blocking. For more check https://hpbn.co/building-blocks-of-tcp/#head-of-line-blocking https://hpbn.co/building-blocks-of-tcp/#head-of-line-blockin...
- eeZah7Ux 8y ago
  head of line blocking in HTTP and TCP are different issues. The latter impacts HTTP/2.0 as well, and it's an important selling point for QUIC.
riobard 8y ago
The unfortunate fact of QUIC is that it's based on encrypted UDP packets, which many ISP routers will happily drop during peak traffic/high load periods. EDIT: oh come on down voters. This comment is stating a fact. What's your problem with it?
- ianlevesque 8y ago
  Given that it retransmits and TCP will also be suffering (slightly less?) during such periods, it doesn't seem like a show stopper.
  - riobard 8y ago
    Depends on the ISP's traffic policy. I've witnessed many times when UDP packet loss is over 50% while TCP loss is below 10%.
    - pilif 8y ago
      If you are an ISP and you're preventing people using Google Chrome from accessing Google's properties (like youtube and gmail) during peak hours because you're dropping UDP packets, then you'll hear from your customers.
      - throwaway2048 8y ago
        if UDP drops too much, chrome will almost certianly switch back to TCP anyways.
  - JoeAltmaier 8y ago
    UDP is the stepchild of protocols. Many devices drop it first during congestion. It is not the peer of TCP in that regard.
- zAy0LfpBZLC8mAC 8y ago
  The fortunate fact of QUIC is that it's based on encrypted UDP packets, so ISPs will have to adapt to the fact that not randomly dropping encrypted UDP packets is essential to their users' experience.
dochtman 8y ago
Note that this seems to be gQUIC (the Google-internal variant of QUIC that it has been using for a few years), and is thus not interoperable with most non-Google QUIC implementations, which follow the IETF drafts.
- deleted 8y ago
  [deleted]
- pilif 8y ago
  > most non-Google QUIC implementations, which follow the IETF drafts ... which in practice means that you would be able to actually use QUIC with the most widely deployed client implementation (which might in-fact be the only client implementation that's not a development build right now). IETF QUIC doesn't really exist yet, so offering it over Google's QUIC which exists in Chrome would provide next to no benefit to everybody. However, I do hope, or rather, expect google to very quickly move both Chrome and the load balancers to IETF QUIC once that is finalized. Until then, I keep my pitchfork safely stored in its closet because what is being offered here certainly is the practical offering and the correct offering would be completely impractical.
  - delroth 8y ago
    Similarly, SPDY was not compatible with HTTP2 before the standard was finalized, but then Google quickly switched to the standard once it was available. I don't think there is too much to worry about there.
    - jacksmith21006 8y ago
      Plus Google giving them SPDY got us the new http2 standard much more quickly and the benefits. Google controlled the two biggest clients and the two biggest servers and still gave the IP to the community. Love how they just want to push the broader industry forward.
  - dochtman 8y ago
    I don't necessarily disagree with all that; however, I do think it would have been good for Google to disambiguate in their announcement and documentation that they don't mean standards-track QUIC here, or more generally, what version of QUIC they mean. At least with SPDY it was clearer that SPDY was something different from HTTP 2.
deno 8y ago
How does QUIC work together with µTP/LEDBAT? Also is there any plan to implement this type of congestion control in QUIC? Would it make sense? It could certainly be useful for background synchronization in PWAs.
zbjornson 8y ago
I'm afraid to enable this. When I had AT&T Business Internet (maybe irrelevant), we frequently got QUIC protocol errors for many Google sites like YouTube (see others crying for help in [1]). My fix was to disable QUIC in Chrome. I don't want our users to hit that as it's not something a normal user could easily diagnose and remedy. No idea how common it is. [1] https://support.google.com/chrome/forum/AAAAP1KN0B0xKFfhDvhjU8/?hl=en https://support.google.com/chrome/forum/AAAAP1KN0B0xKFfhDvhj...
souterrain 8y ago
From a client perspective, QUIC seems like it would provide the most value on mobile devices. Any word on how well supported this is on Android clients? I'm assuming iOS adoption will take some time. In many enterprise networks, TLS MitMboxes rule QUIC out completely. I'm not aware of any middleboxes that support it. Nevermind that even on many guest networks, I see udp/53 hijacked, and all other UDP dropped.
- deleted 8y ago
  [deleted]

170 points by Sami_Lehtinen 8y ago

puzzle 8y ago

HTTP/1.1 to the backend? How about HTTP/2 so people can run gRPC services straight through without translations?

xfs 8y ago
A QUIC to h2 adapter is probably very complex to implement in a fully compliant way since both protocols are highly stateful. gRPC has a Cronet transport which again has experimental support for QUIC. I don't think it's reached production level yet...
- Matthias247 8y ago
  It shouldn't be that complex. On top of both protocols are HTTP semantics. You get a request, which has some headers in both direction and then contains a bidirectional stream. You forward both in each direction. It's not different than building a HTTP/2 (or HTTP/QUIC) to HTTP/1 gateway. On both sides are HTTP APIs. The statefulness is on the lower level, where streams/requests are multiplexed. However that doesn't need to be conveyed or transferred during proxying.
  - xfs 8y ago
    You're basically describing attaching an h1 to h2 adapter behind the existing QUIC to h2 adapter with HTTP/1 as the intermediary protocol, which is a simplification for both sides. Of course that will not be that complex. But the downside is this approach limits both protocols from reaching their full potential by having to operate at a higher level.
- euyyn 8y ago
  gRPC over QUIC has been in use by some Google apps in production for over a year :)
jaydj 8y ago
It's in Alpha. https://groups.google.com/forum/#!msg/grpc-io/bfURoNLojHo/_Go44ZzHBwAJ https://groups.google.com/forum/#!msg/grpc-io/bfURoNLojHo/_G...
seangrogg 8y ago
To be fair, the majority of benefits for HTTP/2 and QUIC are for the browser and experienced at the edge of your service. While that's not to say having the option to pipe HTTP/2 to your gRPC servers wouldn't be nice the majority I've seen are using HTTP/1.1 at their application layer and what happens from there is a matter between consenting servers.
- euyyn 8y ago
  The counterpoint to that would be that, inside Google, it's all gRPC or Stubby.
  - seangrogg 8y ago
    I counter your counter with this neither being an internal tool nor gRPC/Stubby owning even plurality utilization outside of Google.
- imtringued 8y ago
  If QUIC allows arbitrary length datagrams and retransmits them out of order then it's still going to be useful for video games and other systems with long lived connections. This is how I imagine how a replacement for TCP and UDP should work. But it doesn't look like there are standalone implementations of QUIC so I have no idea what the API actually looks like. There is only google's implementation which is part of chromium.
  - Gazler 8y ago
    There are a few. This page lists the IETF QUIC implementations and gQUIC implementations (which is supported here) at the bottom. https://github.com/quicwg/base-drafts/wiki/Implementations https://github.com/quicwg/base-drafts/wiki/Implementations
- manigandham 8y ago
  HTTP/2 would still be great for upstreams to ensure HTTPS and make things much more efficient via compression and multiplexing. It makes a difference in higher-scale situations.

dogecoinbase 8y ago

QUIC makes the web faster, particularly for slow connections QUIC is a characteristic Google standard -- dictated, poorly thought through, and destructive for the rest of the web. See e.g. https://blog.apnic.net/2018/01/29/measuring-quic-vs-tcp-mobile-desktop/ https://blog.apnic.net/2018/01/29/measuring-quic-vs-tcp-mobi... "our experiments showed that QUIC always consumes more than half of the bottleneck bandwidth, even as the number of competing TCP flows increases." QUIC improves speed for high-latency connections (at the cost of cohoused TCP flows, as above), but is worse than no change for the reality of poor connectivity on the ground, which has highly variable latency, packet loss and/or duplication, and (sometimes profoundly) asymmetric routing.

eklitzke 8y ago
Criticizing QUIC becasue of its interaction with existing TCP flows is tricky, because the most efficient protocol isn't necessarily the one that interoperates best with existing protocols. There have been a lot of iterations on TCP congestion control algorithms that improve bandwidth/latency of TCP connections under the new congestion control algorithms at the detriment of legacy TCP connections. I don't know enough about QUIC to say whether it's good or bad, but the standardization process by which SPDY evolved into HTTP/2 seemed good to me: there was a lot of external feedback that was incorporated and addressed, and the protocol evolution and standardization happened over a reasonable timeline. Do you see a problem with how QUIC is being developed by the IETF?
- Sylos 8y ago
  As the article that the guy linked to points out, this unfairness is due an aggressive configuration of the CUBIC congestion control algorithm in QUIC. TCP also uses CUBIC. So, you would have just as well already been able to be a dick by configuring your CUBIC in TCP to be more aggressive. Lots of research has gone into an optimal configuration of CUBIC for TCP, under the presumption that everyone will use this implementation, and therefore with the goal of optimal results for everyone. QUIC now falls out of line because early adopters get better results, as they can unfairly compete against the fair TCP implementation. In the long term however, if QUIC is adopted by everyone, this will lead to a worse result for everyone, because this optimal configuration that's been researched under TCP has been thrown out for no good reason. At least I'm not aware of a good reason. Maybe QUIC is actually somewhat less impacted by packets being dropped or something like that and does work better with everyone just trying to squeeze everything through aggressively, but even if that is the case, I cannot imagine it being the case to the degree that this CUBIC configuration has been made more aggressive.

proyb2 8y ago

Quicly is another interesting repo which is developed by the same author who built H2O HTTP server now at Fastly (https://github.com/h2o/h2o/ https://github.com/h2o/h2o/) https://github.com/h2o/quicly https://github.com/h2o/quicly

wmf 8y ago
Because QUIC + Fastly = Quicly. Genius.

textmode 8y ago

"But HTTP/2 uses TCP as its transport, so all of its streams can be blocked when a single TCP packet is lost-a problem called head-of-line blocking." But see https://datatracker.ietf.org/wg/httpbis/charter/ https://datatracker.ietf.org/wg/httpbis/charter/ "It is expected that HTTP/2.0 will: ... Address the "head of line blocking" problem in HTTP." Dislaimer: I am a longtime (satisfied) HTTP pipelining user for retrieving multiple resources from single domain over single connection and a CurveCP user since 2011 on own network. Never had any problems with HOL blocking that I am aware of, but I am only a dumb end user and wish to learn what I might be missing.

charleslmunger 8y ago
Different kind of head of line blocking. In HTTP it refers to a technique where a client sends multiple requests without waiting for their respective responses. However, the server has to send the responses in order - so if the client makes request A and request B, the response for B can't be delivered until A's response is delivered. So if A's response is large, B will be delayed. In HTTP/2, it's all multiplexed - so the server can interleave responses.
euyyn 8y ago
My understanding is that was head of line blocking "at the HTTP level": When multiple HTTP/1.1 requests shared a TCP connection, the responses had to come in order, so if the first response was slow, all others were blocked behind it. The authors instead talk of the analogous problem at the TCP level, which I don't remember enough of TCP to comment on.
zeronone 8y ago
It is about TCP Head of line blocking. For more check https://hpbn.co/building-blocks-of-tcp/#head-of-line-blocking https://hpbn.co/building-blocks-of-tcp/#head-of-line-blockin...
eeZah7Ux 8y ago
head of line blocking in HTTP and TCP are different issues. The latter impacts HTTP/2.0 as well, and it's an important selling point for QUIC.

riobard 8y ago

The unfortunate fact of QUIC is that it's based on encrypted UDP packets, which many ISP routers will happily drop during peak traffic/high load periods. EDIT: oh come on down voters. This comment is stating a fact. What's your problem with it?

ianlevesque 8y ago
Given that it retransmits and TCP will also be suffering (slightly less?) during such periods, it doesn't seem like a show stopper.
- riobard 8y ago
  Depends on the ISP's traffic policy. I've witnessed many times when UDP packet loss is over 50% while TCP loss is below 10%.
  - pilif 8y ago
    If you are an ISP and you're preventing people using Google Chrome from accessing Google's properties (like youtube and gmail) during peak hours because you're dropping UDP packets, then you'll hear from your customers.
    - throwaway2048 8y ago
      if UDP drops too much, chrome will almost certianly switch back to TCP anyways.
- JoeAltmaier 8y ago
  UDP is the stepchild of protocols. Many devices drop it first during congestion. It is not the peer of TCP in that regard.
zAy0LfpBZLC8mAC 8y ago
The fortunate fact of QUIC is that it's based on encrypted UDP packets, so ISPs will have to adapt to the fact that not randomly dropping encrypted UDP packets is essential to their users' experience.

dochtman 8y ago

Note that this seems to be gQUIC (the Google-internal variant of QUIC that it has been using for a few years), and is thus not interoperable with most non-Google QUIC implementations, which follow the IETF drafts.

deleted 8y ago
[deleted]
pilif 8y ago
> most non-Google QUIC implementations, which follow the IETF drafts ... which in practice means that you would be able to actually use QUIC with the most widely deployed client implementation (which might in-fact be the only client implementation that's not a development build right now). IETF QUIC doesn't really exist yet, so offering it over Google's QUIC which exists in Chrome would provide next to no benefit to everybody. However, I do hope, or rather, expect google to very quickly move both Chrome and the load balancers to IETF QUIC once that is finalized. Until then, I keep my pitchfork safely stored in its closet because what is being offered here certainly is the practical offering and the correct offering would be completely impractical.
- delroth 8y ago
  Similarly, SPDY was not compatible with HTTP2 before the standard was finalized, but then Google quickly switched to the standard once it was available. I don't think there is too much to worry about there.
  - jacksmith21006 8y ago
    Plus Google giving them SPDY got us the new http2 standard much more quickly and the benefits. Google controlled the two biggest clients and the two biggest servers and still gave the IP to the community. Love how they just want to push the broader industry forward.
- dochtman 8y ago
  I don't necessarily disagree with all that; however, I do think it would have been good for Google to disambiguate in their announcement and documentation that they don't mean standards-track QUIC here, or more generally, what version of QUIC they mean. At least with SPDY it was clearer that SPDY was something different from HTTP 2.

deno 8y ago

How does QUIC work together with µTP/LEDBAT? Also is there any plan to implement this type of congestion control in QUIC? Would it make sense? It could certainly be useful for background synchronization in PWAs.

zbjornson 8y ago

I'm afraid to enable this. When I had AT&T Business Internet (maybe irrelevant), we frequently got QUIC protocol errors for many Google sites like YouTube (see others crying for help in [1]). My fix was to disable QUIC in Chrome. I don't want our users to hit that as it's not something a normal user could easily diagnose and remedy. No idea how common it is. [1] https://support.google.com/chrome/forum/AAAAP1KN0B0xKFfhDvhjU8/?hl=en https://support.google.com/chrome/forum/AAAAP1KN0B0xKFfhDvhj...

souterrain 8y ago

From a client perspective, QUIC seems like it would provide the most value on mobile devices. Any word on how well supported this is on Android clients? I'm assuming iOS adoption will take some time. In many enterprise networks, TLS MitMboxes rule QUIC out completely. I'm not aware of any middleboxes that support it. Nevermind that even on many guest networks, I see udp/53 hijacked, and all other UDP dropped.

deleted 8y ago
[deleted]