14 ms·

Why We Love QUIC and HTTP/3

210 points by kickdaddy 8y ago

http333 8y ago
It seems that QUIC is a new transfer protocol created to replace TCP. QUIC uses UDP and LTS/3 and solves the head of line problem addressed in HTTP2. Furthermore, sending the data encrypted allows QUIC to begin transfering data earlier. An experiment by google shows that in connections with high latency or loss QUIC gives a 15% reduction in highest latency.
ioquatix 8y ago
aka let's just put everything in the application layer because solving it at the protocol layer is too difficult.
- vbezhenar 8y ago
  Some people implement network stack in user-space. Abstractions are good, but they incur performance penalty or other restrictions and sometimes you need to remove those abstractions. I guess, web is too important today, so optimizing it might worth it, even if that requires unconventional measures.
- ralphm 8y ago
  Is this not a valid approach then? The issues of ossification and strict allowance for just known protocols appear to be big enough to cause things like SCTP to not have a viable, widespread use in their future.
  - yardstick 8y ago
    I believe that we will see ossification of QUIC eventually too. TCP has been around for decades, anything around that long is going to have issues rolling out new changes in a backwards compatible way. TLS 1.3 and the lengths it had to go to with backwards compatibility with middleboxes is another good example. I hope that QUIC has used these lessons from TCP and TLS to make changes in the future as easy and effective as possible, but I’m sure it’ll still have its limits.
    - agrover 8y ago
      The IETF QUIC working is well-aware of this, and is attempting to save design room for future QUIC versions as much as possible. The "QUIC invariants" spec documents everything that is guaranteed not to change, but other than than everything in a future QUICv2 could be updated (e.g. tls version, features, large parts of packet header layout).
      - yardstick 8y ago
        I fear that unless they are already actively using different values for those updatable values that middleboxes will implement a de facto version of QUIC that “just works” with what is out now, but no regard to gracefully handling forwards-compatibility. Example is the SSL version field which has become static because many implementations didn’t handle an unknown value gracefully.
- blackflame7000 8y ago
  It runs over UDP so they did go back to the protocol layer when designing a solution. Thats the whole point is to fix the tcp halt and retransmit delay when a packet gets lost.
  - josteink 8y ago
    So it will be firewalled by most corporate firewalls then? Hardly sounds useful.
    - blackflame7000 8y ago
      Why would it get firewalled? DNS runs over UDP on port 53
- cdmckay 8y ago
  The article mentions gives an example of why it’s difficult to improve TCP further. TCP Fast Open was standardized 8 years ago and is barely used. This is because updating TCP requires kernel updates, which just isn’t going to happen on most mobile devices. Thus, moving the protocol to userspace makes a lot of sense.
  - kijin 8y ago
    That doesn't sound like a particularly convincing reason. In order for mobile devices to benefit from HTTP/3, commonly used HTTP client libraries will have to be updated. Which usually happens on a similar timescale as kernel updates anyway.
    - lclarkmichalek 8y ago
      Much easier to update an app and its HTTP library than mobile device kernels. Particularly given how a large proportion of mobile devices are unsupported and won't get kernel upgrades any more.
      - michaelt 8y ago
        The thing is, there are two possible extremes: 1. Our design is complete, error-free and designed to stand the tests of time. It will be in common use, largely unchanged, in 40 years time. The TCP of the 2010s. There is widespread industry support. We want to move to the application layer to ease the initial roll-out. 2. Our design will need to change every few years, even we authors don't think it's finished. This is a Google-only project and most vendors are refusing to support it as they think it's badly designed crap. The ActiveX of the 2010s. We want to move to the application layer so we can force it through without anyone else's support. Where are we on the spectrum between those two options? I don't know.
        chrisweekly 8y ago
        Great question!
    - 8y ago
- carnagii 8y ago
  It is not necessarily bad. Moving complex stuff to user space can be good sometimes. There are things this would break like splice but also allows much tighter integration with applications because you can get and set state without system calls. As you layer on more complex app level adaptations to connection failures having it in user space means the client has the same capabilities irregardless of platform which is important so that both client and server can make mutual complimentary adaptations to the same conditions. This stuff is not really needed for web, it is mostly video conferencing type stuff that gets the biggest benefit. Googs new game service for instance.
- patrickmcmanus 8y ago
  The way I look at it a lot of what we logically think of as the network layer often exists in userspace anyhow. That's the point of DPDK/snabb/netmap and other kinds of driver bypass. The important design distinction about the layers is what element has access to what data. (e.g. routers need to see IP addresses to do their job, port numbers help kernels segment permission models, etc..) The rest of it is just about logical models, real-world workarounds, and luck... HTTP/3 will be able to get high bandwidth, buttery responsive restarts to connections far too long idle to keep "open" because it integrates security, application, and transport. That's thoughtful design, not a workaround. I'm going to paraphrase (and maybe bungle, because I don't have it at hand) from my favorite networking book of all time - the underappreciated _Network Algorithmics_ by George Varghese. He describes layers as a lovely way to model and think about a protocol or design, but often a terrible way to build one. I've spent a lot of time thinking about that, and I think QUIC gets it right - the layers are clear in how they inter-relate but they do so without being independent.
ex3ndr 8y ago
Are there ready to use mobile libraries for QUIC?
- charleslmunger 8y ago
  https://developer.android.com/guide/topics/connectivity/cronet https://developer.android.com/guide/topics/connectivity/cron...
felixhandte 8y ago
This is a timely post, since IETF 104 is happening this week in Prague[1]. The QUIC working group will be meeting on Tuesday and Wednesday to make progress on standardization[2]. [1] https://datatracker.ietf.org/meeting/104/agenda.html https://datatracker.ietf.org/meeting/104/agenda.html [2] https://datatracker.ietf.org/doc/draft-ietf-quic-transport/ https://datatracker.ietf.org/doc/draft-ietf-quic-transport/
- jabl 8y ago
  What about QUIC and L4S / TCP Prague? Are people working on something equivalent for QUIC as well, or are they reimplementing TCP Reno in QUIC?
  - patrickmcmanus 8y ago
    tl;dr; congestion control is basically pluggable. Much like in TCP, congestion control really isnt something required for interoperation between peers. Given the userspace nature of QUIC I would expect to see a lot of iteration on this front - for good and bad. (but hopefully the bad iterates quickly). The current drafts describe newreno is detail, but also explicitly call out the ability to run other things. I've seen reno, cubic, and bbr all run with quic and anticipate others to happen as well. That's one of the exciting things here.
tyingq 8y ago
Hopefully vendors like Forcepoint are trying to keep up. The first rollout of QUIC worked terribly in a lot of corporate environments because these MITM content filtering solutions didn't pay attention.
- zzzcpan 8y ago
  What reason MITM solutions have to support more client protocols that terminate on a local network?
  - tyingq 8y ago
    If it just drops QUIC requests, then the browser has to do 2 parallel requests, one HTTP, one QUIC, and pick the winner. I believe that's what Chrome does now.
    - zzzcpan 8y ago
      So, no reason then. That overhead is like microseconds and you can disable it if it bothers you (you have to configure web browsers for MITM anyway). I actually disable QUIC myself, because I've noticed it slows everything down too much on some home routers.
      - windexh8er 8y ago
        > I actually disable QUIC myself, because I've noticed it slows everything down too much on some home routers. I'm curious how you measured and came to this conclusion since the design and most all metrics claim the opposite? And are you sure it isn't a bufferbloat issue rather than QUIC?
        zzzcpan 8y ago
        I was just browsing websites that were loading unusually slowly, so did the usual ping/mtr to investigate, which pointed to the router. From there and a bit of tcpdumping the cause turned out to be UDP traffic to Google from another person, who was watching videos I think.
drewg123 8y ago
QUIC costs something like 2x to 4x as much CPU time to serve large files or streams per byte as compared to TCP. This is because the anti-middlebox protections also mean that modern network hardware and software offloads that greatly reduce CPU time cannot work with QUIC. When combined with the fact that QUIC is userspace, that's just deadly for performance. I'm talking about TSO, LRO (aka GRO), kTLS, and kTLS + hw encryption. Let's compare a 100MB file served via TCP to the same file served via QUIC. TCP: - web server sends 2MB at a time, 50x times, via async sendfile (50 syscalls & kqueue notifications) - kernel reads data from disk, and encrypts. The data is read once and written once by KTLS in the kernel. - TCP sends data to the NIC in large-sh chunks 1.5k to 64k at a time, lets say an average of 16k. So the network stack runs 6250 times to transmit. - The client acks every other frame, so that's 33333 acks. Let's say they are collapsed 2:1 by LRO, so the TCP stack runs 16,666 times to process acks QUIC: - web server mmaps or read()'s the file and encrypts it in userspace and sends it 1500b at a time (1 extra memory copy & 66,666 system calls) - UDP stack runs 66,666 to send data - UDP stack runs 33,333 number of times to receive QUIC acks (no idea what the aggregation is, lets say 2:1) - kernel wakes up web server to process QUIC acks 33,333 times. So for QUIC we have: - 4x as many network stack traversals due to the lack of TSO/LRO. - 1000x as many system calls, due to doing all the packet handing in userspace - at least one more data copy (kernel -> user) due to data handling in userspace. Some of these can be solved, by either moving QUIC into the kernel, or by using a DPDK-like userspace networking solution. However, the lack of TSO/LRO even by itself is a killer for performance. Disclaimer: I work on CDN performance. We've served 90Gb/s with a 12-core Xeon-D. To serve the same amount of traffic with QUIC, you'd probably need multiple Xeon Gold CPUS. I guess that Google can afford this.
- vfaronov 8y ago
  How much of this applies to 1~100KB responses?
  - drewg123 8y ago
    1k, not so much since there is no aggregation that can happen there anyway. 100k is not that much different than 100mb, except the TCP window will not be as far open, so TSO will not be as effective. Note that I work on a CDN that serves large media files, so I'm biased towards that workload.
move-on-by 8y ago
As far as I’ve been able to determine, QUIC suffers from the same SNI data leak that existing TLS versions with TCP has. I understand that ESNI is being (or is already?) included in the TLS 1.3 spec, but it’s obviously optional at this point. Anyways, since QUIC is being touted everywhere as being very secure: > [QUIC] protects both the data and the transport protocol itself It seems like missing ESNI as a required feature is a bit of a glaring omission. Does anyone have a better understanding? To me, it seems like a great opportunity to make ESNI required for HTTP/3. Much like how browsers made TLS required for HTTP/2. I would love further insights if anyone has any.
- londons_explore 8y ago
  A few people have been very vocal about not encrypting the SNI. Mostly firewall makers who obviously want to sniff it...
  - vbezhenar 8y ago
    I have no idea how one could deploy website with encrypted SNI. A lot of companies and countries block websites. If they can't determine the website, they will block IP addresses and that will cause a lot of other websites to break. It might work for simple websites which don't share IP address with other websites (but why encrypt SNI then), but it won't work for CDNs.
    - izacus 8y ago
      That sounds more like a feature than a bug.
      - vbezhenar 8y ago
        Broken website is a bug.
    - bitt 8y ago
      Blocking IP addresses can be very problematic though. If I understand it correctly, I think that countries might start to block ESNI altogether. If it is not widely implemented, websites/apps using it will standout which sadly could limit its adoption. For instance, if Signal decided to use ESNI, it will probably get blocked in those countries, but this can change if big companies wanted to use it. However, I still don't know how it will work exactly.
cagenut 8y ago
so the arista's are gonna be able to ecmp on it?
- wmf 8y ago
  It still has UDP headers with port numbers that can be used for ECMP. https://tools.ietf.org/id/draft-ietf-quic-manageability-00.html https://tools.ietf.org/id/draft-ietf-quic-manageability-00.h...
ignoramous 8y ago
/offtopic Good folks at fastly, I hope you're reading... I've been waiting very patiently for part 3 of this series for a good part of 2yrs now: https://www.fastly.com/blog/building-and-scaling-fastly-network-part-2-balancing-requests https://www.fastly.com/blog/building-and-scaling-fastly-netw...
mcguire 8y ago
"TCP Fast Open is a stellar example of one such modification to TCP: eight years after it was first proposed, it is still not widely deployed, largely due to middleboxes." Anyone remember TTCP?
- londons_explore 8y ago
  Fast Open is a bad idea for a bunch of other reasons, mainly the client spoofing their address yet still being able to use a lot of resources on the server.
  - tialaramex 8y ago
    Where would the client get a valid cookie from if they are "spoofing their address" ? If they don't have a valid cookie Fast Open costs the same as regular TCP in the face of adversaries trying to DOS you. You examine the packet, it doesn't have a valid cookie, you discard it. No further work, just like ordinary TCP.
collinf 8y ago
> These interposing network elements, called middleboxes, often unwittingly disallow changes to TCP headers and behavior, even if the server and the client are both willing. There is nothing worse than finding out that someone not even at the company anymore decided years before to deploy some crap like this. Drives me absolutely crazy to impose stuff like this where silos in companies means transitioning involves on the order of 4-5 different "components" need to change.
- marcosdumay 8y ago
  Oh, modern networks are basically just a single huge middlebox with servers on one side and intra|internet on the other side. There isn't much opportunity for people to plug random stuff between your server and the middlebox (the main middlebox would disallow it, like anything else), but there is still plenty of crappy rules everywhere and nobody knows why they exist or what they are. And you can't even call your ex-coworker and ask for help, because it's an ex-employee of the middlebox company, not yours.
scurvy 8y ago
QUIC will also usher in a new era of volumetric DDoS attacks. No longer can content providers use upstream ACLs to block udp garbage and fragments. The only option will be to use Fastly, AWS, or Cloudflare to ride out attacks. QUIC is the tool to bring about the next phase of Internet centralization by the mega players.
- lclarkmichalek 8y ago
  QUIC actually requires that request packets must be larger than the responses, until a handshake has been performed, in order to prevent reflection attacks.
  - skybrian 8y ago
    Do you mean larger or smaller? I thought the problem was amplification.
    - lclarkmichalek 8y ago
      Woops, fixed, thanks :)
  - scurvy 8y ago
    Does QUIC allow for UDP fragments? If so, we're screwed since most traffic is still IPv4.
  - mcpherrinm 8y ago
    The previous post's point is not that QUIC can be used for reflection attacks. It is that it uses UDP, which means UDP cannot be blocked by simple ACLs. Blocking all UDP is a simple technique for avoiding other protocols that allow for reflection DDoS attacks. I haven't got experience with how effective blocking UDP is as a mechanism for avoiding DDoS, but it does sound pretty simple and easy to deploy.
- londons_explore 8y ago
  It's mostly DNS amplification attacks your provider will be filtering out on UDP. They can still filter UDP port 53 to do that.
Improvotter 8y ago
I'm currently working with HTTP/2 (more specifically HAS with HTTP/2 Server Push) and it's just a huge pain to find a high-level library that can help with this. I fear that it'll take even longer for HTTP/3 to be adopted or HTTP/2 might just be skipped altogether. Why are there so many server-side implementations available for a variety of languages though many still lack some features or a client-side implementation altogether?
- vbezhenar 8y ago
  You can implement HTTP with a few hundreds LoC. It's an extremely simple protocol. TLS is not simple, but it's independent of HTTP, so you can use a separate implementation. HTTP/2 seems much harder.
- Matthias247 8y ago
  The short answer: It's hard! I implemented a HTTP/2 library for .NET (https://github.com/Matthias247/http2dotnet https://github.com/Matthias247/http2dotnet). It took quite a lot of time and dedication to get it spec conformant. I doubt that most employers (apart from some CDN) would have allowed me spending the time to get it to that level. And yet it still has lots of potential for improvement. HTTP/3 might be even harder (I haven't read the spec yet, but the whole UDP assembly and inclusion of encryption sounds more complicated). Compared to that building a small HTTP/1.1 library or a framework around it is much more approachable and might be also more rewarding.

210 points by kickdaddy 8y ago

It seems that QUIC is a new transfer protocol created to replace TCP. QUIC uses UDP and LTS/3 and solves the head of line problem addressed in HTTP2. Furthermore, sending the data encrypted allows QUIC to begin transfering data earlier. An experiment by google shows that in connections with high latency or loss QUIC gives a 15% reduction in highest latency.

ioquatix 8y ago

aka let's just put everything in the application layer because solving it at the protocol layer is too difficult.

vbezhenar 8y ago
Some people implement network stack in user-space. Abstractions are good, but they incur performance penalty or other restrictions and sometimes you need to remove those abstractions. I guess, web is too important today, so optimizing it might worth it, even if that requires unconventional measures.
ralphm 8y ago
Is this not a valid approach then? The issues of ossification and strict allowance for just known protocols appear to be big enough to cause things like SCTP to not have a viable, widespread use in their future.
- yardstick 8y ago
  I believe that we will see ossification of QUIC eventually too. TCP has been around for decades, anything around that long is going to have issues rolling out new changes in a backwards compatible way. TLS 1.3 and the lengths it had to go to with backwards compatibility with middleboxes is another good example. I hope that QUIC has used these lessons from TCP and TLS to make changes in the future as easy and effective as possible, but I’m sure it’ll still have its limits.
  - agrover 8y ago
    The IETF QUIC working is well-aware of this, and is attempting to save design room for future QUIC versions as much as possible. The "QUIC invariants" spec documents everything that is guaranteed not to change, but other than than everything in a future QUICv2 could be updated (e.g. tls version, features, large parts of packet header layout).
    - yardstick 8y ago
      I fear that unless they are already actively using different values for those updatable values that middleboxes will implement a de facto version of QUIC that “just works” with what is out now, but no regard to gracefully handling forwards-compatibility. Example is the SSL version field which has become static because many implementations didn’t handle an unknown value gracefully.
blackflame7000 8y ago
It runs over UDP so they did go back to the protocol layer when designing a solution. Thats the whole point is to fix the tcp halt and retransmit delay when a packet gets lost.
- josteink 8y ago
  So it will be firewalled by most corporate firewalls then? Hardly sounds useful.
  - blackflame7000 8y ago
    Why would it get firewalled? DNS runs over UDP on port 53
cdmckay 8y ago
The article mentions gives an example of why it’s difficult to improve TCP further. TCP Fast Open was standardized 8 years ago and is barely used. This is because updating TCP requires kernel updates, which just isn’t going to happen on most mobile devices. Thus, moving the protocol to userspace makes a lot of sense.
- kijin 8y ago
  That doesn't sound like a particularly convincing reason. In order for mobile devices to benefit from HTTP/3, commonly used HTTP client libraries will have to be updated. Which usually happens on a similar timescale as kernel updates anyway.
  - lclarkmichalek 8y ago
    Much easier to update an app and its HTTP library than mobile device kernels. Particularly given how a large proportion of mobile devices are unsupported and won't get kernel upgrades any more.
    - michaelt 8y ago
      The thing is, there are two possible extremes: 1. Our design is complete, error-free and designed to stand the tests of time. It will be in common use, largely unchanged, in 40 years time. The TCP of the 2010s. There is widespread industry support. We want to move to the application layer to ease the initial roll-out. 2. Our design will need to change every few years, even we authors don't think it's finished. This is a Google-only project and most vendors are refusing to support it as they think it's badly designed crap. The ActiveX of the 2010s. We want to move to the application layer so we can force it through without anyone else's support. Where are we on the spectrum between those two options? I don't know.
      - chrisweekly 8y ago
        Great question!
  - 8y ago
carnagii 8y ago
It is not necessarily bad. Moving complex stuff to user space can be good sometimes. There are things this would break like splice but also allows much tighter integration with applications because you can get and set state without system calls. As you layer on more complex app level adaptations to connection failures having it in user space means the client has the same capabilities irregardless of platform which is important so that both client and server can make mutual complimentary adaptations to the same conditions. This stuff is not really needed for web, it is mostly video conferencing type stuff that gets the biggest benefit. Googs new game service for instance.
patrickmcmanus 8y ago
The way I look at it a lot of what we logically think of as the network layer often exists in userspace anyhow. That's the point of DPDK/snabb/netmap and other kinds of driver bypass. The important design distinction about the layers is what element has access to what data. (e.g. routers need to see IP addresses to do their job, port numbers help kernels segment permission models, etc..) The rest of it is just about logical models, real-world workarounds, and luck... HTTP/3 will be able to get high bandwidth, buttery responsive restarts to connections far too long idle to keep "open" because it integrates security, application, and transport. That's thoughtful design, not a workaround. I'm going to paraphrase (and maybe bungle, because I don't have it at hand) from my favorite networking book of all time - the underappreciated _Network Algorithmics_ by George Varghese. He describes layers as a lovely way to model and think about a protocol or design, but often a terrible way to build one. I've spent a lot of time thinking about that, and I think QUIC gets it right - the layers are clear in how they inter-relate but they do so without being independent.

ex3ndr 8y ago

Are there ready to use mobile libraries for QUIC?

charleslmunger 8y ago
https://developer.android.com/guide/topics/connectivity/cronet https://developer.android.com/guide/topics/connectivity/cron...

felixhandte 8y ago

This is a timely post, since IETF 104 is happening this week in Prague[1]. The QUIC working group will be meeting on Tuesday and Wednesday to make progress on standardization[2]. [1] https://datatracker.ietf.org/meeting/104/agenda.html https://datatracker.ietf.org/meeting/104/agenda.html [2] https://datatracker.ietf.org/doc/draft-ietf-quic-transport/ https://datatracker.ietf.org/doc/draft-ietf-quic-transport/

jabl 8y ago
What about QUIC and L4S / TCP Prague? Are people working on something equivalent for QUIC as well, or are they reimplementing TCP Reno in QUIC?
- patrickmcmanus 8y ago
  tl;dr; congestion control is basically pluggable. Much like in TCP, congestion control really isnt something required for interoperation between peers. Given the userspace nature of QUIC I would expect to see a lot of iteration on this front - for good and bad. (but hopefully the bad iterates quickly). The current drafts describe newreno is detail, but also explicitly call out the ability to run other things. I've seen reno, cubic, and bbr all run with quic and anticipate others to happen as well. That's one of the exciting things here.

tyingq 8y ago

Hopefully vendors like Forcepoint are trying to keep up. The first rollout of QUIC worked terribly in a lot of corporate environments because these MITM content filtering solutions didn't pay attention.

zzzcpan 8y ago
What reason MITM solutions have to support more client protocols that terminate on a local network?
- tyingq 8y ago
  If it just drops QUIC requests, then the browser has to do 2 parallel requests, one HTTP, one QUIC, and pick the winner. I believe that's what Chrome does now.
  - zzzcpan 8y ago
    So, no reason then. That overhead is like microseconds and you can disable it if it bothers you (you have to configure web browsers for MITM anyway). I actually disable QUIC myself, because I've noticed it slows everything down too much on some home routers.
    - windexh8er 8y ago
      > I actually disable QUIC myself, because I've noticed it slows everything down too much on some home routers. I'm curious how you measured and came to this conclusion since the design and most all metrics claim the opposite? And are you sure it isn't a bufferbloat issue rather than QUIC?
      - zzzcpan 8y ago
        I was just browsing websites that were loading unusually slowly, so did the usual ping/mtr to investigate, which pointed to the router. From there and a bit of tcpdumping the cause turned out to be UDP traffic to Google from another person, who was watching videos I think.

drewg123 8y ago

QUIC costs something like 2x to 4x as much CPU time to serve large files or streams per byte as compared to TCP. This is because the anti-middlebox protections also mean that modern network hardware and software offloads that greatly reduce CPU time cannot work with QUIC. When combined with the fact that QUIC is userspace, that's just deadly for performance. I'm talking about TSO, LRO (aka GRO), kTLS, and kTLS + hw encryption. Let's compare a 100MB file served via TCP to the same file served via QUIC. TCP: - web server sends 2MB at a time, 50x times, via async sendfile (50 syscalls & kqueue notifications) - kernel reads data from disk, and encrypts. The data is read once and written once by KTLS in the kernel. - TCP sends data to the NIC in large-sh chunks 1.5k to 64k at a time, lets say an average of 16k. So the network stack runs 6250 times to transmit. - The client acks every other frame, so that's 33333 acks. Let's say they are collapsed 2:1 by LRO, so the TCP stack runs 16,666 times to process acks QUIC: - web server mmaps or read()'s the file and encrypts it in userspace and sends it 1500b at a time (1 extra memory copy & 66,666 system calls) - UDP stack runs 66,666 to send data - UDP stack runs 33,333 number of times to receive QUIC acks (no idea what the aggregation is, lets say 2:1) - kernel wakes up web server to process QUIC acks 33,333 times. So for QUIC we have: - 4x as many network stack traversals due to the lack of TSO/LRO. - 1000x as many system calls, due to doing all the packet handing in userspace - at least one more data copy (kernel -> user) due to data handling in userspace. Some of these can be solved, by either moving QUIC into the kernel, or by using a DPDK-like userspace networking solution. However, the lack of TSO/LRO even by itself is a killer for performance. Disclaimer: I work on CDN performance. We've served 90Gb/s with a 12-core Xeon-D. To serve the same amount of traffic with QUIC, you'd probably need multiple Xeon Gold CPUS. I guess that Google can afford this.

vfaronov 8y ago
How much of this applies to 1~100KB responses?
- drewg123 8y ago
  1k, not so much since there is no aggregation that can happen there anyway. 100k is not that much different than 100mb, except the TCP window will not be as far open, so TSO will not be as effective. Note that I work on a CDN that serves large media files, so I'm biased towards that workload.

move-on-by 8y ago

As far as I’ve been able to determine, QUIC suffers from the same SNI data leak that existing TLS versions with TCP has. I understand that ESNI is being (or is already?) included in the TLS 1.3 spec, but it’s obviously optional at this point. Anyways, since QUIC is being touted everywhere as being very secure: > [QUIC] protects both the data and the transport protocol itself It seems like missing ESNI as a required feature is a bit of a glaring omission. Does anyone have a better understanding? To me, it seems like a great opportunity to make ESNI required for HTTP/3. Much like how browsers made TLS required for HTTP/2. I would love further insights if anyone has any.

londons_explore 8y ago
A few people have been very vocal about not encrypting the SNI. Mostly firewall makers who obviously want to sniff it...
- vbezhenar 8y ago
  I have no idea how one could deploy website with encrypted SNI. A lot of companies and countries block websites. If they can't determine the website, they will block IP addresses and that will cause a lot of other websites to break. It might work for simple websites which don't share IP address with other websites (but why encrypt SNI then), but it won't work for CDNs.
  - izacus 8y ago
    That sounds more like a feature than a bug.
    - vbezhenar 8y ago
      Broken website is a bug.
  - bitt 8y ago
    Blocking IP addresses can be very problematic though. If I understand it correctly, I think that countries might start to block ESNI altogether. If it is not widely implemented, websites/apps using it will standout which sadly could limit its adoption. For instance, if Signal decided to use ESNI, it will probably get blocked in those countries, but this can change if big companies wanted to use it. However, I still don't know how it will work exactly.

cagenut 8y ago

so the arista's are gonna be able to ecmp on it?

wmf 8y ago
It still has UDP headers with port numbers that can be used for ECMP. https://tools.ietf.org/id/draft-ietf-quic-manageability-00.html https://tools.ietf.org/id/draft-ietf-quic-manageability-00.h...

ignoramous 8y ago

/offtopic Good folks at fastly, I hope you're reading... I've been waiting very patiently for part 3 of this series for a good part of 2yrs now: https://www.fastly.com/blog/building-and-scaling-fastly-network-part-2-balancing-requests https://www.fastly.com/blog/building-and-scaling-fastly-netw...

mcguire 8y ago

"TCP Fast Open is a stellar example of one such modification to TCP: eight years after it was first proposed, it is still not widely deployed, largely due to middleboxes." Anyone remember TTCP?

londons_explore 8y ago
Fast Open is a bad idea for a bunch of other reasons, mainly the client spoofing their address yet still being able to use a lot of resources on the server.
- tialaramex 8y ago
  Where would the client get a valid cookie from if they are "spoofing their address" ? If they don't have a valid cookie Fast Open costs the same as regular TCP in the face of adversaries trying to DOS you. You examine the packet, it doesn't have a valid cookie, you discard it. No further work, just like ordinary TCP.

collinf 8y ago

> These interposing network elements, called middleboxes, often unwittingly disallow changes to TCP headers and behavior, even if the server and the client are both willing. There is nothing worse than finding out that someone not even at the company anymore decided years before to deploy some crap like this. Drives me absolutely crazy to impose stuff like this where silos in companies means transitioning involves on the order of 4-5 different "components" need to change.

marcosdumay 8y ago
Oh, modern networks are basically just a single huge middlebox with servers on one side and intra|internet on the other side. There isn't much opportunity for people to plug random stuff between your server and the middlebox (the main middlebox would disallow it, like anything else), but there is still plenty of crappy rules everywhere and nobody knows why they exist or what they are. And you can't even call your ex-coworker and ask for help, because it's an ex-employee of the middlebox company, not yours.

scurvy 8y ago

QUIC will also usher in a new era of volumetric DDoS attacks. No longer can content providers use upstream ACLs to block udp garbage and fragments. The only option will be to use Fastly, AWS, or Cloudflare to ride out attacks. QUIC is the tool to bring about the next phase of Internet centralization by the mega players.

lclarkmichalek 8y ago
QUIC actually requires that request packets must be larger than the responses, until a handshake has been performed, in order to prevent reflection attacks.
- skybrian 8y ago
  Do you mean larger or smaller? I thought the problem was amplification.
  - lclarkmichalek 8y ago
    Woops, fixed, thanks :)
- scurvy 8y ago
  Does QUIC allow for UDP fragments? If so, we're screwed since most traffic is still IPv4.
- mcpherrinm 8y ago
  The previous post's point is not that QUIC can be used for reflection attacks. It is that it uses UDP, which means UDP cannot be blocked by simple ACLs. Blocking all UDP is a simple technique for avoiding other protocols that allow for reflection DDoS attacks. I haven't got experience with how effective blocking UDP is as a mechanism for avoiding DDoS, but it does sound pretty simple and easy to deploy.
londons_explore 8y ago
It's mostly DNS amplification attacks your provider will be filtering out on UDP. They can still filter UDP port 53 to do that.

Improvotter 8y ago

I'm currently working with HTTP/2 (more specifically HAS with HTTP/2 Server Push) and it's just a huge pain to find a high-level library that can help with this. I fear that it'll take even longer for HTTP/3 to be adopted or HTTP/2 might just be skipped altogether. Why are there so many server-side implementations available for a variety of languages though many still lack some features or a client-side implementation altogether?

vbezhenar 8y ago
You can implement HTTP with a few hundreds LoC. It's an extremely simple protocol. TLS is not simple, but it's independent of HTTP, so you can use a separate implementation. HTTP/2 seems much harder.
Matthias247 8y ago
The short answer: It's hard! I implemented a HTTP/2 library for .NET (https://github.com/Matthias247/http2dotnet https://github.com/Matthias247/http2dotnet). It took quite a lot of time and dedication to get it spec conformant. I doubt that most employers (apart from some CDN) would have allowed me spending the time to get it to that level. And yet it still has lots of potential for improvement. HTTP/3 might be even harder (I haven't read the spec yet, but the whole UDP assembly and inclusion of encryption sounds more complicated). Compared to that building a small HTTP/1.1 library or a framework around it is much more approachable and might be also more rewarding.