5 ms·

As a counterpoint to the advantages, the disadvantage is that QUIC is much more expensive on the server side. You loose 25 years of stateless hardware offloads

As a counterpoint to the advantages, the disadvantage is that QUIC is much more expensive on the server side. You loose 25 years of stateless hardware offloads done by the NIC. Measuring the workload that I care about (Netflix CDN serving static videos), loosing TSO, LRO, and software kTLS, like you would with QUIC, takes a server from serving 90Gb/s @ 62% CPU to serving 35Gb/s and CPU limited. So we'd need essentially 3x as many servers to serve via QUIC as we do to serve via TCP. That has real environmental costs. And these are old numbers, from servers that don't support hardware kTLS, so the impact of loosing hardware kTLS would make things even worse.

jeffbee 6y ago
Horses for courses. Is anyone suggesting QUIC for gigantic bulk transfers? TCP isn’t great for latency-sensitive small transfers so we will have QUIC. QUIC will not be perfect for everything, like TCP is not perfect for everything.
- drewg123 6y ago
  I'm pretty sure YouTube uses QUIC for video streaming today, which is a similar workload to ours.
  - jeffbee 6y ago
    I think that indicates more about the economics of YouTube than anything. Probably their frontend CPU costs would have to go up by a factor of ten thousand before it even began to approach their costs for compression and storage of user uploads.
  - virattara 6y ago
    > I'm pretty sure YouTube uses QUIC for video streaming today Is there any media transfer protocol based on QUIC?
    - drewg123 6y ago
      I found an article from 4 years ago saying that 20% of youtube's traffic was QUIC: https://www.fiercewireless.com/wireless/youtube-driving-more-quic-based-traffic-mobile-vasona https://www.fiercewireless.com/wireless/youtube-driving-more...
    - detaro 6y ago
      Youtube is delivering video over HTTP. HTTP3, i.e. it is transported over QUIC.
    - elithrar 6y ago
      Lucas Pardue, who is the IETF QUIC WG chair, has a great talk from Mux's Demuxed conf last year on using QUIC as a transport for UDP streaming: https://www.youtube.com/watch?v=Zdkjd7-EWmQ&feature=emb_title https://www.youtube.com/watch?v=Zdkjd7-EWmQ&feature=emb_titl... This is not dissimilar from SRT.
  - elithrar 6y ago
    Yep: YouTube currently delivers over gQUIC and h3 (IETF QUIC). We advertise the following alt-svc for most devices: > h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43" HTTP/3 (over IETF QUIC) is still not as widely deployed due to lesser client support, but that's growing as both Chrome[1], Firefox and Apple (in iOS Safari, when enabled) adopt it and the latest drafts. [1]: https://blog.chromium.org/2020/10/chrome-is-deploying-http3-and-ietf-quic.html https://blog.chromium.org/2020/10/chrome-is-deploying-http3-...
- masklinn 6y ago
  > Horses for courses. Is anyone suggesting QUIC for gigantic bulk transfers? Anyone who's looking to replace HTTP/1.1 and HTTP/2 with HTTP/3? AFAIK QUIC is completely integral to that, in order to avoid this issue you'll need a mixed deployment of HTTP/3 and an older version of the standard.
  - anderspitman 6y ago
    I just hope we don't remove support for HTTP/1.1 from our stacks entirely. 2/3 are great for performance but an order of magnitude more complex.
    - M2Ys4U 6y ago
      I imagine that HTTP/2 use will almost entirely move to HTTP/3, and HTTP/1.1 usage will decline but there'll still be plenty of it around.
      - cpeterso 6y ago
        Cloudflare’s Radar dashboard has recent measurements of HTTP/1, 2, and 3 traffic (near the bottom of the page): https://radar.cloudflare.com/ https://radar.cloudflare.com/ Right now, HTTP/1.x is 23% of traffic, HTTP/2 is 75%, and HTTP/3 is just 2%. Since this is about overall traffic (visible to Cloudflare), big sites like Google and Facebook dominate. All that HTTP/3 traffic is probably Chrome users watching YouTube.
  - jeffbee 6y ago
    I’m positive that an organization the size of Netflix which has dedicated itself absolutely to pushing 200gbps out of every individual box can figure out how to run HTTP 1.1 forever.
  - otabdeveloper4 6y ago
    "Whee, the magic number went up, better upgrade". Despite Google meddling with the standards, "HTTP/3" isn't HTTP.
anderspitman 6y ago
Interesting. Could this be mitigated at all with QUIC hardware? It seems like if the tradeoffs are this bad at scale Google wouldn't be pushing QUIC.
- drewg123 6y ago
  It can be mitigated, and companies are working on it. However, there is no NIC offering hardware QUIC protocol offload that I'm currently aware of. Part of what makes it challenging is one of the features of QUIC: That the protocol headers are themselves encrypted.
  - anderspitman 6y ago
    But do the headers have much to do with power efficiency? Seems like that would mostly hinge on encryption and state machine efficiency.
    - detaro 6y ago
      They have to do with the viability of offloading. You can't give the NIC an encrypted stream and have it handle the protocol stuff if it needs to put things inside the encryption to do so.
  - jeffbee 6y ago
    The Intel 700 and 800 series NICs have a DDP for QUIC.
me551ah 6y ago
Most applications on the internet are CRUD apps and not everyone is serving Netflix CDN data. TCP hasn't really changed much in the past 25 years due to routers on the internet being strict about its standards. Just look at 'TCP Fast Open', while it was supposed to drastically cut down on connection time in TCP, it can't be used on the internet due to routers behaving erratically. Apple tried it in production and then reverted it. Since QUIC is based on UDP, it doesn't face the same limitations that TCP does. TCP also suffers from head of line blocking, which QUIC also eliminates. It is also more secure due to header being encrypted, unlike TCP. Plus you also get multiplexing built-in. QUIC is far better than TCP and for serving websites & APIs, it is a much better alternative.
- rini17 6y ago
  But CRUDs also benefit from load balancing that is going to be much more expensive without hardware support. For the other problems, mostly satisfactory solutions exist.
  - me551ah 6y ago
    You can't really solve the head of line blocking problem in TCP
    - rini17 6y ago
      In CRUD apps all head of line problems are caused by abject bloat. Protocol isn't going to fix that.
amelius 6y ago
> So we'd need essentially 3x as many servers to serve via QUIC as we do to serve via TCP. That has real environmental costs. So do ads, and no government seems to care much.
- txdv 6y ago
  3x as many servers, it is probably going to be 3x more expensive. Money is something that everyone cares about
- mlthoughts2018 6y ago
  Such false equivalence. Everything costs resources to produce. Simply stating that it costs resources, or even just stating how much resources it costs, is completely meaningless on its own. Pixar movies cost resources, prolonging life with medical treatment costs resources, chocolate candy bars cost resources, operating the Olympic Games costs resources. In each case, various people believe the resource costs are worth it to get the thing that is produced. Same for bitcoin, same for ads. You can’t just say they cost resources. If you’re advocating these things shouldn’t exist, you either need to get into the specifics of why they offer value but just not enough to offset their resource costs, or why they don’t offer value, and persuade others to agree with you. Judging by the success of ads and bitcoin, I think, “these things consume resources” is a pretty weak and useless argument. It would not matter if the resource costs were astronomical or tiny, it’s a specious chain of reasoning in principle.
- detaro 6y ago
  Because other things with environmental cost (which are widely criticized for it!) exist, one shouldn't consider environmental costs?
infogulch 6y ago
How many of these downsides are inherent to the protocol compared to simply not being optimized yet? I get that a QUIC optimized NIC is unlikely to be right around the corner (these things take time), but is there anything that makes such a thing impossible?
vasilvv 6y ago
(I work for Google, opinions my own) For YouTube, the CPU cost of QUIC is comparable to TCP, though we did spent years optimizing it. [0] has a nice deep dive. Other CDN vendors like Fastly seem to have the similar experience [1]. I believe sendmmsg combined with UDP GSO (as discussed, for instance, in [2]) has solved most of the problems that were caused by the absence of features like TSO. From what I understand, most of the benefit of TSO comes not from the hardware acceleration, but rather from the fact that it processes multiple packet as one for most of the transmission code path, meaning that all per-packet operations are only invoked once per chunk (as opposed to once per individual IP packet sent). [0] https://atscaleconference.com/videos/networking-scale-2019-improving-quic-cpu-performance/ https://atscaleconference.com/videos/networking-scale-2019-i... [1] https://www.fastly.com/blog/measuring-quic-vs-tcp-computational-efficiency https://www.fastly.com/blog/measuring-quic-vs-tcp-computatio... [2] https://blog.cloudflare.com/accelerating-udp-packet-transmission-for-quic/ https://blog.cloudflare.com/accelerating-udp-packet-transmis...
- drewg123 6y ago
  GSO is better than nothing, but its still far more expensive than actual hardware TSO that swallows a 64KB send in hardware. At least it was when I did a study of it when I was at Google. This is because it allocates an skb for each MTU sized packet to hold the headers. I think it costs something like 1.5-2.0x real hardware backed TSO, mostly in memory allocation, freeing, and cache misses around header replication. Also, sendmmsg still touches data, and this has a huge cost. With inline kTLS and sendfile, the CPU never touches data we serve. If nvme drives with big enough controller memory buffers existed, we would not even have to DMA NVME data to host RAM, it could all just be served directly from NVME -> NIC with peer2peer DMA. Granted, we serve almost entirely static media. I imagine a lot of what YouTube serves is long-tail, and transcoded on demand, and is thus hot in cache. So touching data is not as painful for YouTube as it is for us, since our hot path is already more highly optimized. (eg, our job is easier) I tried to look at the Networking@Scale link, but I just get a blank page. I wonder if Firefox is blocking something to do with facebook..
- gcblkjaidfj 6y ago
  > For YouTube For the CDN/edge this is irrelevant and should not even be part of the discussion. It is obvious it will not change (or will be even better) for them. The comment you are replying talks exclusively about the "middle boxes". They will have a hardtime with quic, not matter what. (IMHO, a small price to pay, too bad it is something flawed like quic instead of a true distributed solution)
littlecranky67 6y ago
Not a big fan of QUIC (for other reasons) but I didn't consider your very valid points. Not be able to utilize TCP offload engines is another big disadvante of QUIC - but to be fair, I think we will see QUIC offload engines develop over time just as they did with TCP. Maybe those Intel/Altera or AMD/Xilinx deals will pay of for datacenter, for exactly those reasons (quick adaption of emerging network protocols by programming offload engines to FPGAs).
justicezyx 6y ago
> You loose 25 years of stateless hardware offloads done by the NIC. That's a feature. Otherwise the industry will die off slowly. Case in point, OpenStack first, and later kubernetes, were exactly this kind of innovation. It offers very tiny benefit and a whole slew of plugging points for the industry players to hook upon; but is beneficial enough for mostly everyone to be comfortably endorse a piece of technology that at the time were already starting to decaying... Of course in this case, OpenStack has been a much worse tech because of it was developed by people haven't been the mainstream...

jeffbee 6y ago

Horses for courses. Is anyone suggesting QUIC for gigantic bulk transfers? TCP isn’t great for latency-sensitive small transfers so we will have QUIC. QUIC will not be perfect for everything, like TCP is not perfect for everything.

drewg123 6y ago
I'm pretty sure YouTube uses QUIC for video streaming today, which is a similar workload to ours.
- jeffbee 6y ago
  I think that indicates more about the economics of YouTube than anything. Probably their frontend CPU costs would have to go up by a factor of ten thousand before it even began to approach their costs for compression and storage of user uploads.
- virattara 6y ago
  > I'm pretty sure YouTube uses QUIC for video streaming today Is there any media transfer protocol based on QUIC?
  - drewg123 6y ago
    I found an article from 4 years ago saying that 20% of youtube's traffic was QUIC: https://www.fiercewireless.com/wireless/youtube-driving-more-quic-based-traffic-mobile-vasona https://www.fiercewireless.com/wireless/youtube-driving-more...
  - detaro 6y ago
    Youtube is delivering video over HTTP. HTTP3, i.e. it is transported over QUIC.
  - elithrar 6y ago
    Lucas Pardue, who is the IETF QUIC WG chair, has a great talk from Mux's Demuxed conf last year on using QUIC as a transport for UDP streaming: https://www.youtube.com/watch?v=Zdkjd7-EWmQ&feature=emb_title https://www.youtube.com/watch?v=Zdkjd7-EWmQ&feature=emb_titl... This is not dissimilar from SRT.
- elithrar 6y ago
  Yep: YouTube currently delivers over gQUIC and h3 (IETF QUIC). We advertise the following alt-svc for most devices: > h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43" HTTP/3 (over IETF QUIC) is still not as widely deployed due to lesser client support, but that's growing as both Chrome[1], Firefox and Apple (in iOS Safari, when enabled) adopt it and the latest drafts. [1]: https://blog.chromium.org/2020/10/chrome-is-deploying-http3-and-ietf-quic.html https://blog.chromium.org/2020/10/chrome-is-deploying-http3-...
masklinn 6y ago
> Horses for courses. Is anyone suggesting QUIC for gigantic bulk transfers? Anyone who's looking to replace HTTP/1.1 and HTTP/2 with HTTP/3? AFAIK QUIC is completely integral to that, in order to avoid this issue you'll need a mixed deployment of HTTP/3 and an older version of the standard.
- anderspitman 6y ago
  I just hope we don't remove support for HTTP/1.1 from our stacks entirely. 2/3 are great for performance but an order of magnitude more complex.
  - M2Ys4U 6y ago
    I imagine that HTTP/2 use will almost entirely move to HTTP/3, and HTTP/1.1 usage will decline but there'll still be plenty of it around.
    - cpeterso 6y ago
      Cloudflare’s Radar dashboard has recent measurements of HTTP/1, 2, and 3 traffic (near the bottom of the page): https://radar.cloudflare.com/ https://radar.cloudflare.com/ Right now, HTTP/1.x is 23% of traffic, HTTP/2 is 75%, and HTTP/3 is just 2%. Since this is about overall traffic (visible to Cloudflare), big sites like Google and Facebook dominate. All that HTTP/3 traffic is probably Chrome users watching YouTube.
- jeffbee 6y ago
  I’m positive that an organization the size of Netflix which has dedicated itself absolutely to pushing 200gbps out of every individual box can figure out how to run HTTP 1.1 forever.
- otabdeveloper4 6y ago
  "Whee, the magic number went up, better upgrade". Despite Google meddling with the standards, "HTTP/3" isn't HTTP.

anderspitman 6y ago

Interesting. Could this be mitigated at all with QUIC hardware? It seems like if the tradeoffs are this bad at scale Google wouldn't be pushing QUIC.

drewg123 6y ago
It can be mitigated, and companies are working on it. However, there is no NIC offering hardware QUIC protocol offload that I'm currently aware of. Part of what makes it challenging is one of the features of QUIC: That the protocol headers are themselves encrypted.
- anderspitman 6y ago
  But do the headers have much to do with power efficiency? Seems like that would mostly hinge on encryption and state machine efficiency.
  - detaro 6y ago
    They have to do with the viability of offloading. You can't give the NIC an encrypted stream and have it handle the protocol stuff if it needs to put things inside the encryption to do so.
- jeffbee 6y ago
  The Intel 700 and 800 series NICs have a DDP for QUIC.

me551ah 6y ago

Most applications on the internet are CRUD apps and not everyone is serving Netflix CDN data. TCP hasn't really changed much in the past 25 years due to routers on the internet being strict about its standards. Just look at 'TCP Fast Open', while it was supposed to drastically cut down on connection time in TCP, it can't be used on the internet due to routers behaving erratically. Apple tried it in production and then reverted it. Since QUIC is based on UDP, it doesn't face the same limitations that TCP does. TCP also suffers from head of line blocking, which QUIC also eliminates. It is also more secure due to header being encrypted, unlike TCP. Plus you also get multiplexing built-in. QUIC is far better than TCP and for serving websites & APIs, it is a much better alternative.

rini17 6y ago
But CRUDs also benefit from load balancing that is going to be much more expensive without hardware support. For the other problems, mostly satisfactory solutions exist.
- me551ah 6y ago
  You can't really solve the head of line blocking problem in TCP
  - rini17 6y ago
    In CRUD apps all head of line problems are caused by abject bloat. Protocol isn't going to fix that.

amelius 6y ago

> So we'd need essentially 3x as many servers to serve via QUIC as we do to serve via TCP. That has real environmental costs. So do ads, and no government seems to care much.

txdv 6y ago
3x as many servers, it is probably going to be 3x more expensive. Money is something that everyone cares about
mlthoughts2018 6y ago
Such false equivalence. Everything costs resources to produce. Simply stating that it costs resources, or even just stating how much resources it costs, is completely meaningless on its own. Pixar movies cost resources, prolonging life with medical treatment costs resources, chocolate candy bars cost resources, operating the Olympic Games costs resources. In each case, various people believe the resource costs are worth it to get the thing that is produced. Same for bitcoin, same for ads. You can’t just say they cost resources. If you’re advocating these things shouldn’t exist, you either need to get into the specifics of why they offer value but just not enough to offset their resource costs, or why they don’t offer value, and persuade others to agree with you. Judging by the success of ads and bitcoin, I think, “these things consume resources” is a pretty weak and useless argument. It would not matter if the resource costs were astronomical or tiny, it’s a specious chain of reasoning in principle.
detaro 6y ago
Because other things with environmental cost (which are widely criticized for it!) exist, one shouldn't consider environmental costs?

infogulch 6y ago

How many of these downsides are inherent to the protocol compared to simply not being optimized yet? I get that a QUIC optimized NIC is unlikely to be right around the corner (these things take time), but is there anything that makes such a thing impossible?

vasilvv 6y ago

(I work for Google, opinions my own) For YouTube, the CPU cost of QUIC is comparable to TCP, though we did spent years optimizing it. [0] has a nice deep dive. Other CDN vendors like Fastly seem to have the similar experience [1]. I believe sendmmsg combined with UDP GSO (as discussed, for instance, in [2]) has solved most of the problems that were caused by the absence of features like TSO. From what I understand, most of the benefit of TSO comes not from the hardware acceleration, but rather from the fact that it processes multiple packet as one for most of the transmission code path, meaning that all per-packet operations are only invoked once per chunk (as opposed to once per individual IP packet sent). [0] https://atscaleconference.com/videos/networking-scale-2019-improving-quic-cpu-performance/ https://atscaleconference.com/videos/networking-scale-2019-i... [1] https://www.fastly.com/blog/measuring-quic-vs-tcp-computational-efficiency https://www.fastly.com/blog/measuring-quic-vs-tcp-computatio... [2] https://blog.cloudflare.com/accelerating-udp-packet-transmission-for-quic/ https://blog.cloudflare.com/accelerating-udp-packet-transmis...

drewg123 6y ago
GSO is better than nothing, but its still far more expensive than actual hardware TSO that swallows a 64KB send in hardware. At least it was when I did a study of it when I was at Google. This is because it allocates an skb for each MTU sized packet to hold the headers. I think it costs something like 1.5-2.0x real hardware backed TSO, mostly in memory allocation, freeing, and cache misses around header replication. Also, sendmmsg still touches data, and this has a huge cost. With inline kTLS and sendfile, the CPU never touches data we serve. If nvme drives with big enough controller memory buffers existed, we would not even have to DMA NVME data to host RAM, it could all just be served directly from NVME -> NIC with peer2peer DMA. Granted, we serve almost entirely static media. I imagine a lot of what YouTube serves is long-tail, and transcoded on demand, and is thus hot in cache. So touching data is not as painful for YouTube as it is for us, since our hot path is already more highly optimized. (eg, our job is easier) I tried to look at the Networking@Scale link, but I just get a blank page. I wonder if Firefox is blocking something to do with facebook..
gcblkjaidfj 6y ago
> For YouTube For the CDN/edge this is irrelevant and should not even be part of the discussion. It is obvious it will not change (or will be even better) for them. The comment you are replying talks exclusively about the "middle boxes". They will have a hardtime with quic, not matter what. (IMHO, a small price to pay, too bad it is something flawed like quic instead of a true distributed solution)

littlecranky67 6y ago

Not a big fan of QUIC (for other reasons) but I didn't consider your very valid points. Not be able to utilize TCP offload engines is another big disadvante of QUIC - but to be fair, I think we will see QUIC offload engines develop over time just as they did with TCP. Maybe those Intel/Altera or AMD/Xilinx deals will pay of for datacenter, for exactly those reasons (quick adaption of emerging network protocols by programming offload engines to FPGAs).

justicezyx 6y ago

> You loose 25 years of stateless hardware offloads done by the NIC. That's a feature. Otherwise the industry will die off slowly. Case in point, OpenStack first, and later kubernetes, were exactly this kind of innovation. It offers very tiny benefit and a whole slew of plugging points for the industry players to hook upon; but is beneficial enough for mostly everyone to be comfortably endorse a piece of technology that at the time were already starting to decaying... Of course in this case, OpenStack has been a much worse tech because of it was developed by people haven't been the mainstream...