20 ms·
A Technology Preview of Nginx Support for QUIC and HTTP/3
- baybal2 6y agoQuestion to ones following quic/http3 development, what does it bring over http2? From my point of view, http3 is more of a sidegrade and has no net benefit over http2. We can make TCP fast, very fast, and the use of UDP was not a prerequisite for any of its functionality. The "multistream" functionality of QUIC is mainly to benefit companies with big, highly loaded CDNs, and kind of obviates the economic point of squeezing multiple virtual streams into a single TCP connection as used in http2. By throwing away TCP, they are throwing away decades of optimisations, and hardware offloading that network hardware makers made to handle TCP well. If the talk was really about extracting single digit improvements from it, I think it would've made more sense to finally put SCTP, and DCCP to good use.
- Lichtso 6y agoTCP has some downsides which can not be fixed. And QUIC solves them by going below TCP. It just uses UDP because all other protocols are usually blocked and thus unusable. Transparent handover (when changing IP addresses) Multipath Multiplexing: No head-of-line blocking Faster handshake Enforce authentication and encryption (also of some metadata) Forward Error Correction and other improvements for wire less networks So the improvements are mostly noticeable for for mobile devices.
- the8472 6y agoThere are some mistakes here. MPTCP definitely exists and has been deployed, which also encompasses handover. Faster handshake is covered by TFO in principle. FEC was only a proposal and has been dropped.
- JoshTriplett 6y ago> MPTCP definitely exists and has been deployed, which also encompasses handover. I wouldn't be surprised if many stacks and networks are able to deploy QUIC more easily than MPTCP.
- baybal2 6y ago> UDP because all other protocols are usually blocked and thus unusable. Well, UDP does not fare better on a lot of ISPs.
- the8472 6y agoIf your ISP is blocking UDP and not just your router (which you can reconfigure if you want to use some other IP protocols) then you're arguably not on an Internet Service Provider at all, you're on some sort of WWW Service Provider.
- manquer 6y agoWell that applies to any protocol , ISPs don’t block anything , your company usually do though . UDP is very commonly blocked in almost every corporate environment I have seen, it is a pain get it enabled for applications like video streaming which benefits from UDP
- the8472 6y agoWell, then you don't have internet access in your company. Only some fossilized subset of internet. They have intentionally excluded themselves from any kind of progress, so arguing against any new kind of standard because some people deliberately exclude themselves from new standards is a non-argument in my opinion. It's like arguing against a developing new TLS version because people configure their firewalls to block any flavor of TLS they don't know.
- virattara 6y ago> TCP has some downsides which can not be fixed. I guess the problem is more with getting to adopt the fixes on a large scale even if the downsides of TCP are fixed. > It just uses UDP because all other protocols are usually blocked UDP is also blocked on several firewalls because of the fear of UDP flood attacks, QUIC fallbacks to TCP in such cases.
- starfallg 6y agoThose aren't TCP downsides per se, just different design decisions based on different requirements back when TCP was standardised. QUIC eliminates the TLS handshake time on top of TCP. You can't really improve on the TCP 3-way handshake - you need a minimum of 3 messages to ensure that both sides have agreed to a connection. You also have to question whether FEC should be implemented in layer 4, rather than implemented over the lower layers only as when needed. It seems to be terribly inefficient to implement FEC end-to-end when the packetloss is due to the last mile (or several meters for WiFi). To me, QUIC seems to be the wrong approach to the problem. If you want a connection-oriented protocol with the advantages above, it's much better just to push through a new version of TCP with the extensions. Sure, it's going to take longer than approving a new protocol over UDP, but the potential benefits are substantial. On the other hand, there is an argument that by pushing through QUIC, there is pressure to improve TCP in those ways. We'll have to see how this pans out.
- ricardobeat 6y agoThere is no way to 'push a new version of TCP', I believe that's one of the strong motivations for these protocols. Billions of devices and the world's entire network infrastructure would need to be upgraded, you'd only start seeing decent support after a decade. So you have to work with what we have - TCP and UDP.
- clarry 6y agoGiven how IPv6 went, a decade seems optimistic.
- fiddlerwoaroof 6y agoIPv6 had a really bad case of second system effect, though.
- starfallg 6y ago>There is no way to 'push a new version of TCP' Yes there is. TCP has a variable options range of up to 320 bits starting at the 20th octet to handle exactly feature extensions (This is exactly how Multi-Path TCP is implemented). There are also 3 reserved bits set for future use (13th octet), which can also form the basis of even larger extended option ranges for stuff like TLS negotiation.
- dehrmann 6y agoI'm not sure if forward error correction works with UDP. UDP has a checksum field that's optional in IPv4, mandatory in IPv6 (why IPvAnything dictates anything in the transport layer is beyond me). Assuming you treat the checksum as absent and set it to zeros, it could still get added at any hop, and if that gets corrupted, the packet would be dropped. If routers either ignore the checksum or never add it in, it could work.
- p1mrx 6y agoUDP's checksum depends on information from the IP header, for both IPv4 and IPv6: https://en.wikipedia.org/wiki/User_Datagram_Protocol#Checksum_computation https://en.wikipedia.org/wiki/User_Datagram_Protocol#Checksu... Omitting the UDP/IPv4 checksum is permitted because the IPv4 header has its own redundant checksum, but omitting the UDP/IPv6 checksum would leave the IPv6 header with no checksum at all, which is why that's not permitted. https://en.wikipedia.org/wiki/UDP-Lite https://en.wikipedia.org/wiki/UDP-Lite lets you checksum the UDP/IP header plus an arbitrary (possibly zero-length) prefix of the payload, but I'm not sure if people actually use that.
- comex 6y agoI believe the "errors" in this context are generally not actual corrupted packets, which are rare, but dropped packets seen in the context of a larger data transmission.
- nyanpasu64 6y agoIs transparent handover in use in QUIC or HTTP/3 today? Can you start a large download or SSH session on Wi-Fi, switch to mobile data, and not break the connection?
- Matthias247 6y agoIt's supported by the protocol. But I'm not sure if it's actually supported by anyone who deployed the protocol. It requires to actually support connection ID migration in the library (which not all of them do), as well as to support it on infrastructure (e.g. L4 load balancers): If those receive a packet belonging to the same connection, but having a new source IP address (due to client having migrated), they still need to route things to the same server which has all the state about the connection. Therefore the whole thing is rather hard to implement.
- wwright 6y agoI believe it also handles degraded connections (ie mobile devices and remote/rural/developing areas) much better than HTTP 2.
- virattara 6y agoYes because it overcomes Head of Line blocking, a major problem in HTTP/2.
- regecks 6y agoIn HTTP/2, because all the streams are muxed over a single TCP connection, packet loss really screws things up - every stream grinds to a halt. It is worse than HTTP/1.x in that case. I observed it myself when Fastmail was having some networking troubles - one response not making it through caused every other request to stall indefinitely. HTTP/3 being over UDP will hopefully fix that, while keeping the benefits of HTTP/2.
- baybal2 6y agoYes, it being muxed into a single TCP stream was marketed as an advantage originally, as for almost all mainstream OSes, and network hardware, handling fever TCP connections faster is easier than more more slower connections. Even if a packet loss was to occur, it would still have a net benefit over pounding a CDN with countless individual requests.
- vbezhenar 6y agoProblem with multiple TCP connections is slow TCP startup. When TCP connections are established and "warmed up", multiple TCP connections are almost always work faster and more reliable. That was exploited back in the day by download managers which opened multiple connections to the save resource and managed to significantly increase download speed.
- derefr 6y ago> That was exploited back in the day by download managers which opened multiple connections to the save resource and managed to significantly increase download speed. Even back then, that wasn't anything to do with multiple TCP connections being fundamentally faster somehow. The speed increase achieved by download managers comes down to: 1. DNS resolution round-robin-ing you to different CDN nodes, such that you're actually making requests to different hardware that each has its own NIC to separately saturate; where if the each server NIC can do 1Mbps, and your link can do 2Mbps, then you need two parallel chunk downloads from separate servers to saturate your link. (This is the same fundamental reason downloading e.g. an OS installer through a torrent is usually faster than downloading it through HTTP from the OS maker's CDN.) 2. Intentional per-connection download-rate throttling applied by many servers (especially those of "download sites" like CNet, which "back in the day" were one of the main places people would experience large downloads); where making multiple concurrent HTTP range-requests to the server at once was a way to bypass the server's throttling. You don't see this advantage any more, as these days the servers of "download sites" have enough smarts to dynamically aggregate downlink quota between nodes in a table keyed by peer IP, such that N concurrent downloads from the same site are always each throttled to 1/Nth the rate.
- the8472 6y agoWhile we can achieve most improvements with TCP too (assuming you're on linux and have a recent kernel) there are a few things that QUIC offers on top of those improvements 0RTT session resumption, while TFO could offer this too most people don't enable it due to broken middleboxes. Header encryption, which hopefully prevents future ossification due to those broken middleboxes. Multiple independent streams within a connection where each stream only suffers from its own packet losses instead of all streams being affected together. This provides a small latency benefit. TCP SACKs and fast retransmits should paper over most of this latency, but not all of it. QUIC also offers more SACK ranges than TCP SACK which might help on links with high, non-bursty packet loss. Then there's path migration, e.g. when switching from mobile to wifi. This could also be achieved with MPTCP, but that's still making its way into the kernel.
- baybal2 6y ago> broken middleboxes I think this notion that you need to work around them, and accommodate, rather than to ignore them, and do things right needs to be thrown out. All those middleboxes are non-essential, and not are not inherent part of the original design. I especially don't understand google engineers who decided on TLS 1.3 version hack when the very same situation has repeated itself with 1.3, and many new TLS extensions. The logic fail for example with IPv6, a giant amount of IPv6 software/hardware is plainly unusable, but this is not a reason to try to hack around it, by trying reimplementing IPv6 in v4.
- virattara 6y agoThe problem with making changes to TCP or using SCTP is that it would take a lot of time for the implementations to be rolled worldwide, one of the main reasons why is that so is that these protocols are implemented in kernel space any change would take years before its widely adopted over the network. TCP fast open is an example of this. For this reason, QUIC's functions are implemented in the user space over UDP(no changes made to UDP). But this has also created problems for QUIC as it turns out that QUIC has significantly higher CPU usage compared to TCP. All of this well explained in this doc: https://http3-explained.haxx.se/ https://http3-explained.haxx.se/
- baybal2 6y agoYes, the biggest thing about Quic deployment is that UDP other than most well known ports is being firewalled almost universally on low end ISPs. In the end, a QUIC probe, and a subsequent TCP fallback would be worse than well done TCP, and no better than MPTCP/TFO/SCTP/DCCP (all of which were supported by mainline OSes for a long time)
- amaranth 6y agoI believe macOS/iOS is the only major OS with MPTCP right now. Some others have patches available or are in the process of landing their own implementations but that'll likely take a bit and still doesn't include Windows. SCTP is supported by most kernels but usually can't get out of your local network before something either explicitly blocks it or can't understand it and so drops it. DCCP won't even work on your network, unless your network is only Linux machines. TFO is blocked by enough junk middleboxes that, on average, it's less efficient than just not trying to use it. Your only real choices are TCP without any extensions created in the past 20 years or UDP. Some firewalls will block UDP but at least it'll survive the open internet so that's a problem you can get IT to fix.
- the8472 6y agoI'm not sure how that really is a problem. I am writing this from a box that has IPv6 and most of the latest TCP stuff including BBR, ECN and TFO. It took a while until we got to that point, but here we are. In the past we were able to wait too, why not now?
- scurvy 6y agoI have that same question, "Why and who does this benefit?" It seems like everything runs in the cloud these days and is fronted by huge CDN's. The performance on 80%+ of websites should be really fast already due to local proximity wins (speed of light/bits). It looks like there's a megacloud provider location just about everywhere these days with more on the way. This seems like a lot of energy spent to make already fast things just a tiny bit faster. Are there any real world estimates or projections for improvement? Like, how quickly will that Facebook page load or show those Google (ad) search results? Things feel largely fine, yet here are more "improvements". And once we have these gains, they will probably be lost in bloated javascript or app layer inefficiencies.
- srj 6y agoHTTP2 has multiplexing implemented on top of TCP, which is not multiplexed. This means one stream can be delayed due to head-of-line blocking from another stream getting retransmits. Changing TCP to support this is complicated due to transparent proxies that would break, and difficult to upgrade segmenter implementations. The benefit is not limited to CDNs, any long fat network where we have high throughput and high latency (e.g. most cellular connectivity) should see improvement. I would guess that CDNs may benefit less actually since they tend to terminate near the user and would generally have lower RTT. A typical modern web page has many resources to be fetched, and by making them independent the user experience is likely to be significantly better. Disclaimer: work at google, not on QUIC but am familiar with the QUIC project and team.
- drewg123 6y agoBy throwing away TCP, they are throwing away decades of optimisations, and hardware offloading that network hardware makers made to handle TCP well Indeed. I work at Netflix on optimizing cpu efficiency on our Open Connect CDN nodes, largely to reduce power use and capital expenses. We use FreeBSD, ngnix & TCP, and make heavy use of offloads like async sendfile(), TSO, LRO, kTLS and more recently hardware kTLS offload. Right now, I have a single socket 32c/64t AMD Rome server delivering over 350Gb/s of real Netflix customer traffic. This traffic is all TLS encrypted, and is served across hundreds of thousands of TCP connections. From measurements we've done, current QUIC would cost about 3x as much as TCP when using software crypto. So my back-of-the-envelope guess is that this box would do about 77Gb/s with QUIC (230Gb/s is the limit when disabling hardware TLS offload and using software crypto). Are the benefits of QUIC really worth an a 4x increase in the amount of energy required per stream? Once QUIC has optimizations similar TCP in place, the story will obviously be different. But we're not there yet.
- vlovich123 6y agoIs it worth it now for your use-case? Maybe not. Is it useful for others? Probably. I’m also curious that Google chose to deliver QUIC for everything when they can including YouTube. I guess the cost tolerance for them is there because they believe they improve the experience enough that it’s a net revenue generator.
- drewg123 6y agoA lot of it depends on whether or not the content you're delivering is a static file, or if you're transcoding something. When its a static file, your job is "easier" in that you can use sendfile, and (with hardware kTLS offload) avoid having the CPU touch any data being sent (the Netflix case). When you have a gigantic long tail, and are transcoding a lot of traffic, then optimizations like sendfile and kTLS hardware offload matter a lot less. I imagine Google falls more on one side of the spectrum, and we fall on the other.
- JoshTriplett 6y ago
- devwastaken 6y agoTcp packets require syn/ack, which makes them work quite poorly on wireless networks where devices are receiving the majority of packets. What happens is now your tcp acks go through the roof, because sending that packet from devices may not be received. But syn/ack is only for data you want to make sure was received. Things like video and audio don't need that to view the content. So it's one big use is it's a protocol that works over wireless much better for content viewing. You can test this yourself with a basic server and a mobile phone over WiFi at distance. Over http/2 video will cut out and have a difficult time loading. Over quic it'll just keep going.
- starfallg 6y agoQUIC has ACKS also. It is a connection oriented protocol that can guarantee delivery of packets. QUIC also has a three-way handshake (on new conntections). The equivalent of TCP's SYN, SYN+ACK, ACK, but stacked with the TLS negotiation at the same time. Also, SYN is only used during this handshake in TCP.
- xg15 6y agoNot an expert, but one interesting feature is the ability to keep a connection open across network changes. In TCP and UDP, connections are identified using local IP address/local port/remote IP address/remote port tuples. This means, if my IP address changes, (e.g. because I connected to a new WiFi network) all TCP connections that were active before must be abandoned. With QUIC, supposedly, an identifier separate from IP addresses is used, so there are ways how a connection could be resumed. (I'm fuzzy on the details though)
- zelly 6y agoIt's a feature enabled by UDP. QUIC keeps track of a connection uuid as an identifier rather than an IP.
- danudey 6y agoOne thing I read about a while ago (can't find the link at the moment) is that QUIC doesn't play well with firewalls that do QoS (at least, by default), because a node will send as much UDP data as it likes and the system has to deal with it as it comes in; with TCP, you can adjust the window size, etc. to slow the transmission rate, and so on, but current firewalls don't have any (practical?) way to manage this. I solved this issue at my last company by just blocking 443/UDP at the firewall so that clients (i.e. Chrome) would fall back to HTTP/2.
- chucke 6y agoHumm, will nginx enable header compression this time? Their HTTP/2 module disabled the hpack dynamic table, as I recall. Will they serve us again a poor-man QUIC and tell us "it's all fixed in the paid version" again?
- thresh 6y agoIs HTTP2 in the paid version different than the opensource implementation?
- taf2 6y agoIIRC wasn’t this due to security exploit (information leak) with compression enabled you can figure out what is in the headers by doing some kind of observation of the changes in byte sizes?
- patrickmcmanus 6y agoyou're thinking of an exploit in spdy (the h2 predecessor) in which the headers were just run through the same gzip context. The HPACK format in h2 and h3 is meant to remove those oracles. (though it is less effective bytewise than gzip).
- Neil44 6y agoOpenLitespeed has QUIC support too, and it can read apache configs.
- mobilio 6y agoI've used them and works great. Also have a plugins for popular CMS platforms (WordPress, Joomla, Drupal, Magento, OpenCart, PrestaShop, MediaWiki, etc.) that REALLY helps.
- l4hel 6y agoGoogle ruined the simplicity and orthogonality of the internet while we all were there looking. There is nothing of the original design grandiosity of the first batch of internet protocols here. It's just engineering work of sacrificing every elegance and modularity to seek some percentage (not order of magnitude) performance gain.
- vbezhenar 6y agoYou're free to use HTTP/1.1. I think that it'll be supported forever both on clients and servers.
- benkuhn 6y agoSimplicity, orthogonality, elegance, modularity, etc. are useful when you want to build lots of different things easily. When you're building one single thing that's used by 4.6 billion people, it turns out that percentage optimizations matter! (I work for a company that built a shitty half-baked homegrown QUIC equivalent because in rural Ethiopia, HTTPS handshakes were so slow that they literally just didn't work. Glad that Google is optimizing our percent-of-a-percent use case!)
- baybal2 6y agoThe thing is, what Google does often doesn't work. Unsound hacks that kind of work "acceptable" in A/B test telemetry and slowly break in real life from inherent design deficiency, are almost always worse than something saying from the start "will not work on bugged os/hardware version, but work really well on standard compliant ones" The TLS 1.3 hack a Google engineer has forced through IETF is now backfiring for example. They did it to hack around a certain brand of middleboxes, but the hack instead broke few other ones, and embedded http servers. They may well errata it, and go back to normal versioning in 1.4, despite putting it on paper in 1.3 that the hack is here permanently.
- tialaramex 6y ago> The TLS 1.3 hack a Google engineer has forced through IETF is now backfiring for example. How is it "backfiring"? It seems to be working for billions of people. If you've got a non-compliant TLS implementation that broke you get to keep both halves, good luck with that.
- garganzol 6y agoI'm very skeptical about QUIC/"HTTP3". TCP works extremely well already. Yes, one can do kind of better for some particular workloads but I have yet to see a successful implementation that goes ahead of TCP. Take a Remote Desktop protocol used in Microsoft Windows. It can work over TCP but recent revisions tend to automatically switch to UDP. And know what? They are not reliable to the point that customers have to turn the UDP layer off. TCP gives a slightly worse latency but it is much more reliable and thus usable. Thanks there is a Group Policy for that. I'm not even talking about Google as a company who constantly tries to attack the network infrastructure with its variant of EEE (Embrace, Extend and Extinguish). What's the end game? The crippled protocols worldwide imposed by the ad casino company? No, thank you. Internet must remain free of all of that.
- floatboth 6y ago> They are not reliable to the point that customers have to turn the UDP layer off I guess they didn't implement reliability then. This has nothing to do with QUIC. > Embrace, Extend and Extinguish Yeah, working on an IETF standard is EEE now? LOL
- garganzol 6y agoEEE 2.0. The social aspect is just an antitrust indulgence. Seems to be working fine for hoi polloi. For now, at least.
- fulafel 6y agoIs it memory safe?
- xfalcox 6y agoI have already tested the quiche patch from Cloudflare, and even reported a bug on it that was fixed. Can someone comment on differences between the two patches?
- mobilio 6y agoCF patch is "unofficial", nginx is official. There isn't huge difference between them from technical point of view. But it's your decision what of them you will use.
- secondcoming 6y agoIt's really interesting that they chose to use rust
- mobilio 6y agoActually seems that someone in CF really love Rust: https://blog.cloudflare.com/boringtun-userspace-wireguard-rust/ https://blog.cloudflare.com/boringtun-userspace-wireguard-ru... They have make WireGuard implementation in Rust. Here is original WG-RS: https://git.zx2c4.com/wireguard-rs/about/ https://git.zx2c4.com/wireguard-rs/about/ But CF is using their own implementation for WARP: https://blog.cloudflare.com/1111-warp-better-vpn/ https://blog.cloudflare.com/1111-warp-better-vpn/ "We built WARP around WireGuard, a modern, efficient VPN protocol that is much more efficient than legacy VPN protocols." Here is other repo about projects written in Rust: https://github.com/cloudflare?q=&type=&language=rust https://github.com/cloudflare?q=&type=&language=rust
- Matthias247 6y agoDepends on what you mean with "technical". Both implement the QUIC specification for obvious reasons, but there is a fair amount of difference in how they for example handle reordering of frames, prioritization of outgoing frames and some other details which will have a yet-to-be-determined impact on performance. Besides that - one of the implementations has unit-tests - one of the implementations is in written in a by-default memory safe programming language - one of the implementations might become available to Nginx users without having to invest extra work
- zelly 6y agoLooking forward to not every browser implementing it and having to implement WebSockets in addition anyway