7 ms·
Employing QUIC Protocol to Optimize Uber’s App Performance
- internals 7y agoWhat a great case study. Successfully shifting 80% of mobile traffic to QUIC for a 50% reduction in latency is amazing. QUIC and the ongoing work with multipath TCP/QUIC will be huge QoL improvements for mobile networking.
- nhf 7y agoCan't wait for the IETF standardization process to finish up! https://quicwg.org/base-drafts/draft-ietf-quic-transport.html https://quicwg.org/base-drafts/draft-ietf-quic-transport.htm...
- kevin_thibedeau 7y agoYou'll just get more JavaScript piled on as managers insist on having X monetization feature deployed.
- api 7y agoThe other awesome thing about QUIC is that it encrypts almost everything including header information, making middlebox traffic shaping worthless and demoting middleboxes in general.
- drewg123 7y agoIt also makes hardware offloads like TSO and LRO impossible, and increases cost-per-byte served by a factor of 4 or more. So if you have infinite CPU to throw at QUIC and/or low bandwidth or connection targets, its great. If you are concerned at all about server-side efficiency, its terrible. FWIW, I work on the Netflix CDN, and specialize in server-side efficiency; we have had 100G flash CDN nodes for years serving at 90G+ in production. None of that would be possible with QUIC as it stands. I suspect our max B/W on these machines would drop from ~95Gb/s to 20Gb/s or less if we were to switch to QUIC.
- scott00 7y agoDoes the protocol actually make it impossible, or is it just not implemented by current OS/hardware?
- vlovich123 7y agoDefinitely just that the HW hasn't caught up. https://www.netdevconf.org/0x13/session.html?talk-quic-offload https://www.netdevconf.org/0x13/session.html?talk-quic-offlo...
- api 7y agoI hear you and those are valid points. Don't blame QUIC. Blame ISPs and middlebox makers for abusing their (literal network-wise) position and breaking the end-to-end principle. History has shown that we just have to encrypt everything and quite literally cut out the middleman/box.
- panarky 7y agoExperiment 1 While we used the NGINX reverse proxy to terminate TCP, it was challenging to find an openly available reverse proxy for QUIC. We built a QUIC reverse proxy in-house using the core QUIC stack from Chromium and contributed the proxy back to Chromium as open source. Experiment 2 Once Google made QUIC available within Google Cloud Load Balancing, we repeated the same experiment setup with one modification: instead of using NGINX, we used the Google Cloud load balancers to terminate the TCP and QUIC connections... Since the Google Cloud load balancers terminate the TCP connection closer to users and are well-tuned for performance, the resulting lower RTTs significantly improved the TCP performance.
- 7ewis 7y agoSo is this essentially HTTP/3?
- layoutIfNeeded 7y agoWhy 3? Just call it HTTP-as-defined-by-Google. It worked for HTML!
- wmf 7y agoYes. https://tools.ietf.org/html/draft-ietf-quic-http-20 https://tools.ietf.org/html/draft-ietf-quic-http-20
- tomchristie 7y agoHTTP/3 is slated as roughly HTTP/2 over QUIC but with some differences as a result. https://tools.ietf.org/html/draft-ietf-quic-http-20#appendix-A https://tools.ietf.org/html/draft-ietf-quic-http-20#appendix... I was dissapointed that the article didn't get into any details about the HTTP-level differences, but I think that's basically because they didn't need to know or care about the transport at that level since they were using Google's "cronet" library to dispatch the requests, and Google Cloud Load balancers as the QUIC termination point, so nothing on their ends actually deals with the neccessary differences from HTTP/2.
- ctime 7y agoThis YouTube video does a great job illustrating how well HTTP/2 works in practice. https://www.youtube.com/watch?v=QCEid2WCszM https://www.youtube.com/watch?v=QCEid2WCszM A lesser known *ownside to HTTP/2 over TCP solution was actually caused by one of the improvements - a single reusable (multiplexed) connection - that could end up stalled or blocked due to network issues. This behavior could go unnoticed over the legacy HTTP/1.1 connections due to browsers opening a hugh number of connections (~20) to a host, so when one would fail it wouldn't block everything.
- cpitman 7y agoMy team ran into a surprising behavior for HTTP2. Browsers decide whether to reuse a connection not based on the original domain the connection was made to, browsers base the decision on the domains that the returned certificate is signed for! Our current load balancer doesn't support HTTP2 end-to-end (and we are dong gRPC), so we are load balancing TCP connections to the individual instances. And for certificates, we would use SANs to reduce the number of certificates being requested. Put those two together, and browsers will assume that the first connection they make to serviceA.example.com can also be used for serviceB.example.com. Oops! TLDR, certificates for HTTP2 need to be unique to each endpoint that terminates a browser connection.
- xyzzyz 7y agoSNI spec says: If the server_name is established in the TLS session handshake, the client SHOULD NOT attempt to request a different server name at the application layer. Yeah, looks like the browsers are allowed to do that, although it's not recommended.
- cpitman 7y agoThe HTTP2 spec actually allows it: Connections that are made to an origin server, either directly or through a tunnel created using the CONNECT method (Section 8.3), MAY be reused for requests with multiple different URI authority components. A connection can be reused as long as the origin server is authoritative (Section 10.1). For https resources, connection reuse additionally depends on having a certificate that is valid for the host in the URI. The certificate presented by the server MUST satisfy any checks that the client would perform when forming a new TLS connection for the host in the URI. There is a way to respond with an error code when a server receives a request for the wrong domain, but it seems like a bad idea to depend on it (because it means lots of failed requests that could be avoided with better certificate management): A server that does not wish clients to reuse connections can indicate that it is not authoritative for a request by sending a 421 (Misdirected Request) status code in response to the request (see Section 9.1.2). https://http2.github.io/http2-spec/#rfc.section.9.1.1 https://http2.github.io/http2-spec/#rfc.section.9.1.1
- OrgNet 7y agoThis kind of latency improvement only matters if they are planning to do auto-pilot from the cloud? (that would be crazy, especially if they don't have a fallback)
- eropple 7y agoDoes it necessarily imply that, though? applications that are snappy and responsive tend to improve users' impression of the app and probably (it is/was true for browsers when I worked in this realm, I have no reason to doubt that it's true for mobile today) improve user engagement.
- Phlarp 7y agoI think the prevailing wisdom is that engagement suffers more from latency on mobile.
- wenttomarket 7y agoTry requesting an Uber in Latam on a low end android device. You’ll revise this opinion quickly.
- OrgNet 7y agothat has nothing to do with latency... latency is usually less then 1 second... no matter which phone you are using.
- the8472 7y agoIsn't TLP[0] supposed to fix the largest cause (tail losses) of this issue? It should result in retransmits far sooner than the 30 seconds they mention. > Recently developed algorithms, such as BBR, model the network more accurately and optimize for latency. QUIC lets us enable BBR and update the algorithm as it evolves. Again this is available for TCP in recent linux kernels[1]. And it's sender-side, so it should be unaffacted by ancient android devices. Are they using ancient linux kernels on their load balancers? Or are the sysctl knobs for these features turned off in some distros? [0] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6ba8a3b19e764b6a65e4030ab0999be50c291e6c https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... [1] https://kernelnewbies.org/Linux_4.9#BBR_TCP_congestion_control_algorithm https://kernelnewbies.org/Linux_4.9#BBR_TCP_congestion_contr...
- lossolo 7y agoBBR in kernel works for TCP because it's used for TCP congestion control. QUIC is built on top of UDP and there is no congestion control used for UDP, you implement it in your application/protocol built on top of UDP. From what I remember QUIC implementation supports BBR and Cubic as congestion control mechanisms.
- v5c6 7y agoCorrect. QUIC also implements TLP. Main advantage is deployability since this is a user-space solution.
- toast0 7y agoIf you need it on the client side, it's basically never going to happen on Android. Between manufacturers that never update and Google doesn't seem to care (they never enabled path MTU probing, even though it's been in all the kernels they shipped, and Apple uses it aggressively to great success in terrible networks)
- jefftk 7y agoI'm surprised the "alternatives considered" section doesn't have a "write something custom for core functionality using UDP". I would be curious to read why the decided not to go that way, given their scale and the potential gains from not using a general-purpose protocol. (Something like, make the entire standard journey from opening to the app to requesting a car over something custom, and then leave the rest of the app using TCP)
- telotortium 7y agoI'm assuming their thought process went like this: "we want to optimize a workflow which today goes over HTTP RPC/REST for mobile networks where the retry behavior of TCP at the low level is suboptimal. QUIC already exists, and we don't have to figure out our own solutions to firewalls, security, HTTP semantics. Also our webapps use the same or similar API endpoints, so the less effort spent writing new routing, monitoring, etc., the better. Oh, it works really well. Awesome, avoided 6-24 months debugging a custom protocol over mobile networks around the world that we don't control."
- esaym 7y agoI recently moved and got internet with Spectrum. A 200/10 service yet my upload speeds were rarely above 5mbit. This was a business account with some web and dev servers behind it. I didn't even try to call customer service... With a little more testing using UDP, I could see I was getting very spotty packetloss (<0.5%). I'd never tried changing the TCP algo before but I knew random packetloss is normally interpreted as congestion and hence causes a speed backoff. I tried all of the ones available at the time but the one that stood out not only in performance but also simplicity was TCP-Illinois[0]. The stats provided by `ss -i` also seemed the most accurate with TCP-Illinois. I force enable it on every machine I come across now. 0:https://en.wikipedia.org/wiki/TCP-Illinois https://en.wikipedia.org/wiki/TCP-Illinois
- nullwasamistake 7y agoI highly recommend BBR congestion control if your router supports it.
- m3kw9 7y agoTcp was build for the internet long ago, even though there are changes added, the architecture of the protocol make it hard to do anything drastic. With UDP because it is so simple, you can basically create a new protocol on top, inside the payload and emulate TCP if you wanted to
- sly010 7y agoI wish the mandated minimum MTUs of IP were just a bit bigger. Ubers traffic must be so transactional, they could really just use individual UDP packets for most messaging.
- jacob019 7y agoipv4 - 576 bytes ipv6 - 1280 bytes
- mruts 7y agoThat should be sufficient for uber, I think.
- sly010 7y ago>>> print len("Dear Uber, I need a ride from 26th and 6th Street, Brooklyn to 999 Broadway. I am on the south side of the road. It's 6:99PM UTC, I have 55% battery, my public key is b82c9238e847b. I like Jazz. My phone is a Pixel 3, I am using app version 4.22.4, but you probably already knew that. I am running out of things to add, so here is some personally identifiable information... and we still have plenty of bytes left") Edit: formatting
- boulos 7y agoYou forgot all the HTTP headers :).
- heinrich5991 7y agoFile "<stdin>", line 1 print len("Dear Uber, I need a ride from 26th and 6th Street, Brooklyn to 999 Broadway. I am on the south side of the road. It's 6:99PM UTC, I have 55% battery, my public key is b82c9238e847b. I like Jazz. My phone is a Pixel 3, I am using app version 4.22.4, but you probably already knew that. I am running out of things to add, so here is some personally identifiable information... and we still have plenty of bytes left") ^ SyntaxError: invalid syntax
- 7y ago
- ssvss 7y agoI thought DDOS prevention was difficult with udp, compared to TCP. Is it not the case anymore. Does cloudflare provide DDOS prevention for QUIC/UDP.
- mirashii 7y agoUDP DDOSes are hard to prevent against because they're volumetric in nature. Generally, they rely on UDP's statelessness, poorly configured networks, and various applications that respond with significantly more data than requested over UDP. These types of DDOSes are useful against any service that's running on TCP or UDP, and Cloudflare has protected against them all along.