5 ms·
Investigating the impact of HTTP3 on network latency for search
- giuliomagnifico 3y agoVery interesting reading, thanks. I just wonder why this geographic differences from one region to another.
- abujazar 3y agotl;dr; HTTP3 provides a little improvement, but optimizing search speed is much more important in this case.
- Thiez 3y agoAlthough HTTP3 was a significant win for people with lossy connections and high latency.
- abwizz 3y agolike mobile
- x-complexity 3y ago> tl;dr; HTTP3 provides a little improvement, but optimizing search speed is much more important in this case. The article itself shows otherwise: https://dropbox.tech/frontend/investigating-the-impact-of-http3-on-network-latency-for-search#-comparing-the-results:~:text=22%25%20at%20p95).-,HTTP3%20vs.%20HTTP2,Asia,-p25 https://dropbox.tech/frontend/investigating-the-impact-of-ht... ------------------------------------------------------------------------------------ |HTTP3 vs. HTTP2 | North and Central America | Europe | Asia | |p25 | -3.20ms / -6% | -2.34ms / -2% | -3.73ms / -2% | |p50 | -4.21ms / -5% | -3.84ms / -3% | -5.12ms / -2% | |p75 | -9.03ms / -8% | -11.1ms / -6% | -15.0ms / -4% | |p90 | -44.9ms / -17% | -47.3ms / -13% | -77.3ms / -14% | |p95 | -118ms / -22% | -141ms / -21% | -200ms / -22% | ------------------------------------------------------------------------------------ While the average case was negligible, the edge cases fared significantly better because of HTTP3's removal of the head-of-line blocking in TCP.
- clementmas 3y agoCan we use HTTP3 today? Is it widely supported?
- unmole 3y agoYes. Google and Facebook have deployed HTTP/3 across their properties for quite some time now. Support in popular servers is not mature but if you're using a CDN like Cloudflare, it's trivial to enable HTTP/3 for your users.
- noway421 3y agoWondering the same- how’s the compatibility with very restrictive networks? Say a university/corporate network that only allows port 443 and DPI’d 80 egress. QUIC won’t pass through that since it’s UDP. Can server and client reliably negotiate a fallback to http/2 in such a case?
- oefrha 3y agoHTTP/1.1 isn’t going anywhere in any popular server, so yeah, HTTP/2 fallback isn’t a problem. (HTTP/1.1 is basically required even if you just want to serve an https redirect on port 80, since h2 requires TLS. The clear text h2c protocol has no adoption AFAICT.)
- ignoramous 3y agoYep, just like IPv4 didn't, HTTP/1 and H2 aren't going anywhere anytime soon.
- wraptile 3y agoIPv4 is not going away because it's valuable (as in limited supply creates a market). HTTP 1.1 is already unusuable in modern web as you'll be instantly blocked by cloudflare and gang. HTTP2 is likely to follow in near future as replacing it will be much easier than replacing http1.1.
- re-thc 3y ago
- Wagthesam 3y agoOne thing I'm wondering about is that http3 seems to be from end user to search, which is a general serving problem and not really about search. Do dropbox clients call the search service directly?
- Wagthesam 3y agoI have wondered recently about head of line blocking's impact to latency at AZ level within AWS. How reliable are TCP connections within the same datacenter and do these issues noticeably impact performance at the tail end?
- diroussel 3y agoMaybe there are some answers to that here https://aws.amazon.com/blogs/hpc/in-the-search-for-performance-theres-more-than-one-way-to-build-a-network/ https://aws.amazon.com/blogs/hpc/in-the-search-for-performan... Where AWS explain why TVP isn’t good enough for them and why they are developing a replacement.
- rimtutituki 3y agoJust wandering, what http3 solutions are on Linux side. Can IPVS be http3 compatible at all?
- jeroenhd 3y agoWhy not? As long as you can send and receive UDP traffic, there's no reason why you can't do http/3. Caddy and HAProxy both support the protocol. nginx doesn't have it in the public stable version yet, though there are preview packages available for modern server platforms.
- rimtutituki 3y agoYeah. Should test it, would be interesting to see if DR is working with UDP balancing (via IPVS).
- petecooper 3y ago>nginx doesn't have it in the public stable version yet It's near, roadmapped for 1.25: https://trac.nginx.org/nginx/roadmap https://trac.nginx.org/nginx/roadmap
- bbss 3y agoHTTP/3/QUIC supports migrating connections between two networks, such as if a user switches from WIFI to LTE. IPVS or any UDP load balancer won't handle this scenario properly since it doesn't introspect the QUIC header and load balance based on the QUIC connection ID. This QUIC connection ID allows for a stable connection when the device needs to switch networks. If operators have any sort load balancer (like IPVS) between the client and the point the HTTP/3 connection is terminated, they will need to ensure that it has proper support for QUIC. One example is Katran[1] which has support for this method of load balancing. [1] https://github.com/facebookincubator/katran https://github.com/facebookincubator/katran
- rimtutituki 3y agoAny other L4 OSS around with support for HTTP/3 except Katran? I've tried to use it but it was pain :).
- why_only_15 3y agoThis is a nice post, but it feels a little weird because while HTTP/3 gets them 10ms on average, they could get 30ms on average by reducing Asia latencies (15% of traffic, 300ms) to North America latencies (100ms). I'm sure they have some internal constraint that makes that difficult, but just stuck out at me.
- supriyo-biswas 3y agoThat might require them to put replicas of the search service in Asia, and operating a distributed system over distant geographies is a hard operational problem.
- jefftk 3y agoNot necessarily: they could terminate the initial connection close to the user and then run a new connection. While this sounds like it shouldn't help, because the second connection is fully under their control they can make it highly reliable and very fast. Additionally, any recovery from dropped packets (on the user's home network) only has the latency of an in-region connection. This also sets them up for ruining a light cache in-region, and potentially scaling that up to handle more kinds of API calls.
- Neil44 3y agoYes I'm sure they've considered doing that as it seems so obvious. I would also think that the use of any one account was very region specific so could you actually base their whole data out near them. Be interesting to hear how they've ended up at the compromise that they have.
- chrisweekly 3y agoRelated tangent: highest possible recommendation for https://hpbn.co/ https://hpbn.co/ (High-Performance Browser Networking), an extraordinarily helpful book and website.
- mablopoule 3y agoDidn't knew about this book, it looks extremely interesting, thank you :).
- pksohn 3y agoI second this recommendation, but note that this book predates and does not cover HTTP3/QUIC.
- chrisweekly 3y agoGood disclaimer. HPBN is still recommended reading (almost prereq?) before tackling H3/QUIC, though.
- londons_explore 3y agoIt would be nice to see that 99th percentile, the 99.9th percentile, and the 99.99th percentile. My browser has made over 100,000 network requests so far today, and I expect yours has too. I'm sure at least some of those will be in the 99.99th percentile.
- endorphine 3y agoIt would be interesting to see similar experiments published from big edge networks like Cloudflare and Fastly. Do they have something similar published?
- asymptotic 3y agoWhile HTTP3 does provide some improvements, it's clear that further optimizations are needed, particularly for users in Europe and Asia. One potential solution that hasn't been discussed much here is leveraging AWS managed services like AWS CloudFront or AWS Global Accelerator. It's worth noting that Dropbox's website already uses AWS CloudFront, so they are already leveraging AWS in some capacity. Based on my cost calculations, using AWS CloudFront would cost around $40k a month, while AWS Global Accelerator would be around $22k a month. As of August 22, 2022, AWS CloudFront supports terminating HTTP/3 in a Point of Presence (POP), which could potentially help with the latency issues Dropbox is facing. AWS Global Accelerator, on the other hand, is designed to improve the performance of applications by terminating UDP/TCP as close to users as possible then routing user traffic through the AWS global network infrastructure. This could help reduce latency by ensuring that user traffic is routed through the most optimal path, even if the user is located far from a Dropbox data center. It's hard to estimate what the potential latency reduction of using e.g. AWS Global Accelerator is, especially at higher percentiles. However, using https://speedtest.globalaccelerator.aws/ https://speedtest.globalaccelerator.aws/, and assuming symmetry, my connections to Asia are 35-40% lower latency. Of course, there are trade-offs to consider when using managed services like AWS CloudFront and AWS Global Accelerator. While they can provide significant performance improvements, they also come with additional costs and potential vendor lock-in. However, given the scale of Dropbox's operations and the importance of providing a fast, reliable search experience for their users, it may be worth exploring these options further. --- Cost estimates Assumptions: 1. Dropbox's peak traffic is 1,500 queries per second (QPS). 2. Average data transfer per query is 100 KB. 3. 50% of the traffic comes from North America, 25% from Europe, and 15% from Asia (remaining 10% from other regions, for pricing purposes put it into Asia's calculations). --- 1) AWS CloudFront cost estimation: Data transfer: - North America: 1,500 QPS * 0.5 * 100 KB * 60 seconds * 60 minutes * 24 hours * 30 days = 194.4 TB - Europe: 1,500 QPS * 0.25 * 100 KB * 60 seconds * 60 minutes * 24 hours * 30 days = 97.2 TB - Asia: 1,500 QPS * 0.25 * 100 KB * 60 seconds * 60 minutes * 24 hours * 30 days = 97.2 TB Data transfer cost: - North America: 194.4 TB * $0.085/GB = $16,524 - Europe: 97.2 TB * $0.085/GB = $8,262 - Asia: 97.2 TB * $0.120/GB = $11,664 Total data transfer cost: $16,524 + $8,262 + $7,006 = $36,200 HTTP requests: Total requests: 1,500 QPS * 60 seconds * 60 minutes * 24 hours * 30 days = 3,888,000,000 Using the updated AWS CloudFront pricing (as of May 22, 2023): - HTTP requests cost: 3,888,000,000 * $0.0075/10,000 = $2,916 Total estimated monthly cost for AWS CloudFront: $36,200 (data transfer) + $2,916 (HTTP requests) = $40k --- 2) AWS Global Accelerator cost estimation: Data transfer: - Total data transfer: 194.4 TB (NA) + 97.2 TB (EU) + 97.2 TB (Asia) = 388.8 TB Using the updated AWS Global Accelerator pricing (as of May 22, 2023): - Data transfer cost (averaged across regions): 388.8 TB * $0.035/GB = $13,608 - (Also need to add EC2 egress cost, 388.8 TB * $0.02/GB = $7,776 - Total data transfer cost = $21,384 Accelerator: - Assuming 1 accelerator with 2 endpoints (1 for HTTP/2 and 1 for HTTP/3) - Accelerator cost: 1 * $18/accelerator/day * 30 days = $540 Total estimated monthly cost for AWS Global Accelerator: $21,384 (data transfer) + $540 (accelerator) = $22k
- jupp0r 3y agoNote that some of the users on the worst networks are also users that won't be able to use HTTP3 due to corporate firewall restrictions that block UDP and downgrade TLS. Source: I've worked on network protocols for video conferencing and have seen my share of corporate network horrors.
- cbsmith 3y agoHopefully, as HTTP3 picks up steam, this will stop happening. But yeah, there's no amount of innovation that can't be undone by a motivated adversary. ;-)