2 ms·
Having led API teams at a big tech where we handled similar (slightly lower) request numbers at the edge, I can tell you that the entire stack was engineered to
by Karrot_Kream 3y ago
Having led API teams at a big tech where we handled similar (slightly lower) request numbers at the edge, I can tell you that the entire stack was engineered to be defensive and handle a certain amount of load. As other commenters have said, fan-out means that 80k qps at the edge means you're getting probably 10-100x that on some of the most heavily hit backend systems. A lot of the work we did was very aggressive caching and sharding. Autoscaling to handle load spikes.
Observability was our secret sauce. We would monitor everything. Our caches, NICs, our load balancers, etc. Cache hotspotting and DB problems were the problems that kept us up at night, though my teams didn't deal with much stateful data.