3 ms·
It's a nice accomplishment to build a user-facing feature to that scale, but I'm not all that impressed by the raw number. Let's break it down: 1.3T requests pe
by theideaofcoffee 3y ago
It's a nice accomplishment to build a user-facing feature to that scale, but I'm not all that impressed by the raw number. Let's break it down: 1.3T requests per day is roughly 15M req/sec, and assuming that each of the approximately 300 PoP locations [0] is serving this via anycast, that's 'only' 50k req/sec/PoP, easily doable by a couple, a handful at most, of hosts in each location. Small request and response sizes increase total request rate but keep down overall bit rate so one can handle a larger number of requests with a small number of hosts. Obviously these numbers will vary with location, higher and lower based on overall pop utilization usage, but rough enough to plan for. Even with encryption, these are well within the reach of single digit machine numbers in each location.
I don't think the raw number is difficult, the difficult problem was building up the supporting infrastructure, both technical and product-wise.
My source for the above? I've built at-scale, geo-distributed internet-facing services such as anycast DNS, video streaming delivery and the like.
[0] https://www.cloudflare.com/network/ https://www.cloudflare.com/network/
- mathiasgredal 3y agoFor reference opentrackr.org runs on a single machine[1] and handles ~200k connections pr. second[2] [1]: https://twitter.com/opentrackr/status/1383937485403693062 https://twitter.com/opentrackr/status/1383937485403693062 [2]: http://opentrackr.org/ http://opentrackr.org/
- MuffinFlavored 3y agoWhat is a number that would impress you?
- Aachen 3y agoIt's not the digits themselves but rather their unit and whose bank account it is we're talking about Any number of requests is going to be reasonable when you break it down, because if it were unreasonable, it wouldn't be run this way
- deleted 3y ago[deleted]
- jedberg 3y agoIf you've built global geo-distributed networks, then you should know that anycast will not distribute traffic evenly across the pops. Most likely the USA pops do 10-100x the traffic of many of the others. My guess is that some pops are handling 5M+ req/sec. That's a pretty impressive feat of internal load balancing.
- theideaofcoffee 3y agoDid I not explicitly call that out in my reply? > Obviously these numbers will vary with location I’m not going to give basic lessons in network and traffic engineering in two sentences.
- jedberg 3y agoYou did but you hand waved right over it with "rough enough to plan for". Also I think it's important to call out the high variance specifically for the people who are not network engineers. I think you're being misleading with your analysis.
- theideaofcoffee 3y agoEven with an order of magnitude (or two!) difference it’s still an easy problem, especially for DNS.
- junon 3y agoDNS requests fit within most standard MTUs (1500 bytes) over UDP. Almost all of it can be warm cached, and the datapoints are highly compacted and can be sorted offline. I agree with the OP, it's not really that high.