9 ms·
Building a Billion User Load Balancer [video]
- dorianm 10y agoMap of Google data centers: http://imgur.com/l1dDdQe http://imgur.com/l1dDdQe Map of Facebook data centers and PoP: http://imgur.com/dek8ESX http://imgur.com/dek8ESX
- blauditore 10y agoI wonder what the reasons behind their similar distribution are. One is surely the density of users, but maybe there are others related to the actual hosting, like cost, law or infrastructure.
- di4na 10y agoinfrastructure : electricity, roads, price of land, availability of workforce, water, temeprature, closeness to the backbones. Add that to the density of users and the timing they expect. Once you factor all that, you end up with a limited amount of possibilities.
- y04nn 10y agoBoth companies have looked for optimal choices, taking into account all factor, it is logic that they ended up with the same answers.
- brobinson 10y agoBoth Google and Facebook both have datacenters/PoPs in Taiwan? I can't imagine the 24mm people there justify it, and both services are either largely unused or blocked in the nearby PRC... I'm more surprised that Google has no presence in Japan, though.
- bschwindHN 10y agoGoogle has an office in Tokyo but I guess not a datacenter.
- pyvpx 10y agobut plenty of Google Global Caches, so performance (latency to cached objects and DNS queries) is still quite good
- zeeZ 10y agoThey fixed that, at least for GCP. How the other Google bits fit into that, I do not know. They're planning to expand to a bunch of new regions this year. https://www.blog.google/topics/google-cloud/google-cloud-platform-tokyo-region-now-open-for-business/ https://www.blog.google/topics/google-cloud/google-cloud-pla...
- mikecb 10y agoThey do. The above map is wholly owned DCs, but there are many more PoPs, and an enormous number of edge cache locations: https://peering.google.com/#/infrastructure https://peering.google.com/#/infrastructure
- briandear 10y agoThere's a major Asia-Pacific undersea cable connected to Taiwan, so Taiwan is a useful Asia-Pacific gateway.
- mafribe 10y agoMore precisely, Taiwan is well connected by direct cables to Japan, China, Philippines, Viet Nam, Malaysia, Thailand, Korea, Singapore and the US. As far as I'm aware Google owns a cable connecting Taiwan with Japan (part of [1]). [1] https://en.wikipedia.org/wiki/FASTER_(cable_system) https://en.wikipedia.org/wiki/FASTER_(cable_system)
- true_religion 10y agoIn my experience, having a PoP in Taiwan, means that some data requests from S. Korea, Philippines, and Australia are routed to that. I never know how that works, since Singapore and Tokyo, Japan should be closer---but it still happens.
- pyvpx 10y agoload and network conditions (paths from you to Tokyo are experiencing packet loss for $REASONS, Singapore is overloaded/broken/full, etc).
- Retr0spectrum 10y agoI'm surprised that neither have any datacenters in India. Does anyone know why this might be?
- Joe8Bit 10y agoMy intuition tells me it's a combination of: * They eventually will build one, but Indian consumers are (as of right now) less valuable to them (from an ad targeting perspective) than other geographies e.g. the economic/time/effort investment ROI is better elsewhere in the short term * SE Asia (and their POPs there) have reasonable connectivity to India, so it might good enough for the time being
- noponpop 10y agoThat's not quite the answer he gave to this exact question in the video: geopolitics and cost were the major reasons, they want pops there.
- nocarrier 10y agoCost was a smaller factor than politics; the Indian government wanted the private keys for our certs in order to let FB put a POP there. That was an absolute dealbreaker, so we served India from Singapore and other POPs in nearby countries. Regarding building a datacenter, that's much higher risk since it's a $100M-1B capital investment. I'm guessing both Facebook and Google see too much volatility to make the risk worth it. It could be an expensive paperweight if the Indian government changed their minds.
- degenerate 10y agoWow. Thank you for not bending over on the private key issue. That's terrible.
- deleted 10y ago[deleted]
- theDoug 10y agoGoogle has announced a region for Mumbai (India) coming this year: https://cloud.google.com/about/locations/ https://cloud.google.com/about/locations/ (Can't/won't speculate as to why not earlier.)
- ape4 10y agoToo bad for Africa and Australia.
- sambe 10y agoYeah, massive surprise for me to see Australia/NZ not have anything dedicated. Network reliability over the whole region must be pretty poor and although in population terms it's probably too low, you'd think in purchasing power terms it would still be worth it.
- acqq 10y agoIs this audio/video only or am I missing something? If it is maybe it should be marked in the title.
- uniclaude 10y agoIn case anyone else is looking for a transcript: I can't check right now if this talk is much different from the one he did in 2015, but just in case, here's a link to the Youtube video for the older one: https://www.youtube.com/watch?v=MKgJeqF1DHw https://www.youtube.com/watch?v=MKgJeqF1DHw That's the only way I found to get any sort of text.
- fforflo 10y agoSlides here http://www.slideshare.net/patrickshuff/buildinga-billionuserloadbalancer-may2015srecon15europeshuff http://www.slideshare.net/patrickshuff/buildinga-billionuser...
- ju-st 10y agoThe "cartographer" doesn't use end user related information to select the best PoP? Or does "Sonar" measure the latency/throughput by looking at existing connections to users? Ok I watched the presentation now, Sonar apparently does that indeed.
- patrickshuff 10y agoSonar is the system we use to measure latency from the client devices to all of our PoPs. Cartographer is the system that consumes all of these sonar measurements and uses several other real time data sources (BGP routes, Link capacity, PoP health, PoP capacity, etc) and continually generates a GLB map for the most optimal targetting of requests to our PoPs.
- deleted 10y ago[deleted]
- bogomipz 10y agoI am curious does Cartographer also adjust iBGP preferences for balancing amongst transit providers for egress traffic?
- noponpop 10y agoInteresting that Linux kernel performance (ipvs) is acceptable at l4 vs something like dpdk. I guess you just overcome the limitation by increasing the number of l4 instances load balanced by ecmp. Fun to see DSR in use. Also interesting to see that all the inherent problems with geolocation via gslb (DNS client IP is not the same as the real client IP) don't wind up being a big problem apparently. This seems to be a growing concern in my experience: users aren't located where thier ISP DNS servers are located.
- 79d697i6fdif 10y agoIt's mostly because the point of DPDK and similar is to go around a lot of the processing in kernel, and IPVS does exactly this. I'm surprised IPVS isn't more popular, it's built into the kernel and extremely fast. HTTP proxy type load balancers are slugs in comparison Scaling app servers to nearly unlimited size is easy to explain but really hard in practice. It basically amounts to this: 1) Balance requests using DNS anycast so you can spread load before it hits your servers 2) Setup "Head End" machines with as large pipes as possible (40Gbps?) and load balance at the lowest layer you can. Balance at IP level using IPVS and direct server return. A single reasonable machine can handle a 40Gbps pipe. I guess you could setup a bunch of these but I doubt many people are over 40Gpbs. Oh, and don't use cloud services for these. The virtualization overhead is high on the network plane and even with SR-IOV you don't get access to all hardware NIC queues. Also, I don't know of any cloud provider thats compatible with direct server return since they typically virtualize your "private cloud" at layer 3, whereas IPVS actually touches layer 2 a little. Do yourself a favor and get yourself a few colo's for your load balancers. 3) Setup a ton of HTTP-proxy type load balancers. This includes Nginx, Varnish, Haproxy etc... One of these machines can probably handle 1-5 Gbps of traffic so expect 20 or so behind each layer 3 balancer. These NEED to be hardened substantially because most attacks will be layer 4 and up once an adversary realizes they can't just flood you out(due to powerful IPVS balancers above). SYN cookies are extremely important here since you're dealing with TCP... just try to set everything up to avoid storing TCP state at all costs. This also means no NAT. You might want to keep these in the colo with your L3 load balancers. 4) Now for your app servers. Depending on if you're using a dog slow language or not, you'll want between 3 and 300 app servers behind each HTTP proxy. You don't really need to harden these as much since the traffic is lower and any traffic that reaches here is clean HTTP. Go ahead and throw these on the cloud if want
- tonyplee 10y agoThis also gives interesting overview of SDN, NFV, scale inside google data center. https://www.youtube.com/watch?v=vMgZ_BdipYw https://www.youtube.com/watch?v=vMgZ_BdipYw
- je42 10y agowondering the tinydns and cartographer work together ? i didn't see any dynamic response capability in tinydns ? did i miss it in the docs of tinydns ?
- nocarrier 10y agoCartographer ingests BGP topology and POP and datacenter health (and other things), and then pushes what is basicially a lookup table to tinydns. None of the dynamism is inside the DNS server itself, which lets it focus on what it is good at--responding to a crap ton of DNS requests per second.
- visarga 10y agoGood thing this is out. I'm gonna put one on my blog, in expectation of the traffic I'd like to have!
- luizbafilho 10y agoFor those who are interested in replicate the very same architecture in your environment, I am working on a similar solution. Please check it out https://github.com/luizbafilho/fusis https://github.com/luizbafilho/fusis It is a control plane for IPVS and adds distribution, fault tolerance, self-configuration and a nice JSON API to it. It is almost done, but it needs documentation on how to use it. Moreover, for those who like numbers, I did some benchmarks using a 16 cores machine with two bounded 10Gbit interface Scenarios: 1 request per connection with 1 byte: 115k connections/s 20 requests per connection with 1 byte: 670k requests/s 20 requests per connection with 1 megabyte: 14Gbps It scenario tests one specific aspect of the load balancer.