9 ms·
GLB: GitHub's open source load balancer
- toomuchtodo 8y agoPrevious discussion (September 2016): https://news.ycombinator.com/item?id=12558053 https://news.ycombinator.com/item?id=12558053
- Chris911 8y agoThis new post is about the newly released GLB Director. The title should be changed.
- justinsaccount 8y agoSee also: Facebook: https://code.fb.com/open-source/open-sourcing-katran-a-scalable-network-load-balancer/ https://code.fb.com/open-source/open-sourcing-katran-a-scala... Google: https://cloudplatform.googleblog.com/2016/03/Google-shares-software-network-load-balancer-design-powering-GCP-networking.html https://cloudplatform.googleblog.com/2016/03/Google-shares-s... The design of all 3 is very similar.
- ded31 8y agoThe Github folks could have used XDP similar as with Katran, I think their solution is very elegant in that the same node would still be able to process other workloads/jobs which is one of the reasons why Facebook decided against DPDK: https://atscaleconference.com/videos/networking-scale-2018-layer-4-load-balancing-at-facebook/ https://atscaleconference.com/videos/networking-scale-2018-l...
- lmb 8y agoI haven't looked at what Google has released, but there are big differences between GLB and Katran. (Not affiliated with any of those companies) In terms of technology, Katran uses XDP and IPIP tunnels, both upstream in the Linux kernel. GLB uses DPDK, which allows processing raw packets in user space, and Geberic UDP encapsulation + a custom iptables module. Neither DPDK nor the module are upstream. There are architectural differences as well. Katran is much closer to a classic load balancer, and uses connection tracking at the load balancer to know where to send packets for established flows. GLB has no per-flow state at the load balancer, which gives it the very nice property that load balancers can be added and removed from an ECMP group without disturbing existing connections. There is an academic paper about a system called Beamer, which most likely influenced GLB (or maybe the other way round?). It's a good read, and relatively short. Finally, Katran is really a C++ library you could build a load balancer on, while GLB comes with batteries included. I think GLB looks nice, hats off to GitHub for open sourcing it.
- justinsaccount 8y ago> uses connection tracking at the load balancer to know where to send packets for established flows Kind of, from katran post: > Each L4LB also stores the backend choice for each 5-tuple as a lookup table to avoid duplicate computation of the hash on future packets. This state is a pure optimization and is not necessary for correctness > .. > Katran uses an extended version of the Maglev hash to select the backend server
- lmb 8y agoIf the connection tracking state is really not required I don't understand how Katran works. How does it deal with the set of live backends changing? Using Maglev hashing gives you "minimal disruption", not no disruption.
- justinsaccount 8y agoI haven't looked at the code, but this is what they say: > Katran uses an extended version of the Maglev hash to select the backend server. A few features of the extended hash are resilience to backend server failures, more uniform distribution of load, and the ability to set unequal weights for different backend servers. GLB also does the same thing with caching: > The hashing to choose the primary/secondary server is done once, up front, and is stored in a lookup table, and so doesn’t need to be recalculated on a per-flow or per-packet basis. GLB has that primary/secondary thing which seems to be how it better handles backends coming and going.
- lmb 8y agoOk, I've looked at the code, and based on that I think what I said in my original post is correct. GLB is more complex, but will maintain connections in more circumstances.
- tehnerd 8y agoconnection tracking could be turned off. but yeah, but default it uses it (for cases where you need to drain a host w/o distruption of existing connections). it's easy to make katran behave the same way as glb (by disabling connection tracking + writing host module which is doing more or less the same as what glb's agent is doing)
- jsiepkes 8y agoLooks really cool! Though a simpler solution for most people will probably be OpenBSD's CARP protocol to share a single virtual IP between multiple boxes (with for example relayd). ECMP routing can get complex fast.
- SEJeff 8y agoOr VRRP with the open source keepalived, which has been around for a decade+ and works wonderfully on Linux.
- gbrayut 8y agoThat is exactly what Stack Overflow uses: keepalived to manage a virtual IP between two decent sized baremetal HAProxy servers (w/ bonded 10G nics). Works great and combined with DNS or Anycast based load balancing can scale pretty damn well. Definitely worth investigating as a KISS approach. To quote a recent Atwood tweet: "if I have learned anything in my career, it is the shocking effectiveness of building ... literally the stupidest thing that could work. (And then iterating on it for a decade.)" https://twitter.com/codinghorror/status/1026332543153389569 https://twitter.com/codinghorror/status/1026332543153389569
- SEJeff 8y agoIndeed! It powers all of the internal loadbalancing (non-direct customer facing) for ticketmaster.com. I was on the core systems team ~12 or so years ago and learned all about how great it is.
- ahoka 8y agoUnfortunately it is not an option on some cloud providers, because you don't get true layer 2.
- vbernat 8y agoCARP and the likes rely on the presence of an L2 layer which means you get limited scale or increased risk of a global outage. An L3 design scales very well and is resilient (notably because of the distributed control plane and the inability to create a loop).
- deleted 8y ago[deleted]
- KenanSulayman 8y agoWhy would one use this over HAProxy?
- atombender 8y agoHAProxy is an Layer 7 (i.e. HTTP, for the most part) load balancer and only handles the use case of spreading load across multiple backends. A single instance binds to a single IP. There's no redundancy; lose the HAProxy and you lose traffic. For true redundancy, you need a layer above that handles the distribution of traffic to multiple redundant load balancer instances, and GLB does that via ECMP (Equal-Cost Multi-Path) routing. Github supposedly uses HAProxy as their L7 load balancer. All of this thoroughly explained in the article.
- llama052 8y agoI thought best practice for HAproxy was to run two HAProxy's in parallel with VRRP or DNS load balancing? Does that not achieve the same outcome? I've used HAProxy for layer 4 in the past without any issue this way.
- wmf 8y agoImagine if you need a lot more than two HAProxies and they aren't in the same rack/subnet; that's where more sophisticated techniques come in.
- gbrayut 8y agoIt does but that setup runs into limits on throughput of individual servers, and doesn't have the same drain/fill/failover capabilities discussed in the article. To be clear, the HAProxy+vrrp+dns is often a better solution, but this describes an interesting design for a load balancing system that can handle many orders of magnitude more traffic and have maintenance without breaking established connections (one of it's core design features)
- KenanSulayman 8y agoI have very successfully in the past and still do use HAProxy as level 4 LB. It's one of the fastest to my knowledge. I have used HAProxy as entry to big Mesos clusters without any issue before. One example of using HAProxy as L4 LB instead of letting it do the termination is when it is proxying TLS traffic from and to multiple backends. Or Websocket. Or even as bastion LB for SSH should one bastion go down.
- weberc2 8y agoWhere is the source code? Skimmed but didn’t see a link.
- geospeck 8y agohttps://github.com/github/glb-director https://github.com/github/glb-director
- q3k 8y agoThis basically looks like an open source Maglev [1]. Awesome! [1] - https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44824.pdf https://static.googleusercontent.com/media/research.google.c...
- vbernat 8y agoThis is similar but not totally. Linux has an open source version of Maglev as a scheduler for LVS since 4.18 (to be released). GLB uses a specific consistent hashing algorithm selecting two servers and a module to let the first server redirect the flow to the second in case it doesn't know about it. This helps minimize disruption even more than with Maglev.
- Drdrdrq 8y agoOff-topic: I love the GLB icon, pure genius!
- llama052 8y agoMaybe I don't see the use case since I'm not at that scale, but it seems like a lot of added complexity for what appears to be hacking around using other load balancing solutions as a Layer-4 option? Edit: It's a question, if you downvote please let me know why it's a better solution.
- vbernat 8y agoThe goal is to avoid disruption during topology changes. If you have long-lived connections, this is important to keep them alive. The explain this a bit more here: https://github.com/github/glb-director/blob/master/docs/development/second-chance-design.md#comparison-to-lvs-and-other-director-state-solutions https://github.com/github/glb-director/blob/master/docs/deve... I have also written about this in a past article: https://vincent.bernat.im/en/blog/2018-multi-tier-loadbalancer https://vincent.bernat.im/en/blog/2018-multi-tier-loadbalanc... (which may or may not be easier to understand)
- ngrilly 8y agoI read your link, and it's easy to understand because it's really well explained :-)
- emmericp 8y agoCool use of SR-IOV, I like it. We've done a few (academic) experiments with SR-IOV for flow bifurcation and we've wondered why no one seems to use it like this. The performance was quite good: neglible performance difference between PF and a single VF and only 5-10% when running multiple >= 8 VFs (probably cache contention somewhere in our specific setup). You seem to be running this on X540 NICs, aren't you running into limitations for the VFs. Mostly the number of queues which I believe is limited to 2 per VF in the ixgbe family. I wonder whether the AF_XDP DPDK driver could be used instead if SR-IOV isn't available or feasible for some reason. A more detailed look at performance would have been cool. I might try it myself if I find some time (or a student) :)
- theojulienne 8y agoWe found that we could achieve 10G line rate with just the queues available to the VF, the NIC didn't seem to be a bottleneck providing DPDK was processing packets faster than line rate. It's worth noting that other traffic on the PF was/is minimal in our setup. We tested this using DPDK pktgen on a identically-configured node (GLB Director and pktgen both using DPDK on a VF with flow bifurcation, on 2 separate machines on the same rack/switch), with GLB Director essentially acting as a reflector back to the pktgen node. pktgen was able to generate enough 40 byte TCP packets to saturate 10G with 2 TX cores/queues, and GLB Director was able to process those packets and encapsulate them with a sizeable set of binds/tables with 3 cores doing work (encapsulation) and 1 core doing RX/distribution/TX.
- emmericp 8y agoYeah, 10G just isn't that much nowadays. And bigger NICs have more features in the VFs. I've just built a quick test setup: * two directly connected servers * 6 core 2.4 GHz CPU * XL710 40G NICs * My packet generator MoonGen: https://github.com/emmericp/MoonGen https://github.com/emmericp/MoonGen with a quick & dirty modification to l3-tcp-syn-flood.lua to change dst mac Got these results for 1-5 worker threads in Mpps: 3.84, 6.65, 10.17, 11.57, 11.3. ~10 Mpps is about 10G line rate for the encapsulated packets; this seems a little bit slower than I expected and it looks like I might be hitting the bottleneck of the distributor at 4 worker threads. Didn't look into anything in detail here (spent maybe 30 minutes for setup + tests), but we've done some VXLAN stuff in the past which I recall being faster.
- koolhead17 8y agoWill it have some special love in azure ecosystem?
- tootie 8y agoAzure already offers load balancing. I'm not sure how much differentiates all the products out there now. I've never seen a load balancer be a bottleneck in any system I've worked on.
- crazy_monkey 8y agoGLB vs Great Cannon!
- subleq 8y agoThis is the first I've heard of Rendezvous hashing. It seems superior in every respect to the ring-based consistent hashing I've heard much more about. Why is the ring-based method more common?
- bogomipz 8y agoI am having trouble understanding this passage. I'm wondering is someone could help me understand this as it seems like an important design detail: >"Another benefit to using UDP is that the source port can be filled in with a per-connection hash so that they are flow within the datacenter over different paths (where ECMP is used within the datacenter), and received on different RX queues on the proxy server’s NIC (which similarly use a hash of TCP/IP header fields)." A source port in the UDP header still needs to be be just that a port number no? Or are they actually stuffing a hash value in to that UDP header field? How would the receiving IP stack no how to understand a value other than a port number in that field?
- gbrayut 8y agoJust a guess, but by using UDP transport for the encapsulated data and configuring the module on the proxy to accept UDP on a wide range of ports, you can pick any port you want (not just the destination port of the TCP stream in the encapsulated packet). And if you are using ECMP with a known hashing algorithm you can then use that UDP port to explicitly spread packets across the RX queues on the proxy servers (gaining better performance).
- bogomipz 8y agoThanks I think that might be what they mean - hash the source port for better distribution across the RX queues on the destination proxy. Cheers.
- bogomipz 8y agoThe article states: >"Each server has a bonded pair of network interfaces, and those interfaces are shared between DPDK and Linux on GLB director servers." What's the distinction between DPDK and Linux here? It wasn't clear to me why SR-IOV is needed in this design. Does DPGK need to "own" the entire NIC device is that? In other words using DPDK and regular kernel networking are mutually exclusive option on the NIC? Is that correct?