9 ms·
Let's Create a Simple Load Balancer with Go
- hinkley 7y agoLoad balancer seem like one of those problems that engineers should be cutting their teeth on. And yet we have only a handful and one of the most popular charges money for cool features and does not appear to have an ABI for addons.
- sciurus 7y ago> And yet we have only a handful Apache, nginx, haproxy, envoy, traefik, and probably dozens more that aren't on the top of my head.
- nkozyra 7y agoYou literally listed a handful ;)
- SteveNuts 7y agoFabio, IIS, Varnish, Kong, Squid, lighttpd, emacs (probably).
- lugg 7y agohttps://stackshare.io/stackups/emacs-vs-haproxy https://stackshare.io/stackups/emacs-vs-haproxy
- elcritch 7y agoActually Emacs as a dev proxy like Charles Proxy could be handy... ;) Also there exists an http module for emacs: https://www.emacswiki.org/emacs/HttpServer https://www.emacswiki.org/emacs/HttpServer
- AdieuToLogic 7y ago> > > And yet we have only a handful > > Apache, nginx, haproxy, envoy, traefik, and probably dozens more that aren't on the top of my head. > You literally listed a handful ;) How many solutions to a problem are needed when the problem is well-defined before there is no longer a need to grab for more?
- hombre_fatal 7y agoThere are probably thousands on Github for the use of teeth-cutting like in the blogpost. I've made one. You just lose interest when you impl the easy/naive stuff and need to actually use a real solution in production.
- hinkley 7y agoCmon. Does anybody use Apache anymore? We should send them a care package if so, poor bastards. Ingress routers are something else. They balance load for cloud traffic, but can they act just as reverse proxies, right? Linkerd and Traefik are a different tool than haproxy.
- yjftsjthsd-h 7y agoApache's httpd is perfectly good as a web server, and passable as a proxy. I can't say that it'd be my first choice, but I'm not aware that there's any special reason not to use it.
- hinkley 7y agoThe Byzantine config file syntax was a common complaint shared by the early defectors to nginx. If I never have to edit that conf file again it will be too soon.
- dankohn1 7y agoHere are the ones CNCF tracks: https://landscape.cncf.io/category=service-proxy,api-gateway,service-mesh&format=card-mode&grouping=category https://landscape.cncf.io/category=service-proxy,api-gateway...
- jrockway 7y agoThat is a really neat page. You see "funding: $250k" right next to "market cap: $1.1T". It's almost surreal.
- dankohn1 7y agoThanks. Don't miss the landscape view: https://landscape.cncf.io/ https://landscape.cncf.io/ And we've started creating them for other fields, like visual effects: https://landscape.aswf.io/ https://landscape.aswf.io/
- hinkley 7y agoOk. Three of those are forks of the others. Several of those are ingress routers, and I’ve seen no claims of those being able to run standalone (ie without specifically k8tes) so they are technically load balancer, except they can’t operate standalone. Say I wanted to run one per server instead of NodeJS cluster mode. I could not substitute those, right? One is F5 which nobody says anything nice about anymore, and was positioned as a hardware solution for most of its good years. I’d also note that a number of these are pretty young, indicating that there was a power vacuum that is now filling in.
- Rapzid 7y agoI'm thinking of a "load balancer" that charges for basic features like querying for which downstream servers are in service.. HAProxy on the other hand has been doing some really fantastic stuff lately that is in the open source. It makes me a bit sad that the former is the "go to" and HAProxy doesn't get the usage it deserves. To your specific point though I think it's just super tricky to get all the features people expect at the performance they expect as well. You essentially need to implement an very efficient HTTP server, also apparently not trivial, in order to get expected Layer 7 features.. Embedded scripting language support is quickly becoming de facto.. Simple traffic proxy/forwarding is the easy part but getting something competitive together feature and performance wise feels way beyond teeth cutting? EDIT: Using golang as an example. You would need to either piggy-back the best HTTP server available for your language platform or write your own. The stdlib Golang http server is not competitive performance wise. The reasons for this are pretty well known and seem to be largely accepted by the core team(AFAICT). This won't impact most app and service developers too much, but for an LB service the number of cycles it leaves on the table my not be acceptable at all.
- eloff 7y agoYou can use fasthttp in Go to get performance within about 10% of the best in other languages. See the latest tech empower benchmarks.
- Rapzid 7y agoYup. It's also famously incompatible with the wider ecosystem because it's not a drop in replacement for net/http. There are other criticisms I haven't really looked into as well. Not meant to be my own criticism, just an observation. I believe the net/http interface ossified its own design decisions making it hard/impossible to be compatible with?
- eloff 7y agoThe net/http interface requires too many allocations, which prevents it from running at the same speed. That's why fasthttp uses a different design.
- spyspy 7y agoImplementing a simple load balancer in Go was actually a take home interview problem for me once and I have to say I thoroughly enjoyed making it.
- sciurus 7y agoIt's nice to see a walkthrough of what goes into a load balancer and how simple it is to build on in Go. One nitpick is that the autho reversed the meaning of active and passive health checks. Active generates new traffic to the backends just to determine their healthiness, passive judges this based on the responses to normal traffic.
- mholt 7y agoFun stuff! If you like this kind of thing, we are developing a very powerful and flexible reverse proxy with load balancing into Caddy 2: https://github.com/caddyserver/caddy/wiki/v2:-Documentation#httphandlersreverse_proxy https://github.com/caddyserver/caddy/wiki/v2:-Documentation#... It's mostly "done" actually. It's already looking really promising, especially considering that it can do things that other servers keep proprietary, if they do it at all (for example, NTLM proxying, or coordinated automation of TLS certs in a cluster). If you want to get involved, now's a great time while we're still in beta! It's a fun project that the community is really coming together to help build.
- Twirrim 7y agoAny thought or plans for some kind of back-pressure? Health checks and response times are useful, to a degree, but there are a number of workloads where they don't actually capture the cost of the work involved, and they can also really trip you up something nasty under certain failure conditions :D edit: by way of example, I used to work for a service that customers would upload files to, that's all that the traffic was. There was wild variability in the size, processing cost, and upload speed of each request. None of the standard load-balancing approaches really balance "load" from a service perspective. While things worked, it was rarely optimal.
- mrkurt 7y agoHAProxy has a built in way to do some of this -- you can setup an agent check that lets you dynamically adjust weights: https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#5.2-agent-check https://cbonte.github.io/haproxy-dconv/2.0/configuration.htm... I think most proxies are planning to move logic like this to the control plane. Envoy's gRPC stuff has some ways to dynamically throttle traffic to backends. Load balancers really need to become programming runtimes, imo. Config languages aren't very expressive, and almost everyone needs their own logic at the LB level. I _just_ put together a demo of latency based load balancing using HAProxy + awk and it's neat, but still very rudimentary compared to what I could express in, say, JavaScript: https://github.com/superfly/multi-cloud-haproxy https://github.com/superfly/multi-cloud-haproxy
- AdieuToLogic 7y agoThe author states: > After playing with professional Load Balancers like NGINX I tried creating a simple Load Balancer for fun. And while nginx[0] certainly can perform in this role, another production quality load balancer is HAProxy[1]. Both can do more than this, of course. Reinventing solutions "for fun" certainly can be educational and help others learn key concepts, but the author should clearly state what they are doing is not meant to replace production quality solutions. 0 - https://www.nginx.com/ https://www.nginx.com/ 1 - https://www.haproxy.com/solutions/load-balancing/ https://www.haproxy.com/solutions/load-balancing/
- JustSomeNobody 7y ago> but the author should clearly state what they are doing is not meant to replace production quality solutions. If you know you need a load balancer you should already know this.
- AdieuToLogic 7y ago> > but the author should clearly state what they are doing is not meant to replace production quality solutions. > If you know you need a load balancer you should already know this. If you know you need a load balancer, you go and look for one. If you happen upon one in a programming language you use, then it may be more appealing. Hence the need for an explicit disclaimer.
- cortesoft 7y agoBut if you don't do any research to find out if it is production ready, you aren't a production ready dev anyway. A disclaimer isn't going to save them.
- AdieuToLogic 7y ago> But if you don't do any research to find out if it is production ready, you aren't a production ready dev anyway. Seriously? Is victim blaming somehow more acceptable than just clearly stating that a project is not meant for production? And why is the suggestion of including a simple disclaimer so repugnant? What is so wrong with saying "this is an educational project and should not be used in production systems"?
- kitd 7y agoThis is pretty cool. But I think an implementation that avoids the mutexes (mutices?) when allocating the backends and uses channels instead would probably perform better. 2 channels needed, 1 for available backends and 1 for broken ones. On incoming request, the front end selects an available backend from channel 1. On completion, the backend itself puts itself either back onto channel 1 on success, or channel 2 on error. Channel 2 is periodically drained to test the previously failed backends to see if they're ready to go back onto channel 1.
- biggestdecision 7y agoChannels do not avoid mutexes, they are implemented with them under the hood.
- deleted 7y ago[deleted]
- kitd 7y agoSure, but with the above, there's 1 mutex to read the channel, not 1 per backend. And a single thread reading the channel. Plus you know that if the backend is on the channel, you know it's ready to accept, and you don't need the alive flag with synchronized access.
- biggestdecision 7y ago> there's 1 mutex to read the channel, not 1 per backend Fewer mutexes isn't necessarily an advantage, there will be far higher contention on that single mutex. Channel writes also require a lock.
- kitd 7y agoOne to read, and a single reader thread. That's no contention. One to write, that's used for writing only, not both read/write as in the published design. Edit: it's a variation on the following, but for ReverseProxies, not worker goroutines http://marcio.io/2015/07/handling-1-million-requests-per-minute-with-golang/ http://marcio.io/2015/07/handling-1-million-requests-per-min...
- knicknic 7y agoWith talk of proxy and go. I am surprised gobetween hasn’t been mentioned yet https://github.com/yyyar/gobetween https://github.com/yyyar/gobetween . Last time I looked at the source it was very approachable.
- manigandham 7y agoSeems like most of the work is done by the `ReverseProxy` package and this code is more about health checking. Nice to see how simple it is now though. Go is definitely a great choice for low-level networking, and .NET Core has recently become a great option as well.
- 4gotunameagain 7y agoI get what you mean and I agree, but this is far from low level networking
- snypox 7y agoI’m also a bit confused where is .NET Core coming to the picture. It’s becoming faster and faster but it’s definitely not the best tool for load balancers.
- manigandham 7y agoWhy not? The latest techempower results show ASP.NET Core saturating a 10GbE network card with 7M+ req/sec while being a full-featured framework, compared to custom C++ web servers. [1] The memory management, type safety, and high-level productivity make it great for building load balancers and other infrastructure components. 1. https://www.ageofascent.com/2019/02/04/asp-net-core-saturating-10gbe-at-7-million-requests-per-second/ https://www.ageofascent.com/2019/02/04/asp-net-core-saturati...
- tomohawk 7y agoYou'd have to know how much memory and cpu was being used vs the other solutions. They're all very close in terms of the key metric, but if Rust is using 50% of the memory, then its not much of a competition. Also, what sort of vm tuning, etc.
- manigandham 7y agoTrue but you can dive into the benchmarks more if you visit the techempower site. They run it on the same standardized hardware and in this .NET is very competitive on cpu and ram.
- rndmio 7y agoIf you don't need http smarts on the load balancer LVS is a great option that will give you fantastic performance.
- arberavdullahu 7y agoI found it more convenient to keep the field unnamed when using mutex in a struct. So in the example that would be type Backend struct { URL *url.URL Alive bool sync.RWMutex ReverseProxy *httputil.ReverseProxy }
- mutatio 7y agoThe problem with that is that the methods for `sync.RWMutex` spill into `Backend`.
- thegeekpirate 7y agoThat isn't a problem—in fact, that's the entire point of arberavdullahu's suggestion.
- fiveturns 7y agoStill seems like a problem to me. It breaks encapsulation. The mutex is only used inside of SetAlive() and isAlive(), they're the only things this need to handle locking and unlocking. You don't want anything external to that calling the methods on RWMutex.
- thegeekpirate 7y agoOh of course, if you're not using it then don't expose it. I haven't read the code so I can't verify if that's the case here, but I read OPs post as being worried about method clobbering (which is really a non-issue, if the popularity of embedding mutexes shows us anything).
- westoque 7y ago> Multiple clients will connect to the load balancer and when each of them requests a next peer to pass the traffic on race conditions could occur. I quite don't understand what this means? What race conditions? Can anybody explain? Thanks.
- littlestymaar 7y agoWhen you have 2 threads running in parallel and the outcome of the computation depends on the order in which they do their job first, that's a race condition.
- therealdrag0 7y agoI haven't really read it but I'll take a stab (with psuedo code). I think NextIndex() is incrementing s.current and modding it with servers.length. To do this, there are three operations, SET s.current to +1, and GET s.current so that it can be MODDED with servers.length. If that SET and GET are not coordinated between threads, then a race could sour the index. For example, if two threads call this method at the same time, they could both SET the +1 before either GETs the value, then they will both get the same value +2 from where it started instead of +1 for each caller.
- westoque 7y agoThanks for the explanation. Forgive my lack of knowledge, but can it not be solved by using parallelism instead of concurrency/threads? I thought Go has first class primitives to be able to do this?
- therealdrag0 7y ago> parallelism instead of concurrency/threads. What do you mean? These seem synonymous to me. > I thought Go has first class primitives to be able to do this? TBH I'm only 1 week into learning Go myself (coming from Java/Scala/JS/etc). But it looks like the article used what Go offers. They used the atomic package which says, "provides low-level atomic memory primitives useful for implementing synchronization algorithms." https://golang.org/pkg/sync/atomic/ https://golang.org/pkg/sync/atomic/
- noah-kun 7y agoGreat tool here: https://github.com/superfly/wormhole https://github.com/superfly/wormhole
- kasvith 7y agoThanks for sharing the article and thanks all for the feedbacks :)