9 ms·
Why is load balancing gRPC tricky?
- dmoy 6y agoCool read! Always fun to think about this stuff. > The official implementation uses a Per Call Load Balancing and not a per connection one. So each call is being load balanced separately. This is the ideal and the desired case and it will avoid having heavy sticky connections. I wonder how much of this is just leaking from how stubby & GSLB is done internally at Google. I'll admit, I don't know much about gRPC. Separate point: if you control both the clients and the servers completely, you can have servers respond with load shedding signals if they start to get overloaded, and use that as a hint for the client to go look elsewhere. This doesn't solve all problems, but it does make some cases easier to handle. Also it is obviously not a viable solution if you don't fully control all clients - some of them will just not respect your load shedding and continue to beat an individual server into the ground.
- Sparkyte 6y agoThe trick would be to use the LB as an initial session starter. Don't use LB metrics to scale, just manage what hosts are available for a session.
- alpb 6y agoI am not even sure if the problem described in the article is specific to gRPC. I think it's a HTTP/2 problem. Any sane HTTP/2 client making N queries to the same host/IP ideally should use 1 connection since h2 supports multiplexing (which is bad for load balancing if you have an L4 LB). So even if you don't use gRPC, consider putting a per-call (L7) LB in front of your app if you use HTTP/2 protocol between your apps.
- hcc 6y agoWhy tie the L7 load balancer solution to AWS while the rest of the article does not mention AWS? It already worked fine way before October with envoy or haproxy for example.
- alpb 6y agoYeah, Google's L7 load balancers have been supporting gRPC since forever, but I'm also surprised to hear AWS and Cloudflare only recently added gRPC support. Maybe the author uses only AWS so that's why only AWS is relevant to them. OSS proxies and gateways like envoy had it from the get go.
- CommonGuy 6y agoShameless plug: If you are searching for a Postman alternative for gRPC, give https://kreya.app https://kreya.app a try.
- pritambaral 6y agohttps://insomnia.rest https://insomnia.rest already supports gRPC, but isn't restricted to gRPC.
- alpb 6y agoWireShark has gRPC/Protobuf support now. https://grpc.io/blog/wireshark/ https://grpc.io/blog/wireshark/
- pojntfx 6y agoThat tool is proprietary; BloomRPC[1] has 80% of the features and is free and open source. [1] https://github.com/uw-labs/bloomrpc https://github.com/uw-labs/bloomrpc
- thayne 6y ago> Although this feature from ALBs is relatively new but the arguments for sticky gRPC connections and NLB are also applicable with ALBs. If yu end up with a single sticky and persistent connection from a client that starts sending a large volume of requests, you are overloading a single server instance that holds that connection and the requests are not being load balanced as they should. That seems strange to me. Without end-to-end http/2, ALBs would load balance individual requests, not the entire TCP session, even with keepalives. And it sounds like individual messages on the same connection can be routed to different target groups, which would make it even more stange not to balance individual requests.
- 1787 6y agoYeah, I think the post author may be confused. I've migrated several high traffic gRPC services (basically TensorFlow Serving) from AWS NLBs to ALBs for the improved load balancing. It worked pretty much as desired (although a little expensive).
- jayd16 6y agoI don't think grpc even supports http/1.1 on all (any?) languages so it was never a feasible solution for grpc.
- thayne 6y agoRight. I was talking about normal http/1.1 traffic with http keepalive.
- avinassh 6y agoNice article! Wish it also included (or linked to) some code samples on how to do each of those load balancing methods
- staticassertion 6y ago> Either of these approaches defeats a fundamental gRPC benefit: reusable connections. I feel like that's not quite true. The server could notice it's over-worked and start telling clients to back off or otherwise to try another system. At this point the clients can make the call to either back off or terminate the connection. The server can increase its "back off for N ms" until it finally has to terminate connections. It requires a bit of extra coordination, but not much. Also, even without such a system, forcing a reconnect when a new instance starts up/ periodically still reduces connection load a ton. If you force reconnects every minute, and you have 10/rps, that's 1 reconnect per 600 calls. I guess in theory a load balancer is the most information to make these decisions, so really you'd want it to be aware of 'back off' requests from the server. One way of doing this might be something with health checks? Dunno.
- vishnugupta 6y agoThis is a good survey of options with pros and cons. While one is at it please spend some time understanding failure modes and more importantly recovery from failures. Let me describe a scenario my team ran into while using option #3 i.e., Look-Aside. We used etcd as service discovery store. Servers as part of their startup would register with etcd. Clients would query etcd and randomly pick a server to talk to. Now, for some reason the entire service fleet got isolated. A couple of hours later the network partition got back just fine but all the servers had shut down. The real pain started when we began bringing up servers. As soon as a server was brought up it would register with etcd, immediately get bombarded by hundreds of clients and then would promptly die. No matter what permutation we tried servers would just refuse to come up because of thundering herd. The only option was to shut down the client app, which was another fiasco because we hadn't built a clean way to do it, and bring up all the servers and then gingerly bring up clients hoping they won't kill the servers. The downtime lasted almost for a day. And this is a largish app based ride hailing business I'm talking about so you could imagine the shitstorm it created among customers and investors alike. A key learning for us was to isolate service startup from load-balancer registration. A service should not be responsible for registering itself with a load balancer that should be for someone else to do. While gRPC does have lots of positives it still has some way to go before it reaches the operational maturity. People will discover these deficiencies in a painful manner.
- im3w1l 6y agoMaybe you could bring up a bunch of fake-servers, that "implement" every rpc call with a long sleep or maybe by responding with internal error. Then take them down gradually as real ones go up.
- daniellarusso 6y agoThat is pretty clever.
- jrockway 6y agoI think what you really want to do is not give every client a full view of all the backends. xDS lets you write a service discovery server that meets this condition (it knows the full state, potentially with health information about upstreams, and it knows which client is connecting, so you can adjust this as you see fit). I've also seen people do AZ or regional aggregation, i.e. given some consumer in AZ A, the consumer gets a list of endpoints like, 0.upstream.a, 1.upstream.a, ..., regional-aggregator.B, regional-aggregator.C, etc. It sees all the endpoints in the same AZ, but goes through a proxy to get to other regions/zones. Under non-panic circumstances, you'd want all requests to be served from the same node, then the same zone, and only go to other regions in degraded cases. I don't know what the state of the art around tooling to manage this is. For some reason, I suspect that the service meshes punt on this, because N * M isn't a problem in the demo environments where these systems spend most of their time. Meanwhile, the big companies that hit the scalability limit of N * M connections across upstream/downstream pairs wrote their own service discovery stuff decades ago.
- applepple 6y agoHTTP2 is not a suitable protocol for implementing persistent, stateful communication channels. HTTP2 is bidirectional but it's stateless. gRPC is bidirectional but intended to be stateful. WebSockets (bidirectional and stateful) would have been a better match and not required sticky sessions. gRPC essentially takes us back to the dark ages of long polling. I was making this argument for years. WebSockets was designed for bidirectional data exchange between client and server. On the other hand, HTTP2 was designed for pushing static resources to the client to speed up loading of front ends (it was meant to make bundling front-end scripts redundant). HTTP2 was not designed for bidirectional data exchange over a stateful channel.
- ffk 6y agoI believe grpc only uses the http2 frames, which are bidirectional. Double check this though.
- applepple 6y agoBased on the article and the fact that they implied that sticky sessions are necessary, it doesn't seem that the abstraction is airtight. If it was properly stateful, then sticky sessions would not be required. With WebSockets, sticky sessions are not required when load balancing.
- jayd16 6y agoIt's not sticky sessions in the traditional sense. It's a single connection. Http/1.1 requests are allowed to be stateful within a single connection as well. The difference is there is built support for multiplexed requests within that single connection.
- jayd16 6y agoCan you be more specific? What technical drawbacks are relevant? It's certainly not literally polling any more than a websocket so what do you mean? Long running websockets are also hard to balance for the exact same reasons. Much harder in fact because you can't use a per request LB.
- solatic 6y ago> Making the server to forcefully shut down the connections after sometime, and when they reconnect, it automatically causes the new connections to go to the healthier instances. This is good practice anyway. Having long-lived servers causes all kinds of issues. In the old days, if you had a server that was up for years, you had no idea whether you could reboot the server safely or not. Additionally, from a security perspective, the longer-lived a server is, the more likely that an attacker installed some manner of backdoor on the server that's just waiting for the attacker's opportune moment to come back and exploit it. Much better to make chaos an expected part of your operations - make sure your servers are cattle (not pets), give them maximum lifetimes, and terminate them on a regular basis. The more frequent the termination, the better prepared you are for single-node overload, and the less likely that it happens in your use case in the first place.
- astonex 6y agoA bit off-topic but the moving particles in the background of this page make it really hard for me to focus on the text
- p_l 6y agoEven better - they do not move for me at all, but they put ridiculous amount of load on the system - this was the heaviest tab on my system, and its contenders were big fscking SPAs
- henadzit 6y agoNice read! I can add that from my experience using DNS for load balancing isn't the best idea because of the complexities with the DNS caching. The DNS caching happens on DNS resolvers, OS level but also some clients do it too which might be tricky to watch for. Basically you might end up with an IP address of a failed server and no way of retrying until TTL of the DNS record expires.
- secondcoming 6y agoWe have lots of client instances communicating to other server instances with ZMQ PubSub via a network load balancer. We find that periodically recreating the client side connection works fine for us.
- z77dj3kl 6y agoI was hoping to see something about xDS. I've seen a lot of work being done on adding support for it in gRPC on git but (as usual with gRPC) the docs and blog posts seem to be rare and terse, so I haven't had a chance to learn it yet. Also, does anybody know how gRPC is used at Google? Why do they support its development (other than GCP clients)? I'm under the impression Stubby is a completely different codebase and is far superior and fully integrated in their stack.
- rockwotj 6y agoIIRC, right before I left Google at the end of 2019 you could be added to an allowlist to use gRPC internally. I believe they want to eventually replace Stubby with gRPC. Someone who still works at Google would have to comment on the current state of the world or if my memory is failing. As for Stubby it is a completely different codebase and is tightly integrated with a lot of internal tooling and features. One other thing I miss from Stubby C++ compared to gRPC is that async stubby is super opinionated about it's threading model and came with batteries included, where I find the completion queue approach very hard to use. Lastly, as far as load balancing goes. I think some load balancers with native gRPC support allows you to let different servers get different individual requests instead of requiring all requests go to the same server. Of course this only really works for unary RPC calls. But load balancing HTTP/2 isn't unique to gRPC nor keep alive http/1.1 connections. (Outside of the streaming RPCs of course, but there is an equivalent of websockets)
- z77dj3kl 6y agoThank you! Do you know why they were looking to replace Stubby with gRPC? It doesn't seem to me that gRPC is quite customizable enough currently to hook in all the monitoring and tooling that I imagine Google has. And yes, the completion queue approach is pretty cumbersome, would be very interesting to see how stubby went about it :P
- rockwotj 6y ago> Do you know why they were looking to replace Stubby? Likely to maintain less code long term. It's nice to have a primative like this open source so you can use it everywhere (even on mobile). I'm not sure about the claim that gRPC isn't customizable enough, I think there are a lot of hooks in place. For example Open Census does a lot of the instrumentation that you get for free at Google with Stubby, but for gRPC: https://opencensus.io/zpages/ https://opencensus.io/zpages/ > How stubby went about it: Stubby is a lot closer to the experimental callback API talked about here: https://github.com/grpc/proposal/pull/180 https://github.com/grpc/proposal/pull/180
- imtringued 6y ago>reconnect periodically >Either of these approaches defeats a fundamental gRPC benefit: reusable connections. Let's say the client reconnects every 50 requests. You have reduced the connection overhead by 98%. Are those last 2% really going to ruin your day?
- EricBurnett 6y agoIt very much depends on your clients and the nature of the traffic they send. For a high-volume client, assuming a default of 100 concurrent streams per connection max, this actually works out to <connection establishment>, <50 parallel requests>, <connection close>. Have to ~quadruple your connection count vs what you could have done on one unfettered connection, and every connection will spend 50% of it's time being established. Some hybrid of count and time might do, e.g. `50 requests AND 1+ minute since establishment`. But it's nuanced - surprisingly hard to find logic that works well in all cases.
- morelisp 6y agoAs a concrete case of what EricBurnett says, we have a service that has a "long" warmup (p99 30ms - you can think of it like loading a symbol table) but then very fast requests once that data is loaded (p99 about 400usec, fast specifically because it can exchange indexes into the shared symbol table and not full data).
- viraptor 6y agoI'm not sure if I'm missing something or if the post only wants to discuss load-balancers available in AWS infra, but is there any reason we can't have an active application level load balancer for gRPC? As in, one that connects to all backends ahead of time and... balances the load on a per-request rather than per-connection basis?
- jrockway 6y agoThere are quite a few proxies that can do this; Envoy's probably the most popular. You can also load-balance on the client side. Clients connect to a management server, and the server tells them what percentage of connections to send to which upstream. I don't think anyone has written any publicly available software to manage this, but it can use the same underlying protocol as Envoy's configuration servers (which are also rare in public; I wrote one, but it sends clients Endpoints and Clusters and the gRPC implementation happens to want Listeners). The xDS implementation is a newer version of grpclb, which maybe had more support but I've never heard of anyone using it. (Google internally uses the client side load balancing. Worked great when I used it, so was always surprised it wasn't more popular in the real world.) The article is really negative about client-side load balancing, but you can't complain about the latency properties -- no proxy between you and the other end. I always wished the Internet standards did a better job here; you can serve multiple A records for a DNS lookup, and browsers kind of sort of try to load balance (by selecting one of the endpoints randomly), but it doesn't work too well if one of the backends is broken. If you could just supply a pre-defined health-check function and maybe select a simple balancing algorithm, HTTP would be just a little bit smoother for everyone. (If you put topology information in the DNS lookup, you could even balance geographically without having to run special DNS servers that serve different replies to different users. A lot of upstream DNS caching hacks could then go away. It would be great! People are onto the value of this; Envoy Mobile lets you serve this information to your mobile app, so it can easily survive regional failures in your server-side infrastructure. Probably not a lot of adoption, but something that browsers should start doing.)
- NovemberWhiskey 6y ago>You can also load-balance on the client side. Clients connect to a management server, and the server tells them what percentage of connections to send to which upstream. I don't think anyone has written any publicly available software to manage this, but it can use the same underlying protocol as Envoy's configuration servers I believe this is exactly the way that Hashicorp's Consul Connect service mesh is implemented for L7 routing.
- jeffbee 6y agoArticle doesn't mention the one big con of look-aside load balancing. In a partition scenario it can be that the load director can contact backends that the client can't, so it gets bad advice from the load director. So you need to think about fallbacks and how clients can operate autonomously in this scenario.
- AtNightWeCode 6y agoNot really a gRPC problem. Typically in the cloud one need to drop the connections after something like 2-4 minutes no matter what. Tweak it.
- AtNightWeCode 6y agoI really wonder how one get the powers to down-vote on HN. What is the down-vote for? Is it a specific gRPC problem? For I for sure work with LBs that are set to timeout at 4min. And most of the backend clients are set to drop sockets after 120s. This causes challenges for all traffic.
- strken 6y agoIs there a name for the pattern where your client retrieves a subset of the available instances via service discovery, establishes connections to every instance in the subset, and spreads requests between them? E.g. you have 200 clients connected to 100 servers, your service discovery sends each client a list of 10 servers, and 2000 connections are established instead of 20000. "Partial mesh" is a good description, but it's mostly used to describe routing, not load balancing.
- herval 6y agoFinagle calls this "aperture" (https://twitter.github.io/finagle/guide/ApertureLoadBalancers.html https://twitter.github.io/finagle/guide/ApertureLoadBalancer...)
- morelisp 6y agoThis seems to still rely on (or at least, describes) the client making the determination of what to talk to, not the mesh / service discovery.
- herval 6y agoWell, it does exactly what op wrote: "your client retrieves a subset of the available instances via service discovery, establishes connections to every instance in the subset, and spreads requests between them"
- morelisp 6y agoThe article you linked describes retrieving all instances and establishing connections to a subset of them, not retrieving a subset of them. To achieve deterministic apertures it further needs both the complete serverset and the complete peerset.
- jayd16 6y agoIs this not the Look-Aside pattern described in TFA?
- sigmonsays 6y agowhy not just do a layer 4 tcp connection hash on the client source ip? I don't see anything difficult here...
- mastermojo 6y agoI've seen good results with Linkerd + Kubernetes + GRPC. https://linkerd.io/2018/11/14/grpc-load-balancing-on-kubernetes-without-tears/ https://linkerd.io/2018/11/14/grpc-load-balancing-on-kuberne...