12 ms·
gRPC Is Easy to Misconfigure
- herpderperator 6y ago> If a client sends keepalive pings too often, servers close the connection. This is to prevent large numbers of idle clients from consuming too many resources. This is horrible. Let users configure what they choose. If your network needs higher frequency keepalives, why would the authors of the spec try to know better. I agree with the next statement: > In my opinion, this means this setting is virtually unusable
- rad_gruchalski 6y agoDefine the "user" in the scenario where it is "my" gRPC server but "your" client affecting "my" SLA.
- herpderperator 6y agoIf you can't handle the client, why did you accept the connection in the first place? It has nothing to do with what you're using the connection for. I have an IP address on my ISP connection whether or not I use any data. It's factored into their scaling already.
- drenvuk 6y agoIf 8kB error output limit and incredibly long keepalive means it's easy to misconfigure gRPC then so is everything else.
- astonex 6y agoThe issues in this blog post seem like non-issues. You probably shouldn't be sending back >8kb error messages to the client. The server should log error details locally and the logs should be inspected or scraped separately. The client doesn't need to know the full details like the contents of a variable length DS I develop gRPC clients and servers at work that handle about 40k QPS (typically <50ms response times) whilst using default gRPC configuration. The only changes I had to make was to the max message size.
- cryptonym 6y agoCan be quite useful to enable verbose errors during dev and initial tests.
- lmm 6y ago> You probably shouldn't be sending back >8kb error messages to the client. The server should log error details locally and the logs should be inspected or scraped separately. The client doesn't need to know the full details like the contents of a variable length DS You shouldn't, but it's easy to accidentally do. Having a limit on the server and a limit on the client that's (by default) half the limit on the server seems like an unforced error - given that the server does truncate at a certain limit, surely it would be better for that limit to be smaller than the default client limit. Postel's law and all that.
- Aeolun 6y ago> You probably shouldn't be sending back >8kb error messages to the client While I agree, it seems like a strange limit to have as part of your spec.
- JamesNK 6y agoThe error message is sent as a HTTP trailing header. Their server or client has a header size limit of 8kb.
- thu2111 6y agoThere is no spec, as far as I can tell. That's why the article says different implementations disagree on the max size of the error message. As for large errors, there are plenty of cases where they can be useful. Not every RPC is crossing a privacy boundary: if an administration CLI performs an RPC that fails, it's more convenient to have lots of debugging information printed to the admin than to force them to go and dig through the server logs. Honestly, gRPC makes me a bit sad. It's clearly a duplicate of Stubby. I don't know what Google use these days, but Stubby was a truly great RPC system. For inexplicable reasons Google constantly choose to badly reimplement their internal tech as part of open sourcing it, and gRPC is like some cheap knockoff in comparison.
- compsciphd 6y agoI've seen similar type errors when our ingress proxy switched from nginx to envoy. My golang http(s) rest requests were randomly getting unexpected EOF/transport closed errors. Ended up solving it, by simply putting .Close = true into all the requests to force no pipelining (or at least as I understand it).
- thayne 6y ago> when client keepalive is too aggressive, client RPCs will fail with gRPC code UNAVAILABLE (14) and message "transport is closing" It seems like the correct way for the client to handle this, would be to re-establish the connection and retry if this happens. And that should be done by the grpc client library.
- ishaanbahal 6y agoIt normally does this in my experience (same when disconnection happens due to network unavailability). Have faced issues in dart where it takes some time to reconnect but it always reconnects automatically.
- TeeWEE 6y agoMy experience is that it “Just Works”
- pandemic_region 6y agoSame here, for the vast majority of non-netflix scale use cases it works extremely well out of the box.
- SMFloris 6y agoRecently I started delving into gRPC vs RPC over RabbitMq using json as the mesage format. I saw that for small to medium sized messages, gRPC is actually slower. Of course, this was just a small scale experiment so I don't think nothing of it. Does anyone have a sort of infrastructure/architecture guide at a bigger scale for gRPC? My biggest questions range from: How do you actually load balance the servers? What happens if you have a sudden influx of requests but don't want to auto-scale? Do you still need a sort of queue-ing system in front of the gRPC server? In my research I wasn't able to find some noteworthy articles about this and thus triggered my curiosity.
- KMag 6y agoVery interesting. Did you also try Protobuf or MsgPack / SMILE over RabbitMq?
- SMFloris 6y agoNever tried with MsgPack, but I did try Protobuf. The reason I dislike Protobuf is that there is an extra code generation step I need to do. When using a compiled language, I guess you don't really mind the extra code generation step since you also get some safety from the compiler so you don't wind up miss-using the generated code. It is not the same when you are using an interpreted language that doesn't have really good strong typing. You will have to run a static code analyser and be very careful every time you do a change to the interface. Since in my experiments I was calling a method in Php through RabbitMq from a Rust worker, so it just proved allot simpler to just use json. Also, I measured the time it took to: 1. Make a request to the Php API 2. Php sends the rpc message on the queue 3. Rust processes that message 4. Php catches the response from the worker 5. Php returns the response to the http client It was <10ms running the cluster locally regardless if I used Protobuf or simply json.
- sandGorgon 6y agoThe protobuf vs msgpack benchmarks are not too bad. Msgpack performs very decently. https://github.com/alecthomas/go_serialization_benchmarks https://github.com/alecthomas/go_serialization_benchmarks
- pjc50 6y agoThe keepalive thing is a bit of a problem; we've been using the C# API in a client/server way, between processes on the same system written in different languages, and finding out if the other end of the connection has fallen over is surprisingly hard.
- andy_ppp 6y agoI would say gRPC is fine as long as you don’t have the following: 1) work on a team where people will make assumptions about how gRPC works because they will be wrong, for example see the article, but more generally it can be quite obtuse. 2) want to be able to easily debug over the wire information 3) expect there to be a wealth of stack overflow knowledge to be able to debug weird incompatibilities between go and JavaScript implementations 4) schema that you expect to rapidly change over time and/or versioning of said schema to avoid strange results. It’s extremely difficult to track these changes on a large codebase.
- gameswithgo 6y agoIf you were at a shop that used grpc a lot, would you not have tools ready to go that can turns the protobuf back into text? I've often thought how odd it is that the world is spending time and energy parsing things into text formats and back, when its just two computers communicating...making more work for themselves and the entire internet backbone....just because we can't be bothered to use a simple tool when debugging over the wire.
- ChymeraXYZ 6y agoIf on-the-wire readability is important you can always opt for a different marshaling method, like JSON (or even roll your own). Yes it's some extra work but not a showstopper if that's the only downside of gRPC/Protobufs.
- psanford 6y agoWireshark has supported gRPC for years. Its really not any harder than looking at HTTP2 traffic.
- kortex 6y agoThat is exactly what we do. You can write a python tool in like a dozen lines to cast to and from. There is also a way to use one of the built in protobuf tools to do it but you also have to point at the proto files yourself. I think there are other advantages to text protocols besides debugging on the wire, namely ease of composing tools. Streams of text is extremely versatile and economic for many workflows. Of course you lose that ease once you start embedding binary data.
- dalu 6y agoHow do you use a non-Google oidc idp though? That's the main thing keeping me away from it. You have the non-choice of using Google Auth or include the token in every request outside of credentials
- jahewson 6y agoYou can construct credentials from any ID token, it ends up as a standard Authorization: Bearer header on each RPC. I believe this is the case for Google Auth too.
- bpicolo 6y ago> In my opinion, this means this setting is virtually unusable and should be avoided I've used the gRPC keepalives intentionally for long-lived one directional streaming. You definitely need to pay careful attention to load balancing in the middle, but it does work well if you configure it properly. If your stream messages are infrequent, you should also add the occasional Ping message in order for the server to identify and reap stale connections. I had small ~1 core, 512mb containers handling some ~50k connections without breaking a sweat this way. The main scaling point was the goroutine stack size for each.
- fulafel 6y agoWow, sounds like AWS and Azure network services are really unsuited for a lot of real world TCP applications. You couldn't even use these to implement a load balanced telnet server for playing NetHack.
- hn_asker 6y agogRPC docs do need some tlc. Cloud providers providing support for gRPC also need to do better documenting their support for gRPC. AWS Application Load Balancer is advertised to work with gRPC but we've been seeing sporadic errors. Keepalive isn't the issue, we set the IdleTimeout to the max (4000 seconds) and use server side keepalive to gc old connections.