10 ms·

Yeah. Easy things are easy with most technologies... It's only after a while that you start to see the 'problems'. With grpc... It's designed by Google for Goo

Yeah. Easy things are easy with most technologies... It's only after a while that you start to see the 'problems'. With grpc... It's designed by Google for Google's use case. How they do things and the design trade-offs they made are quite specific, and may not make sense for you. There are no generated language interfaces, so you cannot mock the methods. (Except by mocking abstract classes, and nobody sane does that, right) That's because grpc allows you to implement whatever methods you like of a service interface, and require any fields you like - all are optional, but not really, right. Things that you might expect to be invalid, are valid. A zero byte array deserialised as a protobuf message is a perfectly valid message. All the strings are "" (not null), the bools false, and the ints 0. Load balancing is done by maintaining multiple connections to all upstreams. The messages dont work very well with ALB/ELB. The tooling for web clients was terrible ( I understand this may have changed ) The grpc generated classes are a load of slowly compiling not very nice code. Like I say, if your tech and business is like Google's ( it probably isn't) then it's a shoe-in, else it's definitely worth asking if there is a match for your needs.

buckhx 7y ago
I am by no means arguing with your general point, but some of these and may be language specific. For example in Go, the service definitions are generated as interfaces and come with an "Unimplemented" concrete client. We have a codegen package that builds mock concrete implementations of services for use in tests. Zero values are also the standard in Go and fit most use cases. We have "optional" types defined as messages that wrap primitives such as floats for times when a true null is needed (has been mostly used for update type methods). The web clients work, but generate ALOT of code. We're using the improbable package so we can provide different transports for broswer vs server JS clients btw. The big win we've seen from grpc is being able to reason about the entire system end to end and have a central language for conversation and contracts across teams. Sure there are other ways to accomplish that, but grpc has served that purpose for us.
- ptoomey3 7y ago
  Our main hinderance with gRPC was that several disparate teams had strange issues with the fairly opaque runtime. The “batteries included” approach made attempts to debug the root causes quite difficult. As a result of the above, we have been exploring twirp. You get the benefits of using protobufs for defining the RPC interface, but without quite as much runtime baggage that complicates debugging issues that arise.
  - kjeetgill 7y ago
    That's always the problem with "batteries included". If they don't work it's often not worth the effort to fix them; you gotta toss em. I'm curious what languages you were using gRPC with. The batteries includedness across tons of languages is a big part of gRPC's appeal. I'd assume Java and C++ get enough use to be solid but maybe that's wishful thinking?
    - ptoomey3 7y ago
      We were mostly using ruby (which uses their C bindings) and golang (which are native to golang).
      - cachvico 7y ago
        What kind of problems did you run into, if you don't mind sharing?
        ptoomey3 7y ago
        One that we encountered in several services were gRPC ruby clients that semi-regularly blocked on responses for an indeterminate amount of time. We added lots of tracing data on the client and server to instrument where “slowness” was occurring. We would see every trace span look just like you would hope until the message went into the runtime and failed to get passed up to the caller for some random long period of time. Debugging what was happening between the network response (fast) and the actual parsed response being handed to the caller (slow) was quite frustrating, as it requires trying to dig into C bindings/runtime from the ruby client.
denormalfloat 7y ago
> There are no generated language interfaces, so you cannot mock the methods. For Java that isn't true. Java ships with a lightweight InProcess server to stub out the responses to your client. > Load balancing is done by maintaining multiple connections to all upstreams. Load balancing is fully pluggable. The default balancer only picks the first connection. > The tooling for web clients was terrible Agreed. This is almost entirely the fault of Chrome and Firefox, for not implementing the HTTP/2 spec properly. (missing trailers).
- time4tea 7y ago
  There is a big difference in my book between starting up an in process server (requiring all sorts of grpc naming magic) running your extensions to an abstract class inside the grpc magic, and a language level interface. On the one level you can say new X(new MyService()) or new X(mock(Service.class))) if you have to, and on the other its just loads of jibber-jabber.
  - denormalfloat 7y ago
    There is a reason mocking is not supported: It's incredibly error prone. The previous version of gRPC (Stubby) did actually support mocking of the stub/service interfaces. The issue is that people mock out responses that don't map to reality, causing the tests to pass but the system-under-test to blow up. This happened often enough that ability to mock was ripped out, and the InProcess server-client added. The extra "jibber-jabber" is what makes people confident that their Stub usage is correct. Some sample bugs NOT caught by a mock: * Calling the stub with a NULL message * Not calling close() * Sending invalid headers * Ignoring deadlines * Ignoring cancellation There's more, but these are real bugs that are trivially caught by using a real (and cheap) server.
kjeetgill 7y ago
> With grpc... It's designed by Google for Google's use case. How they do things and the design trade-offs they made are quite specific, and may not make sense for you. Agreed. It's always important to try to pick technologies that 'align' with your use-cases as well as possible. This is easier said than done and gets easier the more often you fail to do it well! I do think people will read "for Google's use case" and hear "only for Google's scale". I actually think the gRPC Java stack is pretty efficient so it "scales down" pretty well. I want to skip over some of what you're saying to address this: > Things that you might expect to be invalid, are valid. A zero byte array deserialised as a protobuf message is a perfectly valid message. All the strings are "" (not null), the bools false, and the ints 0. Using a protobuf schema layer is wayyyy nicer than JSON blobs but I agree that it is misconstrued as type safety and validation. It's fantastic for efficient data marshaling and decent for code generation but it dosn't solve the "semantic correctness" side of things. You should still be writing validation. Its a solid step up from JSON not a panacea.
- alasdair_ 7y ago
  JSON has a bunch of schema systems, including Open API which is a repackaging of Swagger with some extra stuff and is also endorsed by Google. Do you consider protobuf superior to those alternatives for web-based (rather than server to server) projects?
  - kjeetgill 7y ago
    I spend all my time server to server so I don't feel qualified to give real advice. My impression is that if you're going to talk to a browser, that edge stands to gain much more from conforming to HTTP standards. If your edge is more "applicationy" and less "webpagy" then maybe a browser facing gRPC (or GraphQL?) might be more appealing again. As to the other JSON schema systems, I kinda wish one of them won? It feels like a lot of competing standards still. Not really my area of expertise.
    - luhn 7y ago
      I think OpenAPI/Swagger has won. Haven’t heard of any others recently.
dub 7y ago
You can add whatever custom validation you want on top of proto3 (using annotations if you like). Required fields aren't very useful at a serialization level: adding a new required field would always be a backwards incompatible change. You should never do it. They're only useful if you have total certainty that you can define all your APIs perfectly on the first try. But again, if you really want them you can always build something like https://github.com/envoyproxy/protoc-gen-validate https://github.com/envoyproxy/protoc-gen-validate on top. That's the benefit of a simple and extensible system vs one that tries to bake-in unnecessary or problematic default behaviors. Also: why wouldn't grpc work well with load balancers? It's based on HTTP/2. It's well supported by envoy, which is fast-becoming the de facto standard proxy for service meshes.
- alasdair_ 7y ago
  >Also: why wouldn't grpc work well with load balancers? It's based on HTTP/2 You answered your own question. There is always some bit of older infrastructure, like a caching proxy or “enterprise” load balancer that doesn’t quite understand http/2 yet - it is the same reason so much Internet traffic is still on ipv4 when ipv6 exists - the lowest common denominator end up winning for some subsection of traffic.
  - tedk-42 7y ago
    Not wrong on this. We use gRPC in our tech stack but some pods handle far more connections than others due to the multiplexing/reuse of existing connections. Sadly Istio/Envoy solutions are in our backlog for now. We can't fault gRPC otherwise. It's way faster than if we were to encode/decode json after each microservice hop. It's integrates into golang nicely (another google coolaid solution!) so a win-win there.
dilyevsky 7y ago
> The messages dont work very well with ALB/ELB. Doesn't work on ALB because its HTTP/2 support is trash, not gRPC's fault here. Works fine with NLB btw. > Load balancing is done by maintaining multiple connections to all upstreams. Again, this is a "feature" of HTTP/2. Use linkerd or envoy that support subsetting among other useful things. Don't blame your misunderstanding of how technology is meant to be used on said technology.
- adamson 7y ago
  This feels like a circular argument. Not everyone needs HTTP/2 support (especially for internal services for enterprise applications).
  - ptoomey3 7y ago
    Need or not need...a bigger issue is simply the technical reality of what you have now. If your non-trivial infrastructure doesn’t have great http2 support, it might be a pretty big lift to make that change first.
    - dilyevsky 7y ago
      Yes, it is such a foundational thing that it has to be woven into how you run your infrastructure. Can't just drop it in in most cases. If my memory serves me well, Google basically re-started their entire codebase/infra from scratch - google3 (version 2 was skipped, apparently) to accommodate this shift.
adamson 7y ago
> Things that you might expect to be invalid, are valid. A zero byte array deserialised as a protobuf message is a perfectly valid message. All the strings are "" (not null), the bools false, and the ints 0. How does this work? How do you make, say, all fields but the second null? Do you just send a messages that's (after encoding) as long as the first two fields, where the first field is 0x00 and the second contains whatever data you want?
- violinist 7y ago
  Two things: 1) proto buffers intentionally don't allow null values; values that aren't set will return a default value 2) gRPC uses proto3, which does not distinguish between a field unset and a field set to the default value
damnyou 7y ago
You should never use protobuf types directly in your code. Always convert to native types at the edges — that will let you do validation.
- kbwt 7y ago
  At that point, what does Protobuf really buy you?
  - damnyou 7y ago
    A rock solid serialization and RPC system.
mleonhard 7y ago
One more thing that's a show-stopper for any public service: No flow control. This means anybody who can connect to your gRPC server can OOM it.
- dikei 7y ago
  I suppose you can use a proxy to perform rate-limiting.

buckhx 7y ago

I am by no means arguing with your general point, but some of these and may be language specific. For example in Go, the service definitions are generated as interfaces and come with an "Unimplemented" concrete client. We have a codegen package that builds mock concrete implementations of services for use in tests. Zero values are also the standard in Go and fit most use cases. We have "optional" types defined as messages that wrap primitives such as floats for times when a true null is needed (has been mostly used for update type methods). The web clients work, but generate ALOT of code. We're using the improbable package so we can provide different transports for broswer vs server JS clients btw. The big win we've seen from grpc is being able to reason about the entire system end to end and have a central language for conversation and contracts across teams. Sure there are other ways to accomplish that, but grpc has served that purpose for us.

ptoomey3 7y ago
Our main hinderance with gRPC was that several disparate teams had strange issues with the fairly opaque runtime. The “batteries included” approach made attempts to debug the root causes quite difficult. As a result of the above, we have been exploring twirp. You get the benefits of using protobufs for defining the RPC interface, but without quite as much runtime baggage that complicates debugging issues that arise.
- kjeetgill 7y ago
  That's always the problem with "batteries included". If they don't work it's often not worth the effort to fix them; you gotta toss em. I'm curious what languages you were using gRPC with. The batteries includedness across tons of languages is a big part of gRPC's appeal. I'd assume Java and C++ get enough use to be solid but maybe that's wishful thinking?
  - ptoomey3 7y ago
    We were mostly using ruby (which uses their C bindings) and golang (which are native to golang).
    - cachvico 7y ago
      What kind of problems did you run into, if you don't mind sharing?
      - ptoomey3 7y ago
        One that we encountered in several services were gRPC ruby clients that semi-regularly blocked on responses for an indeterminate amount of time. We added lots of tracing data on the client and server to instrument where “slowness” was occurring. We would see every trace span look just like you would hope until the message went into the runtime and failed to get passed up to the caller for some random long period of time. Debugging what was happening between the network response (fast) and the actual parsed response being handed to the caller (slow) was quite frustrating, as it requires trying to dig into C bindings/runtime from the ruby client.

denormalfloat 7y ago

> There are no generated language interfaces, so you cannot mock the methods. For Java that isn't true. Java ships with a lightweight InProcess server to stub out the responses to your client. > Load balancing is done by maintaining multiple connections to all upstreams. Load balancing is fully pluggable. The default balancer only picks the first connection. > The tooling for web clients was terrible Agreed. This is almost entirely the fault of Chrome and Firefox, for not implementing the HTTP/2 spec properly. (missing trailers).

time4tea 7y ago
There is a big difference in my book between starting up an in process server (requiring all sorts of grpc naming magic) running your extensions to an abstract class inside the grpc magic, and a language level interface. On the one level you can say new X(new MyService()) or new X(mock(Service.class))) if you have to, and on the other its just loads of jibber-jabber.
- denormalfloat 7y ago
  There is a reason mocking is not supported: It's incredibly error prone. The previous version of gRPC (Stubby) did actually support mocking of the stub/service interfaces. The issue is that people mock out responses that don't map to reality, causing the tests to pass but the system-under-test to blow up. This happened often enough that ability to mock was ripped out, and the InProcess server-client added. The extra "jibber-jabber" is what makes people confident that their Stub usage is correct. Some sample bugs NOT caught by a mock: * Calling the stub with a NULL message * Not calling close() * Sending invalid headers * Ignoring deadlines * Ignoring cancellation There's more, but these are real bugs that are trivially caught by using a real (and cheap) server.

kjeetgill 7y ago

> With grpc... It's designed by Google for Google's use case. How they do things and the design trade-offs they made are quite specific, and may not make sense for you. Agreed. It's always important to try to pick technologies that 'align' with your use-cases as well as possible. This is easier said than done and gets easier the more often you fail to do it well! I do think people will read "for Google's use case" and hear "only for Google's scale". I actually think the gRPC Java stack is pretty efficient so it "scales down" pretty well. I want to skip over some of what you're saying to address this: > Things that you might expect to be invalid, are valid. A zero byte array deserialised as a protobuf message is a perfectly valid message. All the strings are "" (not null), the bools false, and the ints 0. Using a protobuf schema layer is wayyyy nicer than JSON blobs but I agree that it is misconstrued as type safety and validation. It's fantastic for efficient data marshaling and decent for code generation but it dosn't solve the "semantic correctness" side of things. You should still be writing validation. Its a solid step up from JSON not a panacea.

alasdair_ 7y ago
JSON has a bunch of schema systems, including Open API which is a repackaging of Swagger with some extra stuff and is also endorsed by Google. Do you consider protobuf superior to those alternatives for web-based (rather than server to server) projects?
- kjeetgill 7y ago
  I spend all my time server to server so I don't feel qualified to give real advice. My impression is that if you're going to talk to a browser, that edge stands to gain much more from conforming to HTTP standards. If your edge is more "applicationy" and less "webpagy" then maybe a browser facing gRPC (or GraphQL?) might be more appealing again. As to the other JSON schema systems, I kinda wish one of them won? It feels like a lot of competing standards still. Not really my area of expertise.
  - luhn 7y ago
    I think OpenAPI/Swagger has won. Haven’t heard of any others recently.

dub 7y ago

You can add whatever custom validation you want on top of proto3 (using annotations if you like). Required fields aren't very useful at a serialization level: adding a new required field would always be a backwards incompatible change. You should never do it. They're only useful if you have total certainty that you can define all your APIs perfectly on the first try. But again, if you really want them you can always build something like https://github.com/envoyproxy/protoc-gen-validate https://github.com/envoyproxy/protoc-gen-validate on top. That's the benefit of a simple and extensible system vs one that tries to bake-in unnecessary or problematic default behaviors. Also: why wouldn't grpc work well with load balancers? It's based on HTTP/2. It's well supported by envoy, which is fast-becoming the de facto standard proxy for service meshes.

alasdair_ 7y ago
>Also: why wouldn't grpc work well with load balancers? It's based on HTTP/2 You answered your own question. There is always some bit of older infrastructure, like a caching proxy or “enterprise” load balancer that doesn’t quite understand http/2 yet - it is the same reason so much Internet traffic is still on ipv4 when ipv6 exists - the lowest common denominator end up winning for some subsection of traffic.
- tedk-42 7y ago
  Not wrong on this. We use gRPC in our tech stack but some pods handle far more connections than others due to the multiplexing/reuse of existing connections. Sadly Istio/Envoy solutions are in our backlog for now. We can't fault gRPC otherwise. It's way faster than if we were to encode/decode json after each microservice hop. It's integrates into golang nicely (another google coolaid solution!) so a win-win there.

dilyevsky 7y ago

> The messages dont work very well with ALB/ELB. Doesn't work on ALB because its HTTP/2 support is trash, not gRPC's fault here. Works fine with NLB btw. > Load balancing is done by maintaining multiple connections to all upstreams. Again, this is a "feature" of HTTP/2. Use linkerd or envoy that support subsetting among other useful things. Don't blame your misunderstanding of how technology is meant to be used on said technology.

adamson 7y ago
This feels like a circular argument. Not everyone needs HTTP/2 support (especially for internal services for enterprise applications).
- ptoomey3 7y ago
  Need or not need...a bigger issue is simply the technical reality of what you have now. If your non-trivial infrastructure doesn’t have great http2 support, it might be a pretty big lift to make that change first.
  - dilyevsky 7y ago
    Yes, it is such a foundational thing that it has to be woven into how you run your infrastructure. Can't just drop it in in most cases. If my memory serves me well, Google basically re-started their entire codebase/infra from scratch - google3 (version 2 was skipped, apparently) to accommodate this shift.

adamson 7y ago

> Things that you might expect to be invalid, are valid. A zero byte array deserialised as a protobuf message is a perfectly valid message. All the strings are "" (not null), the bools false, and the ints 0. How does this work? How do you make, say, all fields but the second null? Do you just send a messages that's (after encoding) as long as the first two fields, where the first field is 0x00 and the second contains whatever data you want?

violinist 7y ago
Two things: 1) proto buffers intentionally don't allow null values; values that aren't set will return a default value 2) gRPC uses proto3, which does not distinguish between a field unset and a field set to the default value

damnyou 7y ago

You should never use protobuf types directly in your code. Always convert to native types at the edges — that will let you do validation.

kbwt 7y ago
At that point, what does Protobuf really buy you?
- damnyou 7y ago
  A rock solid serialization and RPC system.

mleonhard 7y ago

One more thing that's a show-stopper for any public service: No flow control. This means anybody who can connect to your gRPC server can OOM it.

dikei 7y ago
I suppose you can use a proxy to perform rate-limiting.