6 ms·

I would argue that, in the long run, gRPC/protobuf has a lower operational cost than JSON as long as you don't need to talk to it from a browser. (Consuming it

I would argue that, in the long run, gRPC/protobuf has a lower operational cost than JSON as long as you don't need to talk to it from a browser. (Consuming it from a browser is a hassle because client-side JavaScript code is unable to speak the full gRPC API, so you need to fuss with reverse proxies to get everything working.) What it doesn't have is a short learning curve. In order to get started, you need to learn the *.proto format, and how to use the code generator, and all the design implementations of the different data types it supports, and all of that. But, once you get over that hump, it makes a lot of the hard stuff much, much easier to get right. What I keep wishing for is some sort of "gRPC-lite" that doesn't include quite as many questionable micro-optimizations as protobuf/gRPC, but does include all of the really good ideas like specification-first API development, service reflection, and a clean logical separation between HTTP semantics and the semantics of the API that's being implemented on top of it.

nawgz 5y ago
> gRPC/protobuf has a lower operational cost than JSON as long as you don't need to talk to it from a browser. So, reading this the other way (browsers are king)... you claim gRPC is so useless as to not be able to power your entire system and requires you to standup duplicate interchange systems for different use cases? Yikes.
- mumblemumble 5y ago
  Browsers are king for some people, not others. For the stuff I'm working on, the browser is just the tip of the iceberg. For everything below the waterline, the operational benefits (strong static typing, well-defined backward- and forward-compatibility semantics, better throughput and latency characteristics) greatly outweigh the, "but we have to use a lightweight Envoy reverse proxy to expose some things to the browser," problem. I also have a tendency to consider that Envoy proxy to be more of a feature than a bug, anyway. It's pretty easy to set up, all told. We want to gatekeep the edge, anyway, for various reasons, so it's not like there was ever a reality in which we weren't going to be fussing with a reverse proxy. And it serves as a nice opportunity to stop and be thoughtful about exactly what we're choosing to expose to the Internet. Speaking purely as a developer, I do find it to be an annoyance. But I also acknowledge that inconveniencing developers for the sake of the greater good can be a wise move.
- bob_roberts 5y ago
  For a cloud-based app, that might literally just be at the application gateway. Beyond that, everything could be whatever protocol.
throwaway894345 5y ago
I tried to get into gRPC/protobuf in Go on a Mac for a little hobby project, but man the effort just to get protoc up and running and then generate the stubs was insane. I'm sure somehow or another, it's user error, but when the barrier of entry is so high, it's hard to justify the effort when JSON-slinging is so rarely the bottleneck.
Cloudef 5y ago
protobuf incurs lots of codebloat (especially with google's runtime / compilers) and the the serialization format is not really that nice IMO. I don't think it's possible to come up with 100% ideal format for all the use cases.
tiew9Vii 5y ago
If you never need to hit a browser Avro, Thrift, Protobuff etc., are way better and more straightforward options to deal with than JSON. You have a strong schema, not a retrofitted on top JSON schema spec. The schema is enforced as the data is serialized to binary format. They have schema evolution functionality. You get the advantages of binary format, faster de/serialization speed, less bandwidth. You are not writing your own decoders/encoders anymore. They all have code gen to generate domain objects in your language of choice. You have the schemas in a central repo and a CI job that reads them publishing a package for your language, i.e. JAR's, NPM's, Crates etc. etc. Users just pull the artifact. This part makes dev easier. You just import the artifact and can read/write the data with no effort/code. I've had success using Protobuff in the browser without the GRPC using protobufjs. My API is a typical HTTP Rest service, but instead of sending/receiving JSON it uses Protobuff as the message format. It has made life really easy as I just import the generated Protobuf NPM, and when I make a fetch call, I deserialize the result to an object by using the auto-generated decode function and then I can call `.getFiled()`. As I'm using Typescript on the client-side, I get type checking and type hints in the IDE. If there's no `getField` function, I know that field doesn't exist. The advantage I now have is I don't need to do OpenAPI stuff. I get that for free. I have the request objects and response objects as protobuf so there's a strongly enforced schema file acting as the contract, and I can produce client packages in any language for free. The other benefit is it has reduced response sizes and latency. The response size was the main reason I did a POC with protobuf as working on a mapping application and sending polylines to be displayed on a map meant I was sending 10Mb+ of JSON to the browser. Protobuff significantly reduced this. The caveat is Protobuff are hard to debug in the browser. You can't just use the net inspector as you have binary data, not plain text. For any personal project going forward, I'll use Protobuf instead of JSON. The biggest annoyance I have is writing JSON de/serialisers in my serverside language then having to do the same for the client-side language/app, them getting out of sync when the API evolves or making typos. Using Protobuf removes all those pain points, import the package, use the package, the specification has schematics around schema evolution so API changes is just updating to the latest package. The only downside is if you have a public API no one else does it so everyone expects JSON, a non issue for private projects.
makeitdouble 5y ago
> gRPC/protobuf has a lower operational cost than JSON I'm a bit torn on this point. We had a project where we needed to talk to an external grpc service, and protobuf was a technical given. Outside of the bootstrapping and typesetting, an issue we hit was the lack of tooling: Google officially supports protobuf libraries for 4 or 5 languages. There are other library providers, in particular for JS/Typescript, but from the get go there needs to be a strong library support or it's out of question. For instance looking at ruby, protobuf libs seem to be young at cursory glance. It's true to some extent for JSON as well (though writing a parser for a limited set wouldn't be impossible), but I'd be hard press to come with a language that doesn't have a good JSON parser. Then debugging also means relying on your library to encode/decode the payloads. If you need to replay a slightly modified query to check how an entry point reacts, you'll need to debug from your application, or build another test app just to emulate the requests. This issue gets compounded if you have doubt in your libraries' reliability. Development, maintenance, logging, monitoring etc. get that much heavier to deal with. All in all I don't see protobuf as aiming for low operational cost. Or at least you need to be ready to bear the weight, and might be lucky if it ends up lighter than expected. JSON can be scaled upward in reliability, with heavy testing and validation layers, while protobuf is hard to lighten.
- mumblemumble 5y ago
  That is a fair consideration. And part of what I was thinking about when I suggested that gRPC needs a slimmed-down alternative. I think that part of the reason for weaker support on languages that Google doesn't personally maintain is that the gRPC protocol and tooling itself seems to be so technically complex that it's not really amenable to community support. And I really don't think all that complexity is necessary. For example, in my own testing I found that the streaming protocol is generally not worth the extra hassle. Just getting rid of that would probably reduce the implementation cost considerably, and also eliminate the reason why client-side code can't speak gRPC.

nawgz 5y ago

> gRPC/protobuf has a lower operational cost than JSON as long as you don't need to talk to it from a browser. So, reading this the other way (browsers are king)... you claim gRPC is so useless as to not be able to power your entire system and requires you to standup duplicate interchange systems for different use cases? Yikes.

mumblemumble 5y ago
Browsers are king for some people, not others. For the stuff I'm working on, the browser is just the tip of the iceberg. For everything below the waterline, the operational benefits (strong static typing, well-defined backward- and forward-compatibility semantics, better throughput and latency characteristics) greatly outweigh the, "but we have to use a lightweight Envoy reverse proxy to expose some things to the browser," problem. I also have a tendency to consider that Envoy proxy to be more of a feature than a bug, anyway. It's pretty easy to set up, all told. We want to gatekeep the edge, anyway, for various reasons, so it's not like there was ever a reality in which we weren't going to be fussing with a reverse proxy. And it serves as a nice opportunity to stop and be thoughtful about exactly what we're choosing to expose to the Internet. Speaking purely as a developer, I do find it to be an annoyance. But I also acknowledge that inconveniencing developers for the sake of the greater good can be a wise move.
bob_roberts 5y ago
For a cloud-based app, that might literally just be at the application gateway. Beyond that, everything could be whatever protocol.

throwaway894345 5y ago

I tried to get into gRPC/protobuf in Go on a Mac for a little hobby project, but man the effort just to get protoc up and running and then generate the stubs was insane. I'm sure somehow or another, it's user error, but when the barrier of entry is so high, it's hard to justify the effort when JSON-slinging is so rarely the bottleneck.

Cloudef 5y ago

protobuf incurs lots of codebloat (especially with google's runtime / compilers) and the the serialization format is not really that nice IMO. I don't think it's possible to come up with 100% ideal format for all the use cases.

tiew9Vii 5y ago

If you never need to hit a browser Avro, Thrift, Protobuff etc., are way better and more straightforward options to deal with than JSON. You have a strong schema, not a retrofitted on top JSON schema spec. The schema is enforced as the data is serialized to binary format. They have schema evolution functionality. You get the advantages of binary format, faster de/serialization speed, less bandwidth. You are not writing your own decoders/encoders anymore. They all have code gen to generate domain objects in your language of choice. You have the schemas in a central repo and a CI job that reads them publishing a package for your language, i.e. JAR's, NPM's, Crates etc. etc. Users just pull the artifact. This part makes dev easier. You just import the artifact and can read/write the data with no effort/code. I've had success using Protobuff in the browser without the GRPC using protobufjs. My API is a typical HTTP Rest service, but instead of sending/receiving JSON it uses Protobuff as the message format. It has made life really easy as I just import the generated Protobuf NPM, and when I make a fetch call, I deserialize the result to an object by using the auto-generated decode function and then I can call `.getFiled()`. As I'm using Typescript on the client-side, I get type checking and type hints in the IDE. If there's no `getField` function, I know that field doesn't exist. The advantage I now have is I don't need to do OpenAPI stuff. I get that for free. I have the request objects and response objects as protobuf so there's a strongly enforced schema file acting as the contract, and I can produce client packages in any language for free. The other benefit is it has reduced response sizes and latency. The response size was the main reason I did a POC with protobuf as working on a mapping application and sending polylines to be displayed on a map meant I was sending 10Mb+ of JSON to the browser. Protobuff significantly reduced this. The caveat is Protobuff are hard to debug in the browser. You can't just use the net inspector as you have binary data, not plain text. For any personal project going forward, I'll use Protobuf instead of JSON. The biggest annoyance I have is writing JSON de/serialisers in my serverside language then having to do the same for the client-side language/app, them getting out of sync when the API evolves or making typos. Using Protobuf removes all those pain points, import the package, use the package, the specification has schematics around schema evolution so API changes is just updating to the latest package. The only downside is if you have a public API no one else does it so everyone expects JSON, a non issue for private projects.

makeitdouble 5y ago

> gRPC/protobuf has a lower operational cost than JSON I'm a bit torn on this point. We had a project where we needed to talk to an external grpc service, and protobuf was a technical given. Outside of the bootstrapping and typesetting, an issue we hit was the lack of tooling: Google officially supports protobuf libraries for 4 or 5 languages. There are other library providers, in particular for JS/Typescript, but from the get go there needs to be a strong library support or it's out of question. For instance looking at ruby, protobuf libs seem to be young at cursory glance. It's true to some extent for JSON as well (though writing a parser for a limited set wouldn't be impossible), but I'd be hard press to come with a language that doesn't have a good JSON parser. Then debugging also means relying on your library to encode/decode the payloads. If you need to replay a slightly modified query to check how an entry point reacts, you'll need to debug from your application, or build another test app just to emulate the requests. This issue gets compounded if you have doubt in your libraries' reliability. Development, maintenance, logging, monitoring etc. get that much heavier to deal with. All in all I don't see protobuf as aiming for low operational cost. Or at least you need to be ready to bear the weight, and might be lucky if it ends up lighter than expected. JSON can be scaled upward in reliability, with heavy testing and validation layers, while protobuf is hard to lighten.

mumblemumble 5y ago
That is a fair consideration. And part of what I was thinking about when I suggested that gRPC needs a slimmed-down alternative. I think that part of the reason for weaker support on languages that Google doesn't personally maintain is that the gRPC protocol and tooling itself seems to be so technically complex that it's not really amenable to community support. And I really don't think all that complexity is necessary. For example, in my own testing I found that the streaming protocol is generally not worth the extra hassle. Just getting rid of that would probably reduce the implementation cost considerably, and also eliminate the reason why client-side code can't speak gRPC.