5 ms·

Strongly agree, it’s has loads of problems, my least favourite being the schema is not checked in the way you might think, there’s not even a checksum to say th

Strongly agree, it’s has loads of problems, my least favourite being the schema is not checked in the way you might think, there’s not even a checksum to say this message and this version of the schema match. So when there’s old services/clients around and people haven’t versioned their schema’s safely (there was no mechanism for this apart from manually checking in PRs) you can get gibberish back for fields that should contain data. It’s basically just a binary blob with whatever schema the client has overlaid so debugging is an absolute pain. Unless you are Google scale use a text based format like JSON and save yourself a lot of hassle.

discreteevent 2y ago
- JSON doesn't have any schema checking either. - You can encode the protocol buffers as JSON if you want a text based format.
jayd16 2y ago
You can trivially make breaking changes in a JSON blob too. GRPC has well documented ways to make non-breaking changes. If you're working somewhere where breaking schema changes go in with little fanfare and much debugging then I'm not sure JSON will save you. The only way to know is to dig through CLs? Write a test. There's also automated tooling to compare protobuff schemas for breaking changes.
- andy_ppp 2y ago
  JSON contains a description of the structure of the data that is readable by both machines and humans. JSON can certainly go wrong but it’s much simpler to see when it has because of this. GRPC is usually a binary black box that adds loads of developer time to upskill, debug, figure out error cases and introduces whole new classes of potential bugs. If you are building something that needs binary performance that GRPC provides, go for it, but pretending there is no extra cost over doing the obvious thing is not true.
  - aseipp 2y ago
    > JSON contains a description of the structure of the data that is readable by both machines and humans. No, it by definition does not, because JSON has no schema. Only your application contains and knows the (expected) structure of the data, but you literally cannot know what structure any random blob of JSON objects will have without a separate schema. When you read a random /docs page telling you "the structure of the resulting JSON object from this request is ...", that's just a schema but written in English instead of code. This has big downstream ramifications. For example, many APIs make the mistake of parsing JSON and only returning some opaque "Object" type, which you then have to map onto your own domain objects, meaning you actually parse every JSON object twice: once into the opaque structure, and once into your actual application type. This has major efficiency ramifications when you are actually dealing with a lot of JSON. The only way to do better than this is to have a schema in some form -- any form at all, even English prose -- so you can go from the JSON text representation directly into your domain type at parse-time. This is part of the reason why so many JSON libraries in every language tend to have some high level way of declaring a JSON object in the host language, typically as some kind of 'struct' or enum, so that they can automatically derive an actually efficient parsing step and skip intermediate objects. There's just no way around it. JSON doesn't have any schema, and that's part of its appeal, but in practice one always exists somewhere. You can use protobuf in text-based form too, but from what you said, you're probably screwed anyway if your coworkers are just churning stuff and changing the values of fields and stuff randomly. They're going to change the meaning of JSON fields willy nilly too and there will be nothing to stop you from landing back in step 1. I will say that the quality of gRPC integrations tends to vary wildly based on language though, which adds debt, you're definitely right about that.
    - andy_ppp 2y ago
      If I gave you a JSON object with name, age, position, gender etc. etc. would you not say it has structure? If I give you a GRPC binary you need the separate schema and tools to be able to comprehend it. That’s all I’m saying is the separation of the schema from some minimal structure makes the debugging of services more difficult. I would also add the GRPC implementation I used in Javascript (long ago) was not actually checking the types of the field in a lot of cases so rather than being a schema that rejects if some field is not a text field it would just return binary junk. JSON Schema or almost anything else will give you a parsing error instead. Maybe the tools are fantastic not but I still think being able to debug messages without them is an advantage in almost all systems, you probably don’t need the level of performance GRPC provides. If you’re using JSON Protobufs why would you add this extra complexity - it will mean messaging is just as slow as using JSON. What are the core advantages of GRPC under these conditions?
jeffbee 2y ago
There is an art to having forwards and backwards compatible RPC schemas. It is easy, but it is surprisingly difficult to get people to follow easy rules. The rules are as follows: 1) Never change the type of a field 2) Never change the semantic meaning of a field 3) If you need a different type or semantics, add a new field Pretty simple if you ask me.
- andy_ppp 2y ago
  If I got to choose my colleagues this would be fine, unfortunately I had people who couldn’t understand eventual consistency. One of the guys writing Go admitted he didn’t understand what a pointer was etc. etc.
  - lrem 2y ago
    How does JSON protect you from that?
    - andy_ppp 2y ago
      People understand JSON fairly commonly as they can see what is happening in a browser or any other system - what is the equivalent for GRPC if I want to do console.log(json)? GRPC for most people is a completely black box with unclear error conditions that are not as clear to me at least. For example what happens if I have an old schema and I'm not seeing a field, there's loads of things that can be wrong - old services, old client, even messages not being routed correctly due to networking settings in docker or k8s. Are you denying there is absolutely tones to learn here and it is trickier to debug and maintain?
      - lrem 2y ago
        I'd go with `cerr << request.DebugString() << '...' << response.DebugString();`, preferably with your favourite logger instead of `stdout`. My browser does the equivalent for me just fine, but that required an extension. I buy the familiarity argument, but I usually don't see the wire format at all. And maintenance-wise, protobufs seem easier to me. But that's because, e.g., someone set up a presubmit for me that yells at me if my change isn't backwards compatible. That's kind of hard to do if you don't have a formal specification of what goes into your protocol.

discreteevent 2y ago

- JSON doesn't have any schema checking either. - You can encode the protocol buffers as JSON if you want a text based format.

jayd16 2y ago

You can trivially make breaking changes in a JSON blob too. GRPC has well documented ways to make non-breaking changes. If you're working somewhere where breaking schema changes go in with little fanfare and much debugging then I'm not sure JSON will save you. The only way to know is to dig through CLs? Write a test. There's also automated tooling to compare protobuff schemas for breaking changes.

andy_ppp 2y ago
JSON contains a description of the structure of the data that is readable by both machines and humans. JSON can certainly go wrong but it’s much simpler to see when it has because of this. GRPC is usually a binary black box that adds loads of developer time to upskill, debug, figure out error cases and introduces whole new classes of potential bugs. If you are building something that needs binary performance that GRPC provides, go for it, but pretending there is no extra cost over doing the obvious thing is not true.
- aseipp 2y ago
  > JSON contains a description of the structure of the data that is readable by both machines and humans. No, it by definition does not, because JSON has no schema. Only your application contains and knows the (expected) structure of the data, but you literally cannot know what structure any random blob of JSON objects will have without a separate schema. When you read a random /docs page telling you "the structure of the resulting JSON object from this request is ...", that's just a schema but written in English instead of code. This has big downstream ramifications. For example, many APIs make the mistake of parsing JSON and only returning some opaque "Object" type, which you then have to map onto your own domain objects, meaning you actually parse every JSON object twice: once into the opaque structure, and once into your actual application type. This has major efficiency ramifications when you are actually dealing with a lot of JSON. The only way to do better than this is to have a schema in some form -- any form at all, even English prose -- so you can go from the JSON text representation directly into your domain type at parse-time. This is part of the reason why so many JSON libraries in every language tend to have some high level way of declaring a JSON object in the host language, typically as some kind of 'struct' or enum, so that they can automatically derive an actually efficient parsing step and skip intermediate objects. There's just no way around it. JSON doesn't have any schema, and that's part of its appeal, but in practice one always exists somewhere. You can use protobuf in text-based form too, but from what you said, you're probably screwed anyway if your coworkers are just churning stuff and changing the values of fields and stuff randomly. They're going to change the meaning of JSON fields willy nilly too and there will be nothing to stop you from landing back in step 1. I will say that the quality of gRPC integrations tends to vary wildly based on language though, which adds debt, you're definitely right about that.
  - andy_ppp 2y ago
    If I gave you a JSON object with name, age, position, gender etc. etc. would you not say it has structure? If I give you a GRPC binary you need the separate schema and tools to be able to comprehend it. That’s all I’m saying is the separation of the schema from some minimal structure makes the debugging of services more difficult. I would also add the GRPC implementation I used in Javascript (long ago) was not actually checking the types of the field in a lot of cases so rather than being a schema that rejects if some field is not a text field it would just return binary junk. JSON Schema or almost anything else will give you a parsing error instead. Maybe the tools are fantastic not but I still think being able to debug messages without them is an advantage in almost all systems, you probably don’t need the level of performance GRPC provides. If you’re using JSON Protobufs why would you add this extra complexity - it will mean messaging is just as slow as using JSON. What are the core advantages of GRPC under these conditions?

jeffbee 2y ago

There is an art to having forwards and backwards compatible RPC schemas. It is easy, but it is surprisingly difficult to get people to follow easy rules. The rules are as follows: 1) Never change the type of a field 2) Never change the semantic meaning of a field 3) If you need a different type or semantics, add a new field Pretty simple if you ask me.

andy_ppp 2y ago
If I got to choose my colleagues this would be fine, unfortunately I had people who couldn’t understand eventual consistency. One of the guys writing Go admitted he didn’t understand what a pointer was etc. etc.
- lrem 2y ago
  How does JSON protect you from that?
  - andy_ppp 2y ago
    People understand JSON fairly commonly as they can see what is happening in a browser or any other system - what is the equivalent for GRPC if I want to do console.log(json)? GRPC for most people is a completely black box with unclear error conditions that are not as clear to me at least. For example what happens if I have an old schema and I'm not seeing a field, there's loads of things that can be wrong - old services, old client, even messages not being routed correctly due to networking settings in docker or k8s. Are you denying there is absolutely tones to learn here and it is trickier to debug and maintain?
    - lrem 2y ago
      I'd go with `cerr << request.DebugString() << '...' << response.DebugString();`, preferably with your favourite logger instead of `stdout`. My browser does the equivalent for me just fine, but that required an extension. I buy the familiarity argument, but I usually don't see the wire format at all. And maintenance-wise, protobufs seem easier to me. But that's because, e.g., someone set up a presubmit for me that yells at me if my change isn't backwards compatible. That's kind of hard to do if you don't have a formal specification of what goes into your protocol.