5 ms·

schema-driven development is great, but I wish people spent more time on JSON-based API schema rather than switch to protobuf. Because protobufs immediately req

by theamk 3y ago

schema-driven development is great, but I wish people spent more time on JSON-based API schema rather than switch to protobuf. Because protobufs immediately require some minimal level of complexity (at least a build system!) and thus effectively block small one-off scripts. And the debuggability is also not great. Can we (as an industry) pick a standard (openapi? jsonschema? something else?) and add the tooling to match protobuf ecosystem functionality?

stubish 3y ago
I don't think a large group can pick a standard. What works well for one programming language does not work for another, and we devolve into tribes. Protobuf gained popularity when it dropped features to better support Go more naturally (no more null/nil for example), and rode in on a wave of Go adoption. And made things worse from a Python developer perspective. Pretty much the same as toml adoption, which fits more naturally with Go and more static languages, unlike the much more dynamic and flexible yaml.
- Supermancho 3y ago
  > I don't think a large group can pick a standard. There are plenty of cases where this has happened. JSON has already become the defacto standard. It's not a the step to formalize a ubiquitous format has been taken many times. Every major platform supports it or uses it exclusively. Some frameworks and tooling, only uses JSON (true of other formats to an extent, ie json more than yaml). > What works well for one programming language does not work for another, I'm not sure why this roundabout whataboutism is compelling. A standard protocol is used in absence of other restrictions or benefits of another This doesn't snap away the other protocols from existence.
- arp242 3y ago
  TOML doesn't "fit static languages" better and is just as dynamically typed as YAML. It's used extensively in the Python ecosystem, which has TOML in stdlib (not YAML). I'm less sure about protobufs in Python (I never tried), but in general relying too much on Python's dynamic nature is not a good thing IMHO.
hamandcheese 3y ago
GRPC <=> Rest transcoding works fine, you can do this with envoy for example. And if you use the connect protocol (also from buf) then you can also use json. So it's quite achievable to get all the benefits you want from the gRPC ecosystem for developing your service while also allowing quick and dirty client usage (i.e. scripting with cURL).
no_circuit 3y ago
I fail to see how running something like protoc before running or compiling code is a show stopper. There are even npm packages that will install protoc for you [1]. That means you can use it in a custom npm build script action, or invoke it before staring the dev server? Put in a git pre-commit to run it to assert there are no changes? Run via a file-watcher? IMO writing a schema in something like JSON for OpenAPI is way more verbose and time consuming than writing a proto file and a script or Makefile to invoke protoc. There also is the mismatch in capabilities. The JS-adjacent schema standards mostly do not directly support native types like uint64, let alone directly specify how to encode numbers like fixed64. I mean, look... the work around in JavaScript still is use "string" [2]. On the other hand, there is a standard in OpenAPI for validating values. But that doesn't make sense in the Protobuf world since usually the API is expected to be forward and backward compatible, within reason, so something like a min and max value needs to be done at runtime, and not fixed within the schema. And don't get me started on trying to define cross-field validation in a declarative schema. It is easier to just write the code. :-) So, IMO, no, a single standard won't happen in the industry. The many dimensions like logical vs physical models, validation, or error handling that make it impossible. [1] https://www.npmjs.com/package/@protobuf-ts/protoc https://www.npmjs.com/package/@protobuf-ts/protoc [2] https://github.com/OAI/OpenAPI-Specification/issues/2617#issuecomment-859375934 https://github.com/OAI/OpenAPI-Specification/issues/2617#iss...
- deleted 3y ago
  [deleted]
closeparen 3y ago
Protobuf has a JSON canonicalization [0]. All our gRPC servers automatically accept HTTP/JSON traffic with the procedure name specified in a header. That's what I use when I'm writing a script. https://protobuf.dev/programming-guides/proto3/#json https://protobuf.dev/programming-guides/proto3/#json
- dontlaugh 3y ago
  Twirp has this built-in https://twitchtv.github.io/twirp/docs/curl.html https://twitchtv.github.io/twirp/docs/curl.html It's very handy in some situations.
- arein3 3y ago
  Could you provide an example of json request to a grpc endpoint (url + headers + body)? Can it be done via http (ex. Postman) or it needs http2?
  - closeparen 3y ago
    You can check the Twirp doc in the sibling comment. Ours is similar. Works on regular HTTP, curl or Postman is fine.
quectophoton 3y ago
If only JSON-RPC[1] were more popular for JSON-based APIs. With JSON-RPC specifically, the trade-off is that it would not work well with binary data; but if that's a problem then instead of JSON it could be BSON, bencode, EBML, or whatever (at that point it wouldn't be JSON-RPC though), the point is that you can nest stuff without switching syntax. But compared to HTTP, if what you want is just JSON, then JSON-RPC saves you from having 5 different places to pass data, each with its own special syntax (one for method name; one for path; one for query params in the path; one for headers; one for body with the actual data). We only not notice this because of just how many libraries are doing the hard work for us and making it seem like HTTP requests are clean and nice stuff; but no, HTTP is certainly not a nice protocol. Its main advantage is its ubiquity. People may ask, "what about caching and stuff?". Well, with HTTP we made that work by passing a special key in the part of the request that is delimited by CRLF and where each line contains a key-value pair separated by a colon-and-space; and the clients know that a response was cached because they can read the first line of the response, which is a space-separated list of keywords that has a specific numeric literal in its second position (yes, I'm oversimplifying). So I'm pretty sure we can do something better with fewer different syntaxes. And reverse proxies would presumably have it harder to be vulnerable to things like request smuggling[2]. [1]: https://en.wikipedia.org/wiki/JSON-RPC https://en.wikipedia.org/wiki/JSON-RPC [2]: https://en.wikipedia.org/wiki/HTTP_request_smuggling https://en.wikipedia.org/wiki/HTTP_request_smuggling
BerislavLopac 3y ago
> pick a standard (openapi? jsonschema? something else?) This is a false dichotomy, as OpenAPI and JSON Schema are complementary: as of version 3.1, OpenAPI use JSON Schema to describe request and response payload content (prior to that version it was almost compatible). They describe different things: OpenAPI describes the request-response API model, while JSON Schema can be used to specify contents of any JSON-encoded data -- for example in streaming messages or document databases. There is also AsyncAPI, which is designed to document and specify the event-based API, for example in a system using Apache Kafka or a similar queue to communicate.

by theamk 3y ago

stubish 3y ago

I don't think a large group can pick a standard. What works well for one programming language does not work for another, and we devolve into tribes. Protobuf gained popularity when it dropped features to better support Go more naturally (no more null/nil for example), and rode in on a wave of Go adoption. And made things worse from a Python developer perspective. Pretty much the same as toml adoption, which fits more naturally with Go and more static languages, unlike the much more dynamic and flexible yaml.

Supermancho 3y ago
> I don't think a large group can pick a standard. There are plenty of cases where this has happened. JSON has already become the defacto standard. It's not a the step to formalize a ubiquitous format has been taken many times. Every major platform supports it or uses it exclusively. Some frameworks and tooling, only uses JSON (true of other formats to an extent, ie json more than yaml). > What works well for one programming language does not work for another, I'm not sure why this roundabout whataboutism is compelling. A standard protocol is used in absence of other restrictions or benefits of another This doesn't snap away the other protocols from existence.
arp242 3y ago
TOML doesn't "fit static languages" better and is just as dynamically typed as YAML. It's used extensively in the Python ecosystem, which has TOML in stdlib (not YAML). I'm less sure about protobufs in Python (I never tried), but in general relying too much on Python's dynamic nature is not a good thing IMHO.

hamandcheese 3y ago

GRPC <=> Rest transcoding works fine, you can do this with envoy for example. And if you use the connect protocol (also from buf) then you can also use json. So it's quite achievable to get all the benefits you want from the gRPC ecosystem for developing your service while also allowing quick and dirty client usage (i.e. scripting with cURL).

no_circuit 3y ago

I fail to see how running something like protoc before running or compiling code is a show stopper. There are even npm packages that will install protoc for you [1]. That means you can use it in a custom npm build script action, or invoke it before staring the dev server? Put in a git pre-commit to run it to assert there are no changes? Run via a file-watcher? IMO writing a schema in something like JSON for OpenAPI is way more verbose and time consuming than writing a proto file and a script or Makefile to invoke protoc. There also is the mismatch in capabilities. The JS-adjacent schema standards mostly do not directly support native types like uint64, let alone directly specify how to encode numbers like fixed64. I mean, look... the work around in JavaScript still is use "string" [2]. On the other hand, there is a standard in OpenAPI for validating values. But that doesn't make sense in the Protobuf world since usually the API is expected to be forward and backward compatible, within reason, so something like a min and max value needs to be done at runtime, and not fixed within the schema. And don't get me started on trying to define cross-field validation in a declarative schema. It is easier to just write the code. :-) So, IMO, no, a single standard won't happen in the industry. The many dimensions like logical vs physical models, validation, or error handling that make it impossible. [1] https://www.npmjs.com/package/@protobuf-ts/protoc https://www.npmjs.com/package/@protobuf-ts/protoc [2] https://github.com/OAI/OpenAPI-Specification/issues/2617#issuecomment-859375934 https://github.com/OAI/OpenAPI-Specification/issues/2617#iss...

deleted 3y ago
[deleted]

closeparen 3y ago

Protobuf has a JSON canonicalization [0]. All our gRPC servers automatically accept HTTP/JSON traffic with the procedure name specified in a header. That's what I use when I'm writing a script. https://protobuf.dev/programming-guides/proto3/#json https://protobuf.dev/programming-guides/proto3/#json

dontlaugh 3y ago
Twirp has this built-in https://twitchtv.github.io/twirp/docs/curl.html https://twitchtv.github.io/twirp/docs/curl.html It's very handy in some situations.
arein3 3y ago
Could you provide an example of json request to a grpc endpoint (url + headers + body)? Can it be done via http (ex. Postman) or it needs http2?
- closeparen 3y ago
  You can check the Twirp doc in the sibling comment. Ours is similar. Works on regular HTTP, curl or Postman is fine.

quectophoton 3y ago

If only JSON-RPC[1] were more popular for JSON-based APIs. With JSON-RPC specifically, the trade-off is that it would not work well with binary data; but if that's a problem then instead of JSON it could be BSON, bencode, EBML, or whatever (at that point it wouldn't be JSON-RPC though), the point is that you can nest stuff without switching syntax. But compared to HTTP, if what you want is just JSON, then JSON-RPC saves you from having 5 different places to pass data, each with its own special syntax (one for method name; one for path; one for query params in the path; one for headers; one for body with the actual data). We only not notice this because of just how many libraries are doing the hard work for us and making it seem like HTTP requests are clean and nice stuff; but no, HTTP is certainly not a nice protocol. Its main advantage is its ubiquity. People may ask, "what about caching and stuff?". Well, with HTTP we made that work by passing a special key in the part of the request that is delimited by CRLF and where each line contains a key-value pair separated by a colon-and-space; and the clients know that a response was cached because they can read the first line of the response, which is a space-separated list of keywords that has a specific numeric literal in its second position (yes, I'm oversimplifying). So I'm pretty sure we can do something better with fewer different syntaxes. And reverse proxies would presumably have it harder to be vulnerable to things like request smuggling[2]. [1]: https://en.wikipedia.org/wiki/JSON-RPC https://en.wikipedia.org/wiki/JSON-RPC [2]: https://en.wikipedia.org/wiki/HTTP_request_smuggling https://en.wikipedia.org/wiki/HTTP_request_smuggling

BerislavLopac 3y ago

> pick a standard (openapi? jsonschema? something else?) This is a false dichotomy, as OpenAPI and JSON Schema are complementary: as of version 3.1, OpenAPI use JSON Schema to describe request and response payload content (prior to that version it was almost compatible). They describe different things: OpenAPI describes the request-response API model, while JSON Schema can be used to specify contents of any JSON-encoded data -- for example in streaming messages or document databases. There is also AsyncAPI, which is designed to document and specify the event-based API, for example in a system using Apache Kafka or a similar queue to communicate.