6 ms·

Though I dislike the hyperbolic tone and personal attacks, the author isn't entirely wrong. There are many design choices in Protocol Buffers that seem directly

Though I dislike the hyperbolic tone and personal attacks, the author isn't entirely wrong. There are many design choices in Protocol Buffers that seem directly related to the scale and complexity at which Google operates, and which sacrifice safety, clarity and language integration. The utter awkwardness of Protobuf-generated code is particularly problematic. I've had pretty good results with the TypeScript code generator, but the Go generator is just egregiously bad. There's no way to write a client or server that uses the Protobuf structs directly as first-class data types without ending up looking like a ball of spaghetti, at least not if you venture into the realm of oneofs, timestamps or the "well known types" set of types. I think we're at a juncture where we really desperately need a better, unified way to express schemas and pass type-safe data over the network that flows through these schemas. I'm currently working on a project that involves both gRPC (which uses Protobuf), GraphQL and JSON Schema, with the backends written in Go, and the frontend in TypeScript, and the overlap between all of these is ridiculous. TypeScript is a fresh breath of air that mostly solves the brittleness of JavaScript's type system, and it's absolutely crucial to extend this type-safety all the way to the backend. The challenge is that no language is the same, and a common schema format ends up targeting a lowest-common denominator. For example, GraphQL errs (in my opinion, wrongly) on the side of simplicity and doesn't support maps, so now you have a whole layer that needs to either custom types (JSON as an escape hatch, basically) or emulate maps using arrays of pairs, neither of which is ideal.

nirelBRACIT0S 8y ago
Write your own generator then?
- hellllllllooo 8y ago
  People should be able to provide constructive and useful criticism of a technology without having to build their own solution.
enneff 8y ago
An approach I've used on the Go side is to build nice native Go types and converters to/from proto representations for wire marshaling. There's a minor performance hit but it's insignificant for what I'm doing. The result is quite nice, and you can largely ignore protobufs except for defining service APIs (which it's actually decent at). Which tool are you using for TypeScript protobuf code generation? This is the next step for me, and I'd be keen to hear more of your experience there.
- atombender 8y ago
  Yes, we use the same technique for our larger apps. We treat the Protobuf layer as a distinct layer related to the API; we have two packages, "protofy" and "unprotofy", so in the server implementation, you do something like: func (srv *Server) GetFoo( ctx context.Context, req *proto.GetFooRequest) (*proto.GetFooResponse, error) { foo, err := srv.db.getFoo(foo.ID) if err != nil { ... } pFoo, err := protofy.Foo(foo) if err != nil { ... } return &proto.GetFooResponse{ Foo: pFoo, }, nil } Obviously, a bit less pretty in reality. But the principle is the same. Having two packages makes it easier to read — e.g. protofy.Foo() is always about taking a "native" Foo and turning into a *proto.Foo, and unprotofy.Foo() is the reverse. For TypeScript, I'm using ts-protoc-gen [1]. The weird part, which I don't fully understand, is that the serialization code is emitted as JavaScript code. All the type definitions end up in a .d.ts file, but the client is a .js file. Which means you still get type safety and autocompletion, just like plain TS, but it's still weird to me. [1] https://github.com/improbable-eng/ts-protoc-gen https://github.com/improbable-eng/ts-protoc-gen
  - fixermark 8y ago
    I believe that means you're more or less using it as intended. Protobuffers are intended for serialization. In go, serialization is often handled at a separate layer from the business logic.
  - traviscline 8y ago
    You might consider https://godoc.org/github.com/tmc/grpcutil/protoc-gen-tstypes https://godoc.org/github.com/tmc/grpcutil/protoc-gen-tstypes
hardwaresofton 8y ago
I'd like to humbly suggest that we use JSON please, in particular: JSON + JSONSchema[0] +/- JSON Hyperschema[1] +/- JSON LD[2] It's a bit to learn but I promise you, it's worth it. The technologies are not redundant (jsonschema spec is for validation, hyperschema spec is for specifying how you interact, and LD is for semantics like language and more). If you take a few hours, read all 3 specs, you're almost guaranteed to imagine a new (albeit somewhat daunting) future, where APIs and clients can be smarter. There's no reason I can think of that a sufficiently specified JSON schema cannot be automatically converted into a efficient binary format. I have absolutely zero reason to use Protobuf and it's whole ecosystem of tools if that's the case, I'd much rather go with tools built on a standard like JSON. GraphQL's functionality is a subset of the above set of tools -- IIRC GraphQL can basically be boiled down to an option (or two) on a resource endpoint -- How you query on the server side does indeed change a bit to support the syntax, but this is basically the same as if you'd implemented with horizontal filtering like in POSTgrest[3], or resource embedding feature[4] (which everyone ends up doing at some point with REST-ish APIs). I honestly don't know who or what is trying so hard to drive GraphQL adoption so hard but I think it's mostly hype, and in terms of general technology, by default it forces clients to know the inner workings of the data model and that's bad for any kind of future in which you don't want to have to learn X APIs to interact with X systems. BTW I absolutely love a good well-supported rant that agrees with all my biases (yes, Google as well as every other company has a bunch of amateurs, as well as a small amount of very skilled engineers), and all the problems noted with protobuf are real problems, and this viewpoint is sorely missed in all the talks and conferences where people are touting GRPC as the next sliced bread with Protobufs as the transport (you can change GRPC's transport layer to be JSON for example[5]) [0]: https://json-schema.org/ https://json-schema.org/ [1]: https://datatracker.ietf.org/doc/draft-handrews-json-schema-hyperschema/ https://datatracker.ietf.org/doc/draft-handrews-json-schema-... (also on json-schema.org) [2]: https://json-ld.org/ https://json-ld.org/ [3]: https://postgrest.org/en/v5.1/api.html#horizontal-filtering-rows https://postgrest.org/en/v5.1/api.html#horizontal-filtering-... [4]: https://postgrest.org/en/v5.1/api.html#resource-embedding https://postgrest.org/en/v5.1/api.html#resource-embedding [5]: https://grpc.io/blog/grpc-with-json https://grpc.io/blog/grpc-with-json
- Fellshard 8y ago
the_clarence 8y ago
Yup. No stream API to unmarshall protobuffers suck majorly. You have to re-invent framing
- romed 8y ago
  Which language are you talking about? C++ proto is built around CodedInputStream which is what it sounds like. You can use it with or without the generated message code. https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.io.coded_stream https://developers.google.com/protocol-buffers/docs/referenc...
  - the_clarence 8y ago
    Golang
mikelward 8y ago
Could you elaborate on what you don't like about the generated Go code? My main complaint initially was that everything was a pointer, which made constructing things harder. Then I realized it made zero values much easier to implement. And it lets you foo.GetA().GetB().GetC() without all the intermediate steps, which feels like a violation of the Law of Demeter, but in a glue language I think it's probably the right trade-off. Yes, you can't use them as classes/types themselves, but you're not supposed to subclass them in any language, so they're really better thought of as structs/dicts.
- shabbyrobe 8y ago
  Oneofs, at least with gogoproto, are quite unpleasant to work with. There's a separate intermediate type generated for each member of a oneof field. With the following message: message Foo { oneof bar { Baz baz = 1; Qux qux = 2; } } You'll have the types Foo, Baz and Qux like you'd expect, but you also get Foo_Baz, which does nothing but wrap the Baz for use with the Foo.bar property, and Foo_Qux, which does the same thing but for Qux instead of Baz. To make matters worse, the types all implement an interface which is declared but not exported. Stringing the whole thing together to assign to a oneof looks like this: foo := &rpc.Foo{Bar: &rpc.Foo_Baz{Baz: &rpc.Baz{}}} That example makes it look better than it really is; in practice, you'll probably have more properties and longer names.

nirelBRACIT0S 8y ago

Write your own generator then?

hellllllllooo 8y ago
People should be able to provide constructive and useful criticism of a technology without having to build their own solution.

enneff 8y ago

An approach I've used on the Go side is to build nice native Go types and converters to/from proto representations for wire marshaling. There's a minor performance hit but it's insignificant for what I'm doing. The result is quite nice, and you can largely ignore protobufs except for defining service APIs (which it's actually decent at). Which tool are you using for TypeScript protobuf code generation? This is the next step for me, and I'd be keen to hear more of your experience there.

atombender 8y ago
Yes, we use the same technique for our larger apps. We treat the Protobuf layer as a distinct layer related to the API; we have two packages, "protofy" and "unprotofy", so in the server implementation, you do something like: func (srv *Server) GetFoo( ctx context.Context, req *proto.GetFooRequest) (*proto.GetFooResponse, error) { foo, err := srv.db.getFoo(foo.ID) if err != nil { ... } pFoo, err := protofy.Foo(foo) if err != nil { ... } return &proto.GetFooResponse{ Foo: pFoo, }, nil } Obviously, a bit less pretty in reality. But the principle is the same. Having two packages makes it easier to read — e.g. protofy.Foo() is always about taking a "native" Foo and turning into a *proto.Foo, and unprotofy.Foo() is the reverse. For TypeScript, I'm using ts-protoc-gen [1]. The weird part, which I don't fully understand, is that the serialization code is emitted as JavaScript code. All the type definitions end up in a .d.ts file, but the client is a .js file. Which means you still get type safety and autocompletion, just like plain TS, but it's still weird to me. [1] https://github.com/improbable-eng/ts-protoc-gen https://github.com/improbable-eng/ts-protoc-gen
- fixermark 8y ago
  I believe that means you're more or less using it as intended. Protobuffers are intended for serialization. In go, serialization is often handled at a separate layer from the business logic.
- traviscline 8y ago
  You might consider https://godoc.org/github.com/tmc/grpcutil/protoc-gen-tstypes https://godoc.org/github.com/tmc/grpcutil/protoc-gen-tstypes

hardwaresofton 8y ago

I'd like to humbly suggest that we use JSON please, in particular: JSON + JSONSchema[0] +/- JSON Hyperschema[1] +/- JSON LD[2] It's a bit to learn but I promise you, it's worth it. The technologies are not redundant (jsonschema spec is for validation, hyperschema spec is for specifying how you interact, and LD is for semantics like language and more). If you take a few hours, read all 3 specs, you're almost guaranteed to imagine a new (albeit somewhat daunting) future, where APIs and clients can be smarter. There's no reason I can think of that a sufficiently specified JSON schema cannot be automatically converted into a efficient binary format. I have absolutely zero reason to use Protobuf and it's whole ecosystem of tools if that's the case, I'd much rather go with tools built on a standard like JSON. GraphQL's functionality is a subset of the above set of tools -- IIRC GraphQL can basically be boiled down to an option (or two) on a resource endpoint -- How you query on the server side does indeed change a bit to support the syntax, but this is basically the same as if you'd implemented with horizontal filtering like in POSTgrest[3], or resource embedding feature[4] (which everyone ends up doing at some point with REST-ish APIs). I honestly don't know who or what is trying so hard to drive GraphQL adoption so hard but I think it's mostly hype, and in terms of general technology, by default it forces clients to know the inner workings of the data model and that's bad for any kind of future in which you don't want to have to learn X APIs to interact with X systems. BTW I absolutely love a good well-supported rant that agrees with all my biases (yes, Google as well as every other company has a bunch of amateurs, as well as a small amount of very skilled engineers), and all the problems noted with protobuf are real problems, and this viewpoint is sorely missed in all the talks and conferences where people are touting GRPC as the next sliced bread with Protobufs as the transport (you can change GRPC's transport layer to be JSON for example[5]) [0]: https://json-schema.org/ https://json-schema.org/ [1]: https://datatracker.ietf.org/doc/draft-handrews-json-schema-hyperschema/ https://datatracker.ietf.org/doc/draft-handrews-json-schema-... (also on json-schema.org) [2]: https://json-ld.org/ https://json-ld.org/ [3]: https://postgrest.org/en/v5.1/api.html#horizontal-filtering-rows https://postgrest.org/en/v5.1/api.html#horizontal-filtering-... [4]: https://postgrest.org/en/v5.1/api.html#resource-embedding https://postgrest.org/en/v5.1/api.html#resource-embedding [5]: https://grpc.io/blog/grpc-with-json https://grpc.io/blog/grpc-with-json

Fellshard 8y ago

the_clarence 8y ago

Yup. No stream API to unmarshall protobuffers suck majorly. You have to re-invent framing

romed 8y ago
Which language are you talking about? C++ proto is built around CodedInputStream which is what it sounds like. You can use it with or without the generated message code. https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.io.coded_stream https://developers.google.com/protocol-buffers/docs/referenc...
- the_clarence 8y ago
  Golang

mikelward 8y ago

Could you elaborate on what you don't like about the generated Go code? My main complaint initially was that everything was a pointer, which made constructing things harder. Then I realized it made zero values much easier to implement. And it lets you foo.GetA().GetB().GetC() without all the intermediate steps, which feels like a violation of the Law of Demeter, but in a glue language I think it's probably the right trade-off. Yes, you can't use them as classes/types themselves, but you're not supposed to subclass them in any language, so they're really better thought of as structs/dicts.

shabbyrobe 8y ago
Oneofs, at least with gogoproto, are quite unpleasant to work with. There's a separate intermediate type generated for each member of a oneof field. With the following message: message Foo { oneof bar { Baz baz = 1; Qux qux = 2; } } You'll have the types Foo, Baz and Qux like you'd expect, but you also get Foo_Baz, which does nothing but wrap the Baz for use with the Foo.bar property, and Foo_Qux, which does the same thing but for Qux instead of Baz. To make matters worse, the types all implement an interface which is declared but not exported. Stringing the whole thing together to assign to a oneof looks like this: foo := &rpc.Foo{Bar: &rpc.Foo_Baz{Baz: &rpc.Baz{}}} That example makes it look better than it really is; in practice, you'll probably have more properties and longer names.