3 ms·

Never having had the opportunity to put protobuf into action but having some interest I've had these questions: 1) What would you say is the best use case 2)

Never having had the opportunity to put protobuf into action but having some interest I've had these questions: 1) What would you say is the best use case 2) and what unfortunate misusecases have you come across

jhumphries131 4y ago
The key "killer" use case is for describing RPC schemas. By describing the schema in an IDL, you can then generate client stubs and server interfaces in a variety of implementation languages, allowing interop between heterogenous clients and servers. They are also useful for describing domain models. This isn't surprising since domain models usually find their way into RPC schemas (since RPCs will often query or define model data). But they can also be used in other cases, such as for persistence and structured logging. Some misuse I have seen involves trying to make a protobuf model the _only_ representation of a domain model: it is almost inevitable that a physical model (a representation of your data in a SQL database, for example) will need to vary from a logical model, and trying to make a single protobuf representation serve double duty can be a source of problems. Making protobuf schemas conform to physical storage constraints often makes for worse abstraction. The model becomes de-normalized (which can make constraints and relationships harder to model/enforce), and can even leak details that consumers shouldn't know about or care about. Another misuse is using it for data that never leaves a process -- a program's private, internal state. If a data structure never needs to be serialized (to persist or send to another process), then you're better off using native data structures in the implementation language (which generally have far greater flexibility/expressibility as well as better performance).
shaftway 4y ago
I do most of my work with protos in Java, and it's nice to have a schema like this that will build a bunch of immutable POJOs with builder classes and enough infrastructure to be able to do some interesting reflection-style stuff on top of the serialization / deserialization. The wide variety of client languages is really nice. I'm fairly certain that I can parse a proto in any language I'm ever going to use. The binary wire format is fairly straightforward, and is pretty tight without using compression. Fields are byte-aligned, and if you wanted to generate a binary proto message by concatenating a few things together it isn't very hard. And then you can use your proto definitions in whatever language you want to parse it. You can even parse your proto definition into a proto (Google provides the proto proto definitions, I think I got that grammar right) and write tools that generate whatever code you want easily. I think the text format is under-utilized. It's my go-to for configuration files. You create the proto definitions with whatever structures you want to structure your config settings, and then use the official parser to parse a text file. It supports comments (I'm throwing shade at you, JSON), and is simpler than YAML, while adding that structure. You can also use command line tools to validate the file as a pre-commit, or translate it into a binary format if you don't want to rely on the text format.
somenameforme 4y ago
1) Serialization for storage or transit 2) Serialization for storage or transit Unless the spec has changed, there's a fun issue. In protobuf there's no notion of null. So for instance a null list and an empty list are serialized identically. This leads to a nightmare in any system where null can have meaning by itself. I was always confused by this decision since there would be an endless number of ways to represent null that would have near zero impact on size or performance. The only cost I can see is the abstract one of making the representation less "elegant." There are a of course endless ways to overcome this issue, but I don't think serialization implementation details should ever be allowed to leak into the model of what's being serialized. Just begging for issues down the road.
- joshuamorton 4y ago
  This is true for proto3, but not for proto2 (except for repeated fields, which are always non-null but can contain 0 items). Proto2 has `HasFoo` methods, which distinguish between empty (unset/null) and set explicitly to zero fields. I don't like that proto3 made this choice, I think it was wrong, but there was some rationale behind it beyond just "elegance".
  - somenameforme 4y ago
    Can you elaborate on the reasons for the decision? I've always been quite curious on the reasoning there.

jhumphries131 4y ago

The key "killer" use case is for describing RPC schemas. By describing the schema in an IDL, you can then generate client stubs and server interfaces in a variety of implementation languages, allowing interop between heterogenous clients and servers. They are also useful for describing domain models. This isn't surprising since domain models usually find their way into RPC schemas (since RPCs will often query or define model data). But they can also be used in other cases, such as for persistence and structured logging. Some misuse I have seen involves trying to make a protobuf model the _only_ representation of a domain model: it is almost inevitable that a physical model (a representation of your data in a SQL database, for example) will need to vary from a logical model, and trying to make a single protobuf representation serve double duty can be a source of problems. Making protobuf schemas conform to physical storage constraints often makes for worse abstraction. The model becomes de-normalized (which can make constraints and relationships harder to model/enforce), and can even leak details that consumers shouldn't know about or care about. Another misuse is using it for data that never leaves a process -- a program's private, internal state. If a data structure never needs to be serialized (to persist or send to another process), then you're better off using native data structures in the implementation language (which generally have far greater flexibility/expressibility as well as better performance).

shaftway 4y ago

I do most of my work with protos in Java, and it's nice to have a schema like this that will build a bunch of immutable POJOs with builder classes and enough infrastructure to be able to do some interesting reflection-style stuff on top of the serialization / deserialization. The wide variety of client languages is really nice. I'm fairly certain that I can parse a proto in any language I'm ever going to use. The binary wire format is fairly straightforward, and is pretty tight without using compression. Fields are byte-aligned, and if you wanted to generate a binary proto message by concatenating a few things together it isn't very hard. And then you can use your proto definitions in whatever language you want to parse it. You can even parse your proto definition into a proto (Google provides the proto proto definitions, I think I got that grammar right) and write tools that generate whatever code you want easily. I think the text format is under-utilized. It's my go-to for configuration files. You create the proto definitions with whatever structures you want to structure your config settings, and then use the official parser to parse a text file. It supports comments (I'm throwing shade at you, JSON), and is simpler than YAML, while adding that structure. You can also use command line tools to validate the file as a pre-commit, or translate it into a binary format if you don't want to rely on the text format.

somenameforme 4y ago

1) Serialization for storage or transit 2) Serialization for storage or transit Unless the spec has changed, there's a fun issue. In protobuf there's no notion of null. So for instance a null list and an empty list are serialized identically. This leads to a nightmare in any system where null can have meaning by itself. I was always confused by this decision since there would be an endless number of ways to represent null that would have near zero impact on size or performance. The only cost I can see is the abstract one of making the representation less "elegant." There are a of course endless ways to overcome this issue, but I don't think serialization implementation details should ever be allowed to leak into the model of what's being serialized. Just begging for issues down the road.

joshuamorton 4y ago
This is true for proto3, but not for proto2 (except for repeated fields, which are always non-null but can contain 0 items). Proto2 has `HasFoo` methods, which distinguish between empty (unset/null) and set explicitly to zero fields. I don't like that proto3 made this choice, I think it was wrong, but there was some rationale behind it beyond just "elegance".
- somenameforme 4y ago
  Can you elaborate on the reasons for the decision? I've always been quite curious on the reasoning there.