4 ms·

Genuine question for someone who's used Protobufs in production. How do you avoid the sort of schema violation problems that CORBA and XML had back in their day

Genuine question for someone who's used Protobufs in production. How do you avoid the sort of schema violation problems that CORBA and XML had back in their day? Are the schemas in each request?

cgdub 4y ago
We have rules around changing the schema. You can only add new fields and deprecate old fields. You can't rename or reuse fields. All new fields must be optional to make schema evolution possible. In practice, this is pretty similar to how you maintain compatibility with old clients in JSON. But with protobuf, you have the schema file to make code reviewing this kind of change easier.
tommiegannert 4y ago
I don't know what schema violations you refer to, but XML has the issue that everything is a string until it isn't. The wire encoding of Protobuf has more type information, so I'd say it's half-including the schema compared to XML. Similar to ASN.1 binary encodings, but while ASN.1 (DER, CER) allows you to override the actual wire type, Protobuf always says an integer is an integer on the wire. That means you can do way more parsing and validation in library code without knowing the schema. What's not in there is protection against violations such as sending multiple values for non-repeated fields (the latest value wins) or accidentally making non-backwards-compatible changes to the schema, e.g. reusing field numbers or changing types. The internal Google styleguide for Protobuf is a valuable resource for the collective avoidance of foot-firearms, and it's a bit sad it doesn't seem to be published externally. I don't know of any existing linter that could check for backwards compatibility , but I am interested in one. Might break out one I wrote to a separate library. There aren't too many situations that commonly causes breakages. One reason may be that developer names (text identifiers) are separate from on-wire/persistent names (integer tags). This leaves some wiggle-room for improving your development environment by incrementally clarifying names without breaking backwards-compatibility with already compiled code.
- endtime 4y ago
  > I don't know of any existing linter that could check for backwards compatibility , but I am interested in one. Buf, from the OP, does this with `buf breaking`.
AtNightWeCode 4y ago
Predefined. Looking forwards to all the lies in this thread... Theoretically, a protobuf schema allows addition and removals. In practice. schema-nazist will destroy your life and change whatever cause the only little thing that separates them from the peasants is that royal like power of making breaking schema changes in prod and pinning that incident on you. Your code did not even use that data probably.

Genuine question for someone who's used Protobufs in production. How do you avoid the sort of schema violation problems that CORBA and XML had back in their day? Are the schemas in each request?

cgdub 4y ago

We have rules around changing the schema. You can only add new fields and deprecate old fields. You can't rename or reuse fields. All new fields must be optional to make schema evolution possible. In practice, this is pretty similar to how you maintain compatibility with old clients in JSON. But with protobuf, you have the schema file to make code reviewing this kind of change easier.

tommiegannert 4y ago

I don't know what schema violations you refer to, but XML has the issue that everything is a string until it isn't. The wire encoding of Protobuf has more type information, so I'd say it's half-including the schema compared to XML. Similar to ASN.1 binary encodings, but while ASN.1 (DER, CER) allows you to override the actual wire type, Protobuf always says an integer is an integer on the wire. That means you can do way more parsing and validation in library code without knowing the schema. What's not in there is protection against violations such as sending multiple values for non-repeated fields (the latest value wins) or accidentally making non-backwards-compatible changes to the schema, e.g. reusing field numbers or changing types. The internal Google styleguide for Protobuf is a valuable resource for the collective avoidance of foot-firearms, and it's a bit sad it doesn't seem to be published externally. I don't know of any existing linter that could check for backwards compatibility , but I am interested in one. Might break out one I wrote to a separate library. There aren't too many situations that commonly causes breakages. One reason may be that developer names (text identifiers) are separate from on-wire/persistent names (integer tags). This leaves some wiggle-room for improving your development environment by incrementally clarifying names without breaking backwards-compatibility with already compiled code.

endtime 4y ago
> I don't know of any existing linter that could check for backwards compatibility , but I am interested in one. Buf, from the OP, does this with `buf breaking`.

AtNightWeCode 4y ago

Predefined. Looking forwards to all the lies in this thread... Theoretically, a protobuf schema allows addition and removals. In practice. schema-nazist will destroy your life and change whatever cause the only little thing that separates them from the peasants is that royal like power of making breaking schema changes in prod and pinning that incident on you. Your code did not even use that data probably.