3 ms·

It seems like protobufs are sort of a mixed interior / exterior model? In an organization that uses centralized protobuf schemas and generates serialization cod

It seems like protobufs are sort of a mixed interior / exterior model? In an organization that uses centralized protobuf schemas and generates serialization code from that, you can assume every protobuf record of that type came from a serializer that obeys some version of the schema. Why can this guarantee a field's type but not whether it's required? Because fields have a lifecycle. They get added to the schema, used for a while, and later become obsolete and are deleted. The maintainers of a protobuf schema can guarantee that a field number is always associated with a certain type, and that when the field is deleted, that field number is never reused. This makes (field number, type) tuples timeless. But they can't say whether a field will be needed forever, and they can't control which versions of the schema are still being used. Effectively, "required" means "possibly required forever." As long as there is some parser out there that treats the field as required, deserialization will break if you leave out the field. Changing a field's type is easier, because you don't really do it. You add a new field number with a new type and maybe eventually stop emitting the old field. This suggests a strategy: fields can be required as long as they're not top-level. You can define coordinates(42) to always be (x,y) where both x and y are required. If you change it to be a 3D coordinate, then that will be coordinates(43) = (x,y,z), and you never reuse field number 42 in that schema.

kccqzy 3y ago
The strategy illustrated at the end still ignores the learning of the FAQ. What if, let's say, one microservice is responsible for calculating the x coordinate but another microservice is responsible for calculating the y coordinate? The former microservice could rightly fill in only the x coordinate and ask the latter microservice to fill in the y coordinate. Then you start to make compromises such as defining an IncompleteCoordinate message that requires x. But what if later you calculate y first and use that to calculate x? Etc. Etc. I find it more convenient to make everything optional at the wire format level and explicitly validate what is required at the point of use, not at the point of parse.
- skybrian 3y ago
  It seems like an edge case and you could define another message altogether for the microservice traffic. But yeah, for larger child messages, copying fields would be annoying. Often you want to be able to create child messages without creating the parent.

kccqzy 3y ago

The strategy illustrated at the end still ignores the learning of the FAQ. What if, let's say, one microservice is responsible for calculating the x coordinate but another microservice is responsible for calculating the y coordinate? The former microservice could rightly fill in only the x coordinate and ask the latter microservice to fill in the y coordinate. Then you start to make compromises such as defining an IncompleteCoordinate message that requires x. But what if later you calculate y first and use that to calculate x? Etc. Etc. I find it more convenient to make everything optional at the wire format level and explicitly validate what is required at the point of use, not at the point of parse.

skybrian 3y ago
It seems like an edge case and you could define another message altogether for the microservice traffic. But yeah, for larger child messages, copying fields would be annoying. Often you want to be able to create child messages without creating the parent.