3 ms·
> every application has to be updated when a field is added, even if they do not use that field This is when, in a protocol, you reach for the hammer called "e
by derefr 8y ago
> every application has to be updated when a field is added, even if they do not use that field
This is when, in a protocol, you reach for the hammer called "extensibility."
In this case (decoding to native structs), you'd probably have your FooMessage product-type have an "extensions" field, which is a list of zero or more FooExtensionStructs, where a FooExtensionStruct is a sum type of the known extensions to FooMessage.
Then, just make the semantics of your format such that a sum type can have a pragma in its IDL indicating that it isn't required to be decodable for the product-type containing it to be successfully decoded. I.e., the sum type becomes a (DecodeResult = Decoded(sum type) | Opaque(raw wire data)).
Opaque (unrecognized) extension data can't be manipulated by the program, but it can losslessly survive a trip back through the wire encoder. So you can choose to pass it through in your middleware, or not.
Boom: you've reinvented the "chunks" concept of the https://en.wikipedia.org/wiki/Interchange_File_Format https://en.wikipedia.org/wiki/Interchange_File_Format, as seen in PNG and ELF.
- fixermark 8y agoI believe you've also reinvented optional fields in a more generalizable way. Generalizability can be good or bad, depending on how much complexity it adds.
- derefr 8y agoNope—there's a very important difference in the two approaches. You have to choose to add the kind of extensibility I was describing to a particular type, in advance, as part of that type's original specification. You have no choice in having optional fields. With Protobuf optional fields, even if you formally specify a protocol where some particular message is absolutely "final" and will never be extended, anyone can just throw an optional member on there when sending one to you, and your implementation won't reject the struct. This is a big problem if you're trying to specify types that really map one-to-one to fixed-size structs you'll use in the business logic of your non-memory-managed language. With explicit extension points, you get to dictate where and when extension happens; and, as well, you get to formalize exactly what extensions look like (i.e. you don't have to allow for un-recognized extensions; you can simply have extensions be a finite set of known new extensions, where everything else is a real decode error.)
- fixermark 8y ago... however, it's a great feature if the use case is "protocol for client-server network communication," which was the design goal for protobuffers. It follows the "permissive in what you accept, strict in what you emit" design philosophy.