5 ms·

Put me firmly in the camp of "optional fields are bad." I believe all fields should be required. I come from an ONC/RPC background, which is the original UNIX

Put me firmly in the camp of "optional fields are bad." I believe all fields should be required. I come from an ONC/RPC background, which is the original UNIX RPC. Every iteration of an rpc would get versioned, and then you could write a conversion between versions, ex from V1 to V2, from V2 to V3, etc. This allowed for true backwards compatibility. The idea of "forwards compatibility" is a pipedream, in my opinion. Just because something won't crash because a field is or isn't there doesn't mean that it will actually work. Code ages over time, and instead of looking at the spec, which will tell you which fields are required, you need to read the documentation, which may or may not tell you what is required. And especially when dealing with an older client, it simply may not work because what used to be optional-optional fields become required-optional fields. It's the same argument as schemaless NOSQL database like Mongo vs MySQL. Having a schema is a pain in the ass, but it's better over time and ages much more gracefully than schemaless ones, because things won't just break.

jeremyjh 8y ago
If all fields are required, you cannot have middleware that processes multiple versions of the same protobuf, and every application has to be updated when a field is added, even if they do not use that field. This is one of the more important design goals underlying not only protobufs, but most of the non-language specific binary formatters.
- derefr 8y ago
  > every application has to be updated when a field is added, even if they do not use that field This is when, in a protocol, you reach for the hammer called "extensibility." In this case (decoding to native structs), you'd probably have your FooMessage product-type have an "extensions" field, which is a list of zero or more FooExtensionStructs, where a FooExtensionStruct is a sum type of the known extensions to FooMessage. Then, just make the semantics of your format such that a sum type can have a pragma in its IDL indicating that it isn't required to be decodable for the product-type containing it to be successfully decoded. I.e., the sum type becomes a (DecodeResult = Decoded(sum type) | Opaque(raw wire data)). Opaque (unrecognized) extension data can't be manipulated by the program, but it can losslessly survive a trip back through the wire encoder. So you can choose to pass it through in your middleware, or not. Boom: you've reinvented the "chunks" concept of the https://en.wikipedia.org/wiki/Interchange_File_Format https://en.wikipedia.org/wiki/Interchange_File_Format, as seen in PNG and ELF.
  - fixermark 8y ago
    I believe you've also reinvented optional fields in a more generalizable way. Generalizability can be good or bad, depending on how much complexity it adds.
    - derefr 8y ago
      Nope—there's a very important difference in the two approaches. You have to choose to add the kind of extensibility I was describing to a particular type, in advance, as part of that type's original specification. You have no choice in having optional fields. With Protobuf optional fields, even if you formally specify a protocol where some particular message is absolutely "final" and will never be extended, anyone can just throw an optional member on there when sending one to you, and your implementation won't reject the struct. This is a big problem if you're trying to specify types that really map one-to-one to fixed-size structs you'll use in the business logic of your non-memory-managed language. With explicit extension points, you get to dictate where and when extension happens; and, as well, you get to formalize exactly what extensions look like (i.e. you don't have to allow for un-recognized extensions; you can simply have extensions be a finite set of known new extensions, where everything else is a real decode error.)
      - fixermark 8y ago
        ... however, it's a great feature if the use case is "protocol for client-server network communication," which was the design goal for protobuffers. It follows the "permissive in what you accept, strict in what you emit" design philosophy.
- docker_up 8y ago
  > every application has to be updated when a field is added, even if they do not use that field No, you maintain the older versions of the API. V1 of the API uses the V1 struct. V2 of the API uses the V2 struct, etc. Older applications maintain compatibility because it calls the older APIs, and you can convert between V1 to V2 and only keep one version of the API. Or, if you want, you can maintain both versions of the API, V1 and V2, at the expense of maintenance costs. But it's absolutely doable (we did it for decades).
  - taeric 8y ago
    You are missing when you have a middle layer. Message comes in at v3 and hits a layer that only knows v1 then gets passed to a layer that is at v4. I'd wager most places don't have that many layers. But, if you are embracing microservices, you'll find yourself here fairly fast.
    - docker_up 8y ago
      I doubt any api call trying to work in such a chaotic environment would actually work, and having all optional fields won't magically make things work. It will probably fail but in very mysterious ways. This sounds more like an environment where microservices are completely out of control and chaotic.
      - vkou 8y ago
        Doubt all you will, but being able to update different pieces of your service independently is a huge win. Having to update everything in lockstep is what I'd describe as 'out of control'. Which is precisely what forcing every field to be required does. There's a million and one reasons for why you may want to push, or rollback only 1 out of X services. If you follow a few simple rules when adding/removing fields, you can do this safely.
        deleted 8y ago
        [deleted]
        8y ago
mjevans 8y ago
Where something like JSON or (if you must) XML is used I prefer the idea of REQUIRING the preservation of all fields in the original structure UNLESS a field or set of fields is validated and then updated in place. This lets end users and extension authors do things that make sense, such as adding a comment tag ( JSON added key:value "__customcomment": "Any well formed string will be safe here." or XML  ) to a configuration file stored in either format, and actually having it persist.
dweis 8y ago
How do you handle binary rollbacks and rollouts safely with an "everything is required" approach? Do you force binaries to roll out in a strict order with appropriate soak time at each layer? How does that affect developer velocity?
- deleted 8y ago
  [deleted]
- docker_up 8y ago
  No, you create conversion routines that convert between different versions of structs. This keeps things well understood with no ambiguity. This is very easy, we were doing this 20 years ago and autogenerating the conversions using ANTLR that parsed the XDR files for ONC/RPC.
  - joshuamorton 8y ago
    This doesn't make sense with the concept of rollbacks. If I rollout server version 2 and client version 2 which each use a new required field, and then realize that there is some terrible error in server version 2, I can't roll it back to version 1, since it will reject all client calls from v2 clients. The only way to make it work is to add a translation layer, as you suggest, on the server, wait a while, push the new 'required' client, wait a while longer, and then push the server without the translation layer. That's the "strict ordering with appropriate soak time" GP mentions.
    - docker_up 8y ago
      Instead, you're going to get errors from the clients using version 2, because server version 2 was rolled back. You have to roll back the clients as well then. Or you could have client version 2 know how to automatically convert to server version 1, because you're know what version the server is on, and you can convert your client parameters or even behavior to fit version 1. You can't do this with protobufs because there is no such concept, you just add optional fields, and ignore them with different versions, and it's chaos.
AndrewDucker 8y ago
Middle Name. Address Line 2. Date Of Death.
fixermark 8y ago
If all fields are required, how do you add a new required field without breaking all of the clients that were using the previous protobuffer version?

jeremyjh 8y ago

If all fields are required, you cannot have middleware that processes multiple versions of the same protobuf, and every application has to be updated when a field is added, even if they do not use that field. This is one of the more important design goals underlying not only protobufs, but most of the non-language specific binary formatters.

derefr 8y ago
> every application has to be updated when a field is added, even if they do not use that field This is when, in a protocol, you reach for the hammer called "extensibility." In this case (decoding to native structs), you'd probably have your FooMessage product-type have an "extensions" field, which is a list of zero or more FooExtensionStructs, where a FooExtensionStruct is a sum type of the known extensions to FooMessage. Then, just make the semantics of your format such that a sum type can have a pragma in its IDL indicating that it isn't required to be decodable for the product-type containing it to be successfully decoded. I.e., the sum type becomes a (DecodeResult = Decoded(sum type) | Opaque(raw wire data)). Opaque (unrecognized) extension data can't be manipulated by the program, but it can losslessly survive a trip back through the wire encoder. So you can choose to pass it through in your middleware, or not. Boom: you've reinvented the "chunks" concept of the https://en.wikipedia.org/wiki/Interchange_File_Format https://en.wikipedia.org/wiki/Interchange_File_Format, as seen in PNG and ELF.
- fixermark 8y ago
  I believe you've also reinvented optional fields in a more generalizable way. Generalizability can be good or bad, depending on how much complexity it adds.
  - derefr 8y ago
    Nope—there's a very important difference in the two approaches. You have to choose to add the kind of extensibility I was describing to a particular type, in advance, as part of that type's original specification. You have no choice in having optional fields. With Protobuf optional fields, even if you formally specify a protocol where some particular message is absolutely "final" and will never be extended, anyone can just throw an optional member on there when sending one to you, and your implementation won't reject the struct. This is a big problem if you're trying to specify types that really map one-to-one to fixed-size structs you'll use in the business logic of your non-memory-managed language. With explicit extension points, you get to dictate where and when extension happens; and, as well, you get to formalize exactly what extensions look like (i.e. you don't have to allow for un-recognized extensions; you can simply have extensions be a finite set of known new extensions, where everything else is a real decode error.)
    - fixermark 8y ago
      ... however, it's a great feature if the use case is "protocol for client-server network communication," which was the design goal for protobuffers. It follows the "permissive in what you accept, strict in what you emit" design philosophy.
docker_up 8y ago
> every application has to be updated when a field is added, even if they do not use that field No, you maintain the older versions of the API. V1 of the API uses the V1 struct. V2 of the API uses the V2 struct, etc. Older applications maintain compatibility because it calls the older APIs, and you can convert between V1 to V2 and only keep one version of the API. Or, if you want, you can maintain both versions of the API, V1 and V2, at the expense of maintenance costs. But it's absolutely doable (we did it for decades).
- taeric 8y ago
  You are missing when you have a middle layer. Message comes in at v3 and hits a layer that only knows v1 then gets passed to a layer that is at v4. I'd wager most places don't have that many layers. But, if you are embracing microservices, you'll find yourself here fairly fast.
  - docker_up 8y ago
    I doubt any api call trying to work in such a chaotic environment would actually work, and having all optional fields won't magically make things work. It will probably fail but in very mysterious ways. This sounds more like an environment where microservices are completely out of control and chaotic.
    - vkou 8y ago
      Doubt all you will, but being able to update different pieces of your service independently is a huge win. Having to update everything in lockstep is what I'd describe as 'out of control'. Which is precisely what forcing every field to be required does. There's a million and one reasons for why you may want to push, or rollback only 1 out of X services. If you follow a few simple rules when adding/removing fields, you can do this safely.
      - deleted 8y ago
        [deleted]
      - 8y ago

mjevans 8y ago

Where something like JSON or (if you must) XML is used I prefer the idea of REQUIRING the preservation of all fields in the original structure UNLESS a field or set of fields is validated and then updated in place. This lets end users and extension authors do things that make sense, such as adding a comment tag ( JSON added key:value "__customcomment": "Any well formed string will be safe here." or XML  ) to a configuration file stored in either format, and actually having it persist.

dweis 8y ago

How do you handle binary rollbacks and rollouts safely with an "everything is required" approach? Do you force binaries to roll out in a strict order with appropriate soak time at each layer? How does that affect developer velocity?

deleted 8y ago
[deleted]
docker_up 8y ago
No, you create conversion routines that convert between different versions of structs. This keeps things well understood with no ambiguity. This is very easy, we were doing this 20 years ago and autogenerating the conversions using ANTLR that parsed the XDR files for ONC/RPC.
- joshuamorton 8y ago
  This doesn't make sense with the concept of rollbacks. If I rollout server version 2 and client version 2 which each use a new required field, and then realize that there is some terrible error in server version 2, I can't roll it back to version 1, since it will reject all client calls from v2 clients. The only way to make it work is to add a translation layer, as you suggest, on the server, wait a while, push the new 'required' client, wait a while longer, and then push the server without the translation layer. That's the "strict ordering with appropriate soak time" GP mentions.
  - docker_up 8y ago
    Instead, you're going to get errors from the clients using version 2, because server version 2 was rolled back. You have to roll back the clients as well then. Or you could have client version 2 know how to automatically convert to server version 1, because you're know what version the server is on, and you can convert your client parameters or even behavior to fit version 1. You can't do this with protobufs because there is no such concept, you just add optional fields, and ignore them with different versions, and it's chaos.

AndrewDucker 8y ago

Middle Name. Address Line 2. Date Of Death.

fixermark 8y ago

If all fields are required, how do you add a new required field without breaking all of the clients that were using the previous protobuffer version?