3 ms·

I may be in the minority, but after happily using protobuf for years, I believe that there's nothing inherently wrong with required fields - instead, what's "wr

I may be in the minority, but after happily using protobuf for years, I believe that there's nothing inherently wrong with required fields - instead, what's "wrong" is the protocol buffer API. Namely, when constructing a protobuf, theoretically, there might be two different ways: (A) first gather all the fields, and then construct the protobuf from these fields; (B) first construct an empty protobuf, and fill in the fields as necessary. The actual protobuf uses (B) - which is convenient in most cases, because when you start constructing a protobuf usually you don't have all the data ready yet. However, with required fields, this means when you construct the protobuf it starts with all required fields missing - i.e., an invalid state! I'm not sure what's the best way to fix it, because it would be infeasible to rewrite all the code to gather all the fields and then construct the protobuf - also it will be hugely inefficient in many cases. However, I feel the "no required fields" rule is essentially a null pointer (the "billion dollar mistake") in disguise - the actual problem is that the API doesn't enforce type safety.

joshuamorton 6y ago
This isn't actually the issue with required fields (some languages, like java and (usually) python, use a construct-once style). Imagine you have an innocent `required` field. You have a producer and a consumer of that field that communicate over the wire. (or instead of the wire, imagine a database). You send or store an instance of that protobuf. Now let's say that you want to make the field optional (or remove it). With an already-optional field, this is easy. You stop setting it, and maybe eventually you clean it up. With a required field, however, you can't do that. If any of your clients don't have the newest schema version, you can't unset the field (so imagine that you support mobile clients who may never update). Or if there's middleware you don't know about that introspects your proto. Even if you do the dance right and update your server and client before not setting the new field, you could crash outdated middleware that you didn't know about. Whoops! Or with the database, you now need to dual write or something complex because if you need to roll-back to an older version, you'd be unable to read the protos that don't include the required field. Required doesn't do well over time. It has nothing to do with setting the values.
- deleted 6y ago
  [deleted]
- grogers 6y ago
  If you have old clients that expect that field you are removing to be there in a meaningful way, you still have to update all clients before you can stop setting it. Having the protobuf schema itself use optional or required doesn't change that, it just makes the dependency explicit there, instead of only in the code at the endpoints. Changing required to optional isn't a magic fix for protocol compatibility. If it were (for your limited use case) you can just make that change to the protobuf client side as it doesn't affect the wire representation/interpretation.
  - joshuamorton 6y ago
    > If you have old clients that expect that field you are removing to be there in a meaningful way Right, there's the rub. `required` means that anyone who deserializes your proto falls into this category. That's a much larger group than "anyone who reads a specific field". So the list of clients now is forced to include any and all middleware that may read your proto (imagine a routing layer or some kind of analytics system or whatnot). (Note also that there's lots of ways to make reading a field that is empty fallback to doing some reasonable non-catastrophic behavior, required doesn't let you do those things).
- lalos 6y ago
  What about changing it to optional on the server side, and clients that don't upgrade will always send that required payload and you will know to parse it but then ignore it. Updated clients know its optional and also ignore it and don't set it. Once you have low frequency of messages containing that required field you can do the final cleanup.
  - andrewflnr 6y ago
    I think the problem is when outdated clients try to parse responses with a missing, used-to-be-required field.

joshuamorton 6y ago

This isn't actually the issue with required fields (some languages, like java and (usually) python, use a construct-once style). Imagine you have an innocent `required` field. You have a producer and a consumer of that field that communicate over the wire. (or instead of the wire, imagine a database). You send or store an instance of that protobuf. Now let's say that you want to make the field optional (or remove it). With an already-optional field, this is easy. You stop setting it, and maybe eventually you clean it up. With a required field, however, you can't do that. If any of your clients don't have the newest schema version, you can't unset the field (so imagine that you support mobile clients who may never update). Or if there's middleware you don't know about that introspects your proto. Even if you do the dance right and update your server and client before not setting the new field, you could crash outdated middleware that you didn't know about. Whoops! Or with the database, you now need to dual write or something complex because if you need to roll-back to an older version, you'd be unable to read the protos that don't include the required field. Required doesn't do well over time. It has nothing to do with setting the values.

deleted 6y ago
[deleted]
grogers 6y ago
If you have old clients that expect that field you are removing to be there in a meaningful way, you still have to update all clients before you can stop setting it. Having the protobuf schema itself use optional or required doesn't change that, it just makes the dependency explicit there, instead of only in the code at the endpoints. Changing required to optional isn't a magic fix for protocol compatibility. If it were (for your limited use case) you can just make that change to the protobuf client side as it doesn't affect the wire representation/interpretation.
- joshuamorton 6y ago
  > If you have old clients that expect that field you are removing to be there in a meaningful way Right, there's the rub. `required` means that anyone who deserializes your proto falls into this category. That's a much larger group than "anyone who reads a specific field". So the list of clients now is forced to include any and all middleware that may read your proto (imagine a routing layer or some kind of analytics system or whatnot). (Note also that there's lots of ways to make reading a field that is empty fallback to doing some reasonable non-catastrophic behavior, required doesn't let you do those things).
lalos 6y ago
What about changing it to optional on the server side, and clients that don't upgrade will always send that required payload and you will know to parse it but then ignore it. Updated clients know its optional and also ignore it and don't set it. Once you have low frequency of messages containing that required field you can do the final cleanup.
- andrewflnr 6y ago
  I think the problem is when outdated clients try to parse responses with a missing, used-to-be-required field.