5 ms·

So Google's protocol buffers have this feature called "required" fields, which enforce schema in the type system. You should never use it. Never. It's one of th

So Google's protocol buffers have this feature called "required" fields, which enforce schema in the type system. You should never use it. Never. It's one of those things that sound good until you're a few years into the project. Similar to how you should never be using meaningful IDs as primary keys for objects, always use meaningless fingerprint-like integers. Or how all integers should be signed unless you're dead sure the number is unsigned (like a fingerprint). And how many integers should actually be strings, unless you're dead sure this is a number (externally provided IDs, such as for example customer account IDs, are not numbers). Or how you should be careful to use bytes rather than unicode strings. Make your schema permissible and your code paranoid, it will pay off later. Build a data linter if necessary, but don't tie the schema.

nemetroid 6y ago
I feel that this advice is almost opposite to that given in the article. By making all fields optional, your data model no longer helps in making invalid states unrepresentable.
- H8crilA 6y ago
  Exactly! Because today's invalid states are commonplace in the future.
jdmichal 6y ago
> And how many integers should actually be strings, unless you're dead sure this is a number. I phrase this as: If it doesn't make sense to do math on it, it's not a number. What does adding one to a customer account number mean? Absolutely nothing -- you get a completely different account number. So it's not a number, but a numeric string.
- lukasLansky 6y ago
  It's not a string either though: in the same way integer addition (almost always) does not make sense, string concatenation (almost always) does not make sense either. The proper type would allow for equality check and explicit string (de)serialization only.
  - jdmichal 6y ago
    I meant string in a more casual sense. From a more technical sense, it would be a symbol. I'll be more precise in the future.
titanomachy 6y ago
That feature's been dropped in proto3, all fields are optional.
trevor-e 6y ago
Can you elaborate more on the "required" fields point? We've been using a similar feature for several years now in APIs at my work and haven't run into any issues, though we do only use it very sparingly for fields that logically can never be missing. At some point a client has to make the call for what they consider essential, so pushing it in the schema makes this less ambiguous from what I've seen. Maybe it's fine for our use-case (mostly static APIs), whereas what you're saying is good advice in general.
- srtjstjsj 6y ago
  https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers https://capnproto.org/faq.html#how-do-i-make-a-field-require... Required now means requires forever because people can't migrate safely. But technically you can change a protocol descriptor from required to optional, which is invalid (usually, in a distributed non-transactional system (the common kin) but nothing stops you from doing it. So why not make required forever? Well, do you really want to commit to anything forever?
  - dodobirdlord 6y ago
    Protocol buffers already require you to commit to some things forever, like the type of a field, or whether two fields belong in a oneof together. I’m not saying that “required” was a great feature, but it’s not exactly unique.
    - joshuamorton 6y ago
      No they don't. An optional field can be deprecated and replaced with a different field. This can be done to change the type (also some types can be changed, although you probably shouldn't). Required usually cannot be deprecated.
      - dodobirdlord 6y ago
        You can deprecate an optional field and reserve the field number, but if you reintroduce use of that field number with a field of a different type that is not backward compatible. Types can be changed if they are binary compatible, but in that case they haven’t actually changed, because the binary format is the canonical format.
sathorn 6y ago
This might make sense for a transport schema because you can receive messages from the past or the future but it does not translate to internal program state or database schemas where this is not the case. Making invalid states unrepresentable is basically the process of taking human-checked invariants and turning them into type-checked invariants. This reduces the likelihood of bugs and guides humans to use the system correctly.
srtjstjsj 6y ago
"required" is an example of making an invalid state representable and having it ruin your program.
- dodobirdlord 6y ago
  Yea, a better example of making invalid state unrepresentable in Google’s protocol buffers is to use the “oneof” feature to mark that a set of fields are mutually exclusive. If A, B, and C are mutually exclusive you can put them in a oneof, which also saves space in the binary representation. If in future you discover that A and B but not C needs to be a valid state, you can add a 4th AB option inside of the oneof.
yongjik 6y ago
I may be in the minority, but after happily using protobuf for years, I believe that there's nothing inherently wrong with required fields - instead, what's "wrong" is the protocol buffer API. Namely, when constructing a protobuf, theoretically, there might be two different ways: (A) first gather all the fields, and then construct the protobuf from these fields; (B) first construct an empty protobuf, and fill in the fields as necessary. The actual protobuf uses (B) - which is convenient in most cases, because when you start constructing a protobuf usually you don't have all the data ready yet. However, with required fields, this means when you construct the protobuf it starts with all required fields missing - i.e., an invalid state! I'm not sure what's the best way to fix it, because it would be infeasible to rewrite all the code to gather all the fields and then construct the protobuf - also it will be hugely inefficient in many cases. However, I feel the "no required fields" rule is essentially a null pointer (the "billion dollar mistake") in disguise - the actual problem is that the API doesn't enforce type safety.
- joshuamorton 6y ago
  This isn't actually the issue with required fields (some languages, like java and (usually) python, use a construct-once style). Imagine you have an innocent `required` field. You have a producer and a consumer of that field that communicate over the wire. (or instead of the wire, imagine a database). You send or store an instance of that protobuf. Now let's say that you want to make the field optional (or remove it). With an already-optional field, this is easy. You stop setting it, and maybe eventually you clean it up. With a required field, however, you can't do that. If any of your clients don't have the newest schema version, you can't unset the field (so imagine that you support mobile clients who may never update). Or if there's middleware you don't know about that introspects your proto. Even if you do the dance right and update your server and client before not setting the new field, you could crash outdated middleware that you didn't know about. Whoops! Or with the database, you now need to dual write or something complex because if you need to roll-back to an older version, you'd be unable to read the protos that don't include the required field. Required doesn't do well over time. It has nothing to do with setting the values.
  - deleted 6y ago

nemetroid 6y ago

I feel that this advice is almost opposite to that given in the article. By making all fields optional, your data model no longer helps in making invalid states unrepresentable.

H8crilA 6y ago
Exactly! Because today's invalid states are commonplace in the future.

jdmichal 6y ago

> And how many integers should actually be strings, unless you're dead sure this is a number. I phrase this as: If it doesn't make sense to do math on it, it's not a number. What does adding one to a customer account number mean? Absolutely nothing -- you get a completely different account number. So it's not a number, but a numeric string.

lukasLansky 6y ago
It's not a string either though: in the same way integer addition (almost always) does not make sense, string concatenation (almost always) does not make sense either. The proper type would allow for equality check and explicit string (de)serialization only.
- jdmichal 6y ago
  I meant string in a more casual sense. From a more technical sense, it would be a symbol. I'll be more precise in the future.

titanomachy 6y ago

That feature's been dropped in proto3, all fields are optional.

trevor-e 6y ago

Can you elaborate more on the "required" fields point? We've been using a similar feature for several years now in APIs at my work and haven't run into any issues, though we do only use it very sparingly for fields that logically can never be missing. At some point a client has to make the call for what they consider essential, so pushing it in the schema makes this less ambiguous from what I've seen. Maybe it's fine for our use-case (mostly static APIs), whereas what you're saying is good advice in general.

srtjstjsj 6y ago
https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers https://capnproto.org/faq.html#how-do-i-make-a-field-require... Required now means requires forever because people can't migrate safely. But technically you can change a protocol descriptor from required to optional, which is invalid (usually, in a distributed non-transactional system (the common kin) but nothing stops you from doing it. So why not make required forever? Well, do you really want to commit to anything forever?
- dodobirdlord 6y ago
  Protocol buffers already require you to commit to some things forever, like the type of a field, or whether two fields belong in a oneof together. I’m not saying that “required” was a great feature, but it’s not exactly unique.
  - joshuamorton 6y ago
    No they don't. An optional field can be deprecated and replaced with a different field. This can be done to change the type (also some types can be changed, although you probably shouldn't). Required usually cannot be deprecated.
    - dodobirdlord 6y ago
      You can deprecate an optional field and reserve the field number, but if you reintroduce use of that field number with a field of a different type that is not backward compatible. Types can be changed if they are binary compatible, but in that case they haven’t actually changed, because the binary format is the canonical format.

sathorn 6y ago

This might make sense for a transport schema because you can receive messages from the past or the future but it does not translate to internal program state or database schemas where this is not the case. Making invalid states unrepresentable is basically the process of taking human-checked invariants and turning them into type-checked invariants. This reduces the likelihood of bugs and guides humans to use the system correctly.

srtjstjsj 6y ago

"required" is an example of making an invalid state representable and having it ruin your program.

dodobirdlord 6y ago
Yea, a better example of making invalid state unrepresentable in Google’s protocol buffers is to use the “oneof” feature to mark that a set of fields are mutually exclusive. If A, B, and C are mutually exclusive you can put them in a oneof, which also saves space in the binary representation. If in future you discover that A and B but not C needs to be a valid state, you can add a 4th AB option inside of the oneof.

yongjik 6y ago

I may be in the minority, but after happily using protobuf for years, I believe that there's nothing inherently wrong with required fields - instead, what's "wrong" is the protocol buffer API. Namely, when constructing a protobuf, theoretically, there might be two different ways: (A) first gather all the fields, and then construct the protobuf from these fields; (B) first construct an empty protobuf, and fill in the fields as necessary. The actual protobuf uses (B) - which is convenient in most cases, because when you start constructing a protobuf usually you don't have all the data ready yet. However, with required fields, this means when you construct the protobuf it starts with all required fields missing - i.e., an invalid state! I'm not sure what's the best way to fix it, because it would be infeasible to rewrite all the code to gather all the fields and then construct the protobuf - also it will be hugely inefficient in many cases. However, I feel the "no required fields" rule is essentially a null pointer (the "billion dollar mistake") in disguise - the actual problem is that the API doesn't enforce type safety.

joshuamorton 6y ago
This isn't actually the issue with required fields (some languages, like java and (usually) python, use a construct-once style). Imagine you have an innocent `required` field. You have a producer and a consumer of that field that communicate over the wire. (or instead of the wire, imagine a database). You send or store an instance of that protobuf. Now let's say that you want to make the field optional (or remove it). With an already-optional field, this is easy. You stop setting it, and maybe eventually you clean it up. With a required field, however, you can't do that. If any of your clients don't have the newest schema version, you can't unset the field (so imagine that you support mobile clients who may never update). Or if there's middleware you don't know about that introspects your proto. Even if you do the dance right and update your server and client before not setting the new field, you could crash outdated middleware that you didn't know about. Whoops! Or with the database, you now need to dual write or something complex because if you need to roll-back to an older version, you'd be unable to read the protos that don't include the required field. Required doesn't do well over time. It has nothing to do with setting the values.
- deleted 6y ago