3 ms·

Can you elaborate more on the "required" fields point? We've been using a similar feature for several years now in APIs at my work and haven't run into any issu

Can you elaborate more on the "required" fields point? We've been using a similar feature for several years now in APIs at my work and haven't run into any issues, though we do only use it very sparingly for fields that logically can never be missing. At some point a client has to make the call for what they consider essential, so pushing it in the schema makes this less ambiguous from what I've seen. Maybe it's fine for our use-case (mostly static APIs), whereas what you're saying is good advice in general.

srtjstjsj 6y ago
https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers https://capnproto.org/faq.html#how-do-i-make-a-field-require... Required now means requires forever because people can't migrate safely. But technically you can change a protocol descriptor from required to optional, which is invalid (usually, in a distributed non-transactional system (the common kin) but nothing stops you from doing it. So why not make required forever? Well, do you really want to commit to anything forever?
- dodobirdlord 6y ago
  Protocol buffers already require you to commit to some things forever, like the type of a field, or whether two fields belong in a oneof together. I’m not saying that “required” was a great feature, but it’s not exactly unique.
  - joshuamorton 6y ago
    No they don't. An optional field can be deprecated and replaced with a different field. This can be done to change the type (also some types can be changed, although you probably shouldn't). Required usually cannot be deprecated.
    - dodobirdlord 6y ago
      You can deprecate an optional field and reserve the field number, but if you reintroduce use of that field number with a field of a different type that is not backward compatible. Types can be changed if they are binary compatible, but in that case they haven’t actually changed, because the binary format is the canonical format.
      - joshuamorton 6y ago
        Right, I'm not sure what your point is. You can always add more fields, so being unable to change the type of a field isn't a problem, since you can introduce a new field and start using it. You cannot however, stop using a required field. The best you can do is set it to a nonsense value and leave a comment saying "well we need to set this to something, but we don't actually use it anywhere. Because to be sure of that you can remove it, you need to be sure that every storage system and every piece of middleware and every since thing that links your proto anywhere in the world that you might care about is upgraded, otherwise if they encounter a new message they'll crash. If you only have a single client and server, and you control both, this is doable. If you don't have that though, you cannot.
- trevor-e 6y ago
  After reading that article my take-away is not that "required" is bad and should never be used ever, but rather it was bad with how Google wanted to use it. And since this is Google's project, it makes sense for them to remove the feature if it's causing data center outages, it's not worth the risk at that point. For example, in the case of the message bus they say "And even though the message bus doesn’t care about message content", and later on "The right answer is for applications to do validation as-needed in application-level code." Strict schema and validation is most helpful for application developers, not some middleware routing code. Was it not possible for them to write a parser that doesn't fully validate the message for use-cases like this?

srtjstjsj 6y ago

https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers https://capnproto.org/faq.html#how-do-i-make-a-field-require... Required now means requires forever because people can't migrate safely. But technically you can change a protocol descriptor from required to optional, which is invalid (usually, in a distributed non-transactional system (the common kin) but nothing stops you from doing it. So why not make required forever? Well, do you really want to commit to anything forever?

dodobirdlord 6y ago
Protocol buffers already require you to commit to some things forever, like the type of a field, or whether two fields belong in a oneof together. I’m not saying that “required” was a great feature, but it’s not exactly unique.
- joshuamorton 6y ago
  No they don't. An optional field can be deprecated and replaced with a different field. This can be done to change the type (also some types can be changed, although you probably shouldn't). Required usually cannot be deprecated.
  - dodobirdlord 6y ago
    You can deprecate an optional field and reserve the field number, but if you reintroduce use of that field number with a field of a different type that is not backward compatible. Types can be changed if they are binary compatible, but in that case they haven’t actually changed, because the binary format is the canonical format.
    - joshuamorton 6y ago
      Right, I'm not sure what your point is. You can always add more fields, so being unable to change the type of a field isn't a problem, since you can introduce a new field and start using it. You cannot however, stop using a required field. The best you can do is set it to a nonsense value and leave a comment saying "well we need to set this to something, but we don't actually use it anywhere. Because to be sure of that you can remove it, you need to be sure that every storage system and every piece of middleware and every since thing that links your proto anywhere in the world that you might care about is upgraded, otherwise if they encounter a new message they'll crash. If you only have a single client and server, and you control both, this is doable. If you don't have that though, you cannot.
trevor-e 6y ago
After reading that article my take-away is not that "required" is bad and should never be used ever, but rather it was bad with how Google wanted to use it. And since this is Google's project, it makes sense for them to remove the feature if it's causing data center outages, it's not worth the risk at that point. For example, in the case of the message bus they say "And even though the message bus doesn’t care about message content", and later on "The right answer is for applications to do validation as-needed in application-level code." Strict schema and validation is most helpful for application developers, not some middleware routing code. Was it not possible for them to write a parser that doesn't fully validate the message for use-cases like this?