5 ms·

The latest version of protobuf does support passing opaque messages [1] and disallows required fields [2]. So I guess lessons were learned. 1: https://develope

The latest version of protobuf does support passing opaque messages [1] and disallows required fields [2]. So I guess lessons were learned. 1: https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/Any https://developers.google.com/protocol-buffers/docs/referenc... 2: https://stackoverflow.com/questions/31801257/why-required-and-optional-is-removed-in-protocol-buffers-3 https://stackoverflow.com/questions/31801257/why-required-an...

mwcampbell 4y ago
Yes, and I'm disagreeing with the second one.
- jvolkman 4y ago
  Ah, got it. Well as far as I know, proto2 (with required) isn't going anywhere and Any can still be used with it. One nice thing about protobuf is its extensibility, so you can still get required fields with something like protoc-gen-validate, but interpret them closer to the app's logic. https://github.com/envoyproxy/protoc-gen-validate https://github.com/envoyproxy/protoc-gen-validate
- cbarrick 4y ago
  Required fields really are a terrible idea. Having the validation performed in code you don't control (generated) and that is pervasive (middle services that you don't control) is a nightmare for API evolution. If something is truly required, you're better off writing the validation logic yourself, so that you can change it when your requirements change.
  - kelnos 4y ago
    > If something is truly required, you're better off writing the validation logic yourself Validation logic is often tricky, or at least tedious and error-prone (in part due to its tediousness). I would much rather have a battle-tested piece of software generate validation code for me. I don't really get this aversion to required fields. Some data is actually required, and this happens fairly often in APIs in my experience. And if that changes, then often you are fundamentally changing what the API does, so it should be a new API.
    - joshuamorton 4y ago
      What you are now arguing for is that every protobuf field addition or removal needs to be accompanied by a version bump of the underlying message, or in other words proto schema definitions are write-once, because if you add or remove a field, you're fundamentally changing what the api does too. This is, I guess, a technically valid approach to the problem, but there's a reason no one does it. > Some data is actually required, and this happens fairly often in APIs in my experience. And if that changes, then often you are fundamentally changing what the API does, so it should be a new API. I can think of some fabulous counterexamples to this that aren't really public, but suffice to say that there are lots of places in code with msg.foo = "" // This field is required, but has been deprecated in // favor of msg.bar for <efficiency/external change/time> // reasons. And the blame layer is from 2006 or something and it's still uncomfortable to try and fiddle with things today, because of rollout ordering among the potentially hundreds of clients and databases that use some core thing. Which like is the other thing: if you have hundreds of clients, you can't just say "hey use v2 now it's a totally different api, please update your code"

mwcampbell 4y ago

Yes, and I'm disagreeing with the second one.

jvolkman 4y ago
Ah, got it. Well as far as I know, proto2 (with required) isn't going anywhere and Any can still be used with it. One nice thing about protobuf is its extensibility, so you can still get required fields with something like protoc-gen-validate, but interpret them closer to the app's logic. https://github.com/envoyproxy/protoc-gen-validate https://github.com/envoyproxy/protoc-gen-validate
cbarrick 4y ago
Required fields really are a terrible idea. Having the validation performed in code you don't control (generated) and that is pervasive (middle services that you don't control) is a nightmare for API evolution. If something is truly required, you're better off writing the validation logic yourself, so that you can change it when your requirements change.
- kelnos 4y ago
  > If something is truly required, you're better off writing the validation logic yourself Validation logic is often tricky, or at least tedious and error-prone (in part due to its tediousness). I would much rather have a battle-tested piece of software generate validation code for me. I don't really get this aversion to required fields. Some data is actually required, and this happens fairly often in APIs in my experience. And if that changes, then often you are fundamentally changing what the API does, so it should be a new API.
  - joshuamorton 4y ago
    What you are now arguing for is that every protobuf field addition or removal needs to be accompanied by a version bump of the underlying message, or in other words proto schema definitions are write-once, because if you add or remove a field, you're fundamentally changing what the api does too. This is, I guess, a technically valid approach to the problem, but there's a reason no one does it. > Some data is actually required, and this happens fairly often in APIs in my experience. And if that changes, then often you are fundamentally changing what the API does, so it should be a new API. I can think of some fabulous counterexamples to this that aren't really public, but suffice to say that there are lots of places in code with msg.foo = "" // This field is required, but has been deprecated in // favor of msg.bar for <efficiency/external change/time> // reasons. And the blame layer is from 2006 or something and it's still uncomfortable to try and fiddle with things today, because of rollout ordering among the potentially hundreds of clients and databases that use some core thing. Which like is the other thing: if you have hundreds of clients, you can't just say "hey use v2 now it's a totally different api, please update your code"