4 ms·

> Validate your data in application code, at consumption time, where you can handle errors gracefully. Honest question: how can I validate data in application

> Validate your data in application code, at consumption time, where you can handle errors gracefully. Honest question: how can I validate data in application code when optional fields decode to a necessarily-valid value by design? Suppose I'm an application author and I have an integer field called "quantity" which decoded to a 0. How can I tell whether that 0 meant "the quantity was 0 in the database" or "the quantity field was missing" instead? (One answer is that I should opt into a different default value, like -1, which the application can know indicates failure. If that's what I should always do, then why not help me gracefully recover by requiring that I always specify my fallback value explicitly, rather than silently defaulting to a potentially misinterpretable valid value like `0`?) I understand that required fields break message buses that only need to decode the envelope, but if I am working on a client/server application where message buses are not involved (as almost all client/server programmers in the world are), I don't follow how "everything is optional, and optional means always succeed with a valid default value" facilitates graceful recovery in the application layer. In order to gracefully recover, the application has to be informed that something went wrong! It seems to me that this design more directly facilitates bugs in the application layer that are difficult to detect because the information that something unexpected happened during decoding is intentionally discarded by default. It makes the resulting bugs "not the protocol layer's fault" by definition, but that is not a compelling pitch to me as an application author. What am I missing?

xyzzyz 8y ago
> Suppose I'm an application author and I have an integer field called "quantity" which decoded to a 0. How can I tell whether that 0 meant "the quantity was 0 in the database" or "the quantity field was missing" instead? First, this is clear on the level of wire encoding: either the field has encoded 0 value, or it is simply missing from encoding. Second, in proto2, you actually have has_quantity() method on a proto message, which will tell you whether quantity is missing or set to 0. In proto3, the design decision was that the has_foo() methods are available only on embedded message field, and not available on primitive fields, so you'd have to wrap your int64 in a message wrapper, like e.g. the ones available in google/protobuf/wrappers.proto. The point here (and a common pattern inside google3) is that in your handling code you simply manually check the presence of all required fields: if (!foo.has_quantity()) { return FailedPreconditionError("missing quantity"); }. It is a bit of a hassle, but the benefit is that you have control on where the bug originates and how it is handled in your application layer, as opposed to silently dropping the whole proto message on the floor.
- rtfeldman 8y ago
  Gotcha, thank you for the clear explanation!
kentonv 8y ago
In proto2, you could use `has_foo()` to check if `foo` is present, even for integer types. You could also specify what the default value should be, so you could specify e.g. a default of -1 or some other invalid value, if zero is valid for your app. Unfortunately, proto3 removed both of these features (`has_` and non-zero defaults). I personally think that was a mistake. I'm not sure what proto3 considers idiomatic here. Proto3 is after my time. Cap'n Proto also doesn't support `has_` due to the nature of the encoding, but it does support defaults. So you can set a default of -1 or whatever. Alternatively, you can declare a union like: # (Cap'n Proto syntax) foo :union { unset @0 :Void; value @1 :Int32; } This will take an extra 16 bits on the wire to store the tag, but gets the job done. `unset` will be the default state of the union because it has the lowest ordinal number. I suppose in proto3, you ought to be able to use a `oneof` in a similar way.

xyzzyz 8y ago

> Suppose I'm an application author and I have an integer field called "quantity" which decoded to a 0. How can I tell whether that 0 meant "the quantity was 0 in the database" or "the quantity field was missing" instead? First, this is clear on the level of wire encoding: either the field has encoded 0 value, or it is simply missing from encoding. Second, in proto2, you actually have has_quantity() method on a proto message, which will tell you whether quantity is missing or set to 0. In proto3, the design decision was that the has_foo() methods are available only on embedded message field, and not available on primitive fields, so you'd have to wrap your int64 in a message wrapper, like e.g. the ones available in google/protobuf/wrappers.proto. The point here (and a common pattern inside google3) is that in your handling code you simply manually check the presence of all required fields: if (!foo.has_quantity()) { return FailedPreconditionError("missing quantity"); }. It is a bit of a hassle, but the benefit is that you have control on where the bug originates and how it is handled in your application layer, as opposed to silently dropping the whole proto message on the floor.

rtfeldman 8y ago
Gotcha, thank you for the clear explanation!

kentonv 8y ago

In proto2, you could use `has_foo()` to check if `foo` is present, even for integer types. You could also specify what the default value should be, so you could specify e.g. a default of -1 or some other invalid value, if zero is valid for your app. Unfortunately, proto3 removed both of these features (`has_` and non-zero defaults). I personally think that was a mistake. I'm not sure what proto3 considers idiomatic here. Proto3 is after my time. Cap'n Proto also doesn't support `has_` due to the nature of the encoding, but it does support defaults. So you can set a default of -1 or whatever. Alternatively, you can declare a union like: # (Cap'n Proto syntax) foo :union { unset @0 :Void; value @1 :Int32; } This will take an extra 16 bits on the wire to store the tag, but gets the job done. `unset` will be the default state of the union because it has the lowest ordinal number. I suppose in proto3, you ought to be able to use a `oneof` in a similar way.