3 ms·

I wonder how this aligns with the protobuf best practice of having the first value be UNSPECIFIED: https://protobuf.dev/best-practices/dos-donts/#unspecified-e

by zdw 2y ago

I wonder how this aligns with the protobuf best practice of having the first value be UNSPECIFIED: https://protobuf.dev/best-practices/dos-donts/#unspecified-enum https://protobuf.dev/best-practices/dos-donts/#unspecified-e...

jmole 2y ago
The example code used added “other” as the last option, which was the source of the problems he described. This doesn’t happen when you make the first value in the enum unknown/unspecified
- plorkyeran 2y ago
  No, the problem described in the article is entirely unrelated to where in the enum the Other option is located. There is a different problem where keeping the Other option at the end of the enum changes the value of Other, but that is not the problem that the article is about.
  - jmole 2y ago
    Well it simplifies the logic considerably - if you see an enum value you don’t recognize (mint), you treat it as uninitialized (0). So any future new flavor will be read back as ‘0’ in older versions.
beart 2y ago
"Unspecified" is semantically different from "other". The former is more like a default value whereas the latter is actually "specified, but not one of these listed options".
- hamandcheese 2y ago
  Standard practice in protobuf is to never assign semantic meaning to the default value. I think some linters enforce that enum 0 is named "unknown" which is actually more semantically correct than "other" or "unspecified".
seeknotfind 2y ago
This is the same as a null pointer, and the requirement is very deeply tied to protobuf as it is used on large distributed systems that always need to handle version mismatch, and this advice doesn't necessarily apply to API design in general.
- eddd-ddde 2y ago
  Even in the simplest web apps you can encounter version mismatch when a client requests a response from a server that just updated.
  - hansvm 2y ago
    Hence the advice to make that situation not happen. Update the client and server to support both versions and prefer the new one, then update both to not support the old version. With load balancers and other real-world problems you might have to break that down into 4 coordinated steps.
    - Joker_vD 2y ago
      That only really works if you control the clients, or can force them to update.
      - LoganDark 2y ago
        > or can force them to update. I've used a few clients that completely lock me out for every tiniest minor version update. Very top-tier annoying imho.
        eru 2y ago
        But it does make the authors' jobs easier.
  - seeknotfind 2y ago
    This implies an API where the server has a single shared implementation. Imagine for instance that the server implements a shim for each version of the interface, then there isn't a need for the null in the API. Imagine another alternative, that the same API never adds a field, but you add a new method which takes the new type. Imagine yet again an API where you are able to version the clients in lockstep. So, it's a decision about how the API is used and evolves that recommends the API encoding or having a null default. However in a different environment or with different practices, you can avoid the null. Of course the reason to avoid the null is so that you can statically enforce this value is provided in new clients, though this also assumes your client language is typed. So in the end, protobuf teaches us, but it's not always the best in every situation.
bocahtie 2y ago
When the deserializing half of the protobuf definitions encounter an unknown value, it gets deserialized as the zero value. When that client updates, it will then be able to deserialize the new value appropriately (in this case, "Mint"). The advice on that page also specifies to not make the value semantically meaningful, which I take to mean to never set it to that value explicitly.
- chen_dev 2y ago
  > it gets deserialized as the zero value It’s more complicated: https://protobuf.dev/programming-guides/enum/ https://protobuf.dev/programming-guides/enum/ >> What happens when a program parses binary data that contains field 1 with the value 2? >- Open enums will parse the value 2 and store it directly in the field. Accessor will report the field as being set and will return something that represents 2. >- Closed enums will parse the value 2 and store it in the message’s unknown field set. Accessors will report the field as being unset and will return the enum’s default value.
  - vitus 2y ago
    Ugh. I hate how we (Google) launched proto editions. It used to be that we broadly had two sets of semantics (modulo additional customizations): proto2 and proto3. Proto editions was supposed to unify the two versions, but instead now we have the option to mix and match all of the quirks of each of the versions. And, to make matters worse, you also have language-dependent implementations that don't conform to the spec (in fact, very few implementations are conformant). C++ and Java treat everything imported by a proto2 file as closed; C#, Golang, and JS treat everything as open. I don't see a path forward for removing these custom deprecated field features, or else we'd have already begun that effort during the initial adoption of editions.
- dwattttt 2y ago
  > The advice on that page also specifies to not make the value semantically meaningful, which I take to mean to never set it to that value explicitly. I've taken to coding my C enums with the first value being "Invalid", indicating it is never intended to be created. If one is encountered, it's a bug.
MarkMarine 2y ago
I don’t mind the zero value for the proto enums, makes sense, but I require converting to my inner logic to not include this “unknown” and error during the conversion if it fails. I’ve seen engineers bring those unknowns or unspecified through to the business logic and that always made my face flush red with anger.
- fmbb 2y ago
  Why the anger? If you are consuming data from some other system you have no power over what to require from users. You will have data points with unknown properties. Say you are tracking sign ups in some other system, and they collect the users’ browser in the process, and you want to see conversion rate per browser. If the browser could not be identified, you prefer it to say ”other” instead of ”unknown”? I think I prefer the protobuf best practices way: you have a 0 ”unknown”/”unset” value, and you enumerate the rest with a unique name (and number). The enum can be expanded in the future so your code must be prepared for unknown enumerated values tagged with the new (future for your code) number. They are all unique, you just don’t yet know the name of some of the enum values. You can choose to not consume them until your code is updated with a more recent schema. Or you can reconcile later, annotating with the name of you need it. Now personally, I would not pick an enum for any set och things that is not closed when you are designing. But I’m starting to think that such sets hardly exist in the real world. Humans redefine everything over time.
- crabbone 2y ago
  I wrote my own Protobuf implementation (well, with some changes). Ditching the default values was one of the changes I made. I don't see any reason to have that. But I don't think that Protobuf is a reasonable or even decent protocol in general. It has a lot of nonsense and bad planning. Having default values is probably not in the ten worst things about Protobuf.
deleted 2y ago
[deleted]

by zdw 2y ago

jmole 2y ago

The example code used added “other” as the last option, which was the source of the problems he described. This doesn’t happen when you make the first value in the enum unknown/unspecified

plorkyeran 2y ago
No, the problem described in the article is entirely unrelated to where in the enum the Other option is located. There is a different problem where keeping the Other option at the end of the enum changes the value of Other, but that is not the problem that the article is about.
- jmole 2y ago
  Well it simplifies the logic considerably - if you see an enum value you don’t recognize (mint), you treat it as uninitialized (0). So any future new flavor will be read back as ‘0’ in older versions.

beart 2y ago

"Unspecified" is semantically different from "other". The former is more like a default value whereas the latter is actually "specified, but not one of these listed options".

hamandcheese 2y ago
Standard practice in protobuf is to never assign semantic meaning to the default value. I think some linters enforce that enum 0 is named "unknown" which is actually more semantically correct than "other" or "unspecified".

seeknotfind 2y ago

This is the same as a null pointer, and the requirement is very deeply tied to protobuf as it is used on large distributed systems that always need to handle version mismatch, and this advice doesn't necessarily apply to API design in general.

eddd-ddde 2y ago
Even in the simplest web apps you can encounter version mismatch when a client requests a response from a server that just updated.
- hansvm 2y ago
  Hence the advice to make that situation not happen. Update the client and server to support both versions and prefer the new one, then update both to not support the old version. With load balancers and other real-world problems you might have to break that down into 4 coordinated steps.
  - Joker_vD 2y ago
    That only really works if you control the clients, or can force them to update.
    - LoganDark 2y ago
      > or can force them to update. I've used a few clients that completely lock me out for every tiniest minor version update. Very top-tier annoying imho.
      - eru 2y ago
        But it does make the authors' jobs easier.
- seeknotfind 2y ago
  This implies an API where the server has a single shared implementation. Imagine for instance that the server implements a shim for each version of the interface, then there isn't a need for the null in the API. Imagine another alternative, that the same API never adds a field, but you add a new method which takes the new type. Imagine yet again an API where you are able to version the clients in lockstep. So, it's a decision about how the API is used and evolves that recommends the API encoding or having a null default. However in a different environment or with different practices, you can avoid the null. Of course the reason to avoid the null is so that you can statically enforce this value is provided in new clients, though this also assumes your client language is typed. So in the end, protobuf teaches us, but it's not always the best in every situation.

bocahtie 2y ago

When the deserializing half of the protobuf definitions encounter an unknown value, it gets deserialized as the zero value. When that client updates, it will then be able to deserialize the new value appropriately (in this case, "Mint"). The advice on that page also specifies to not make the value semantically meaningful, which I take to mean to never set it to that value explicitly.

chen_dev 2y ago
> it gets deserialized as the zero value It’s more complicated: https://protobuf.dev/programming-guides/enum/ https://protobuf.dev/programming-guides/enum/ >> What happens when a program parses binary data that contains field 1 with the value 2? >- Open enums will parse the value 2 and store it directly in the field. Accessor will report the field as being set and will return something that represents 2. >- Closed enums will parse the value 2 and store it in the message’s unknown field set. Accessors will report the field as being unset and will return the enum’s default value.
- vitus 2y ago
  Ugh. I hate how we (Google) launched proto editions. It used to be that we broadly had two sets of semantics (modulo additional customizations): proto2 and proto3. Proto editions was supposed to unify the two versions, but instead now we have the option to mix and match all of the quirks of each of the versions. And, to make matters worse, you also have language-dependent implementations that don't conform to the spec (in fact, very few implementations are conformant). C++ and Java treat everything imported by a proto2 file as closed; C#, Golang, and JS treat everything as open. I don't see a path forward for removing these custom deprecated field features, or else we'd have already begun that effort during the initial adoption of editions.
dwattttt 2y ago
> The advice on that page also specifies to not make the value semantically meaningful, which I take to mean to never set it to that value explicitly. I've taken to coding my C enums with the first value being "Invalid", indicating it is never intended to be created. If one is encountered, it's a bug.

MarkMarine 2y ago

I don’t mind the zero value for the proto enums, makes sense, but I require converting to my inner logic to not include this “unknown” and error during the conversion if it fails. I’ve seen engineers bring those unknowns or unspecified through to the business logic and that always made my face flush red with anger.

fmbb 2y ago
Why the anger? If you are consuming data from some other system you have no power over what to require from users. You will have data points with unknown properties. Say you are tracking sign ups in some other system, and they collect the users’ browser in the process, and you want to see conversion rate per browser. If the browser could not be identified, you prefer it to say ”other” instead of ”unknown”? I think I prefer the protobuf best practices way: you have a 0 ”unknown”/”unset” value, and you enumerate the rest with a unique name (and number). The enum can be expanded in the future so your code must be prepared for unknown enumerated values tagged with the new (future for your code) number. They are all unique, you just don’t yet know the name of some of the enum values. You can choose to not consume them until your code is updated with a more recent schema. Or you can reconcile later, annotating with the name of you need it. Now personally, I would not pick an enum for any set och things that is not closed when you are designing. But I’m starting to think that such sets hardly exist in the real world. Humans redefine everything over time.
crabbone 2y ago
I wrote my own Protobuf implementation (well, with some changes). Ditching the default values was one of the changes I made. I don't see any reason to have that. But I don't think that Protobuf is a reasonable or even decent protocol in general. It has a lot of nonsense and bad planning. Having default values is probably not in the ten worst things about Protobuf.

deleted 2y ago

[deleted]