4 ms·

Don't forget also the Protobuf C++ compiler's failure to properly namespace user-level identifiers vs. library-level identifiers. For example, if your Protobuf

Don't forget also the Protobuf C++ compiler's failure to properly namespace user-level identifiers vs. library-level identifiers. For example, if your Protobuf has both a "foo" and "has_foo" field (which is perfectly legal by the Protobuf language definition! and works fine with e.g. the Python binding!), you will get a C++ compiler error due to a "has_foo()" method being generated on behalf of both "foo" and "has_foo". This naming clash could have been avoided simply by prepending all generated method names with a defined prefix, but the implementors either didn't recognize this issue, or chose not to do anything about it. (Everything else in the article rings true for me. I've been hoping years for someone to write this article.)

xyzzyz 7y ago
> This naming clash could have been avoided simply by prepending all generated method names with a defined prefix, but the implementors either didn't recognize this issue, or chose not to do anything about it. I'd much rather live with not being able to have fields name "has_foo" in my protobuf than to have to prepend a prefix to every single access method.
- colanderman 7y ago
  While arbitrary, that restriction would be fine if it were part of the language definition, and enforced by the Protobuf compiler. (I say "arbitrary" because "has_" is just the nomenclature the C++ bindings happen to use. The syntactic peculiarities of other host languages may dictate a different prefix. Which then forces the question of whether to amend the list of "prohibited" field names, and potentially break existing code.) Regardless, the restriction you propose is not currently (to my knowledge) part of the language, so you can get into the situation where you've been developing with Protobufs in Python for years, and then decide to add some C++ code, and everything breaks because now you have naming clashes which force you to rename the field and go through and edit all the existing use sites of said field.
kentonv 7y ago
Yes, we were very much aware of this, and chose not to do anything about it. This problem almost never manifests in practice. The issue is raised all the time, but it's basically always observed only as a theoretical problem (by someone who invariably thinks they are sooooo smart for discovering it), not as a real problem preventing compilation of a real schema. Prepending all generated method names with a prefix would be a rather extreme solution that no one would like. Have you ever tried to read libstdc++'s STL implementation, where absolutely everything is prefixed with __? It's really quite awful. I wouldn't want to use a serialization framework that did that. The right solution, in my opinion, is to provide annotations that allow the developer to rename a particular field for the purpose of a particular target language, so that e.g. you can say that "has_foo" should be renamed to "has_foo_" (or whatever) in C++ generated code. Yeah, it's an ugly hack, but it gets the job done. I can't remember if this ever got implemented in Protobuf, because, again, it's almost never actually needed. Cap'n Proto does have such annotations, though. (Disclosure: I'm the author of Protobuf v2 and Cap'n Proto.)
- colanderman 7y ago
  > (by someone who invariably thinks they are sooooo smart for discovering it), not as a real problem preventing compilation of a real schema That's a pretty dismissive view of your users. This has actually bitten me in practice, so consider their foresight vindicated. (Notably, it was actually the inability to easily distinguish between a missing and empty array, which caused us to resort to using "has_foo" fields, only later to hit the issue with the C++ compiler.) If you dismiss this as a valid concern, how can I be confident that there are not other similar issues you simply dismissed as unimportant? Say what you will about STL, but the level of attention to detail there assures me that I'm not likely to get bitten by some weird issue the developers chose to turn a blind eye to. > you can say that "has_foo" should be renamed to "has_foo_" (or whatever) in C++ generated code. This is fine, even if it's a transformation predetermined by the language.
  - kentonv 7y ago
    Sorry for the snark. This issue is a sore spot for me because so many people have reported it without having actually been affected by it, and because they tend to assume the designers were stupidly unaware of the issue, rather than that the issue is actually rather hard to solve in a satisfying way. However, if you actually were affected by it, then you are right to be annoyed by it. The particular case where someone developed a protocol mostly in one language and then later on started targeting a new language is indeed a case that I do worry about. The idea of language-specific annotations defining language-specific renames was designed for that use case. I haven't worked on Protobuf in almost a decade, but Cap'n Proto does address this issue as I said -- without making everyone's code horribly ugly.

xyzzyz 7y ago

> This naming clash could have been avoided simply by prepending all generated method names with a defined prefix, but the implementors either didn't recognize this issue, or chose not to do anything about it. I'd much rather live with not being able to have fields name "has_foo" in my protobuf than to have to prepend a prefix to every single access method.

colanderman 7y ago
While arbitrary, that restriction would be fine if it were part of the language definition, and enforced by the Protobuf compiler. (I say "arbitrary" because "has_" is just the nomenclature the C++ bindings happen to use. The syntactic peculiarities of other host languages may dictate a different prefix. Which then forces the question of whether to amend the list of "prohibited" field names, and potentially break existing code.) Regardless, the restriction you propose is not currently (to my knowledge) part of the language, so you can get into the situation where you've been developing with Protobufs in Python for years, and then decide to add some C++ code, and everything breaks because now you have naming clashes which force you to rename the field and go through and edit all the existing use sites of said field.

kentonv 7y ago

Yes, we were very much aware of this, and chose not to do anything about it. This problem almost never manifests in practice. The issue is raised all the time, but it's basically always observed only as a theoretical problem (by someone who invariably thinks they are sooooo smart for discovering it), not as a real problem preventing compilation of a real schema. Prepending all generated method names with a prefix would be a rather extreme solution that no one would like. Have you ever tried to read libstdc++'s STL implementation, where absolutely everything is prefixed with __? It's really quite awful. I wouldn't want to use a serialization framework that did that. The right solution, in my opinion, is to provide annotations that allow the developer to rename a particular field for the purpose of a particular target language, so that e.g. you can say that "has_foo" should be renamed to "has_foo_" (or whatever) in C++ generated code. Yeah, it's an ugly hack, but it gets the job done. I can't remember if this ever got implemented in Protobuf, because, again, it's almost never actually needed. Cap'n Proto does have such annotations, though. (Disclosure: I'm the author of Protobuf v2 and Cap'n Proto.)

colanderman 7y ago
> (by someone who invariably thinks they are sooooo smart for discovering it), not as a real problem preventing compilation of a real schema That's a pretty dismissive view of your users. This has actually bitten me in practice, so consider their foresight vindicated. (Notably, it was actually the inability to easily distinguish between a missing and empty array, which caused us to resort to using "has_foo" fields, only later to hit the issue with the C++ compiler.) If you dismiss this as a valid concern, how can I be confident that there are not other similar issues you simply dismissed as unimportant? Say what you will about STL, but the level of attention to detail there assures me that I'm not likely to get bitten by some weird issue the developers chose to turn a blind eye to. > you can say that "has_foo" should be renamed to "has_foo_" (or whatever) in C++ generated code. This is fine, even if it's a transformation predetermined by the language.
- kentonv 7y ago
  Sorry for the snark. This issue is a sore spot for me because so many people have reported it without having actually been affected by it, and because they tend to assume the designers were stupidly unaware of the issue, rather than that the issue is actually rather hard to solve in a satisfying way. However, if you actually were affected by it, then you are right to be annoyed by it. The particular case where someone developed a protocol mostly in one language and then later on started targeting a new language is indeed a case that I do worry about. The idea of language-specific annotations defining language-specific renames was designed for that use case. I haven't worked on Protobuf in almost a decade, but Cap'n Proto does address this issue as I said -- without making everyone's code horribly ugly.