3 ms·

Googler, opinions are my own. I don't work on protobuf at all, just use it all the time (like most Googlers) I haven't dug into this in great detail yet, but

by kyrra 4y ago

Googler, opinions are my own. I don't work on protobuf at all, just use it all the time (like most Googlers) I haven't dug into this in great detail yet, but the hard thing about the proto "spec" is that there isn't one, and protoc lets you do all kinds of crazy things that are really hard to model in languages like Antlr. There were some poor choices in protoc dating back to when proto1 was first created that have been carried forward. Having 20 years of proto definitions lets people come up with some crazy use cases. Definitely interesting for this company to create an EBNF definition for protobuf.

jhumphries131 4y ago
It's possible that the internal version of protoc is very different from the open-source version. (I know there are numerous differences, but not sure how pervasive they are in the parser.) The open-source version has a hand-written tokenizer and recursive descent parser that is not too difficult to translate to EBNF. You'll notice that the section on numeric literals is a little wonky, because the tokenizer does a check that is hard to describe in EBNF. But it isn't too bad. Also, some of the constraints of the language are in prose in this spec because they are easier to enforce using a semantic validation pass, instead of trying to model purely with a CFG. (Optionality of the colon in the text format, used in message literals, comes to mind.) There are some things that technically _could_ be handled in the grammar, but they would make the grammar much more cumbersome to read and understand. So those things are also extracted into prose. > Definitely interesting for this company to create an EBNF definition for protobuf. For what it's worth, Google has also published an EBNF definition (the subject blog post contains links to those specs). But they are incomplete and not entirely accurate, which is a non-trivial part of what led us to writing and publishing this spec.
- kyrra 4y ago
  One place protoc doesn't align well is the descriptor object. https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.descriptor https://developers.google.com/protocol-buffers/docs/referenc... Comment placement is basically allowed anywhere by protoc, but how to get those comments within a Descriptor object for a proto is not well defined (there are places where you can put comments that are not available within Descriptor). It provides leading/trailing comments, but there are many other cases that are missed today (like comments embedded within a list of items in an array). Maybe this is a mismatch between what protoc allows and what Descriptor presents, but it's definitely annoying.
  - jvolkman 4y ago
    I totally forgot about .proto comments. I spent a while re-implementing that [1] for the IntelliJ editor and even longer testing it [2]. Apparently I opened a bug while doing so; is b/33539835 still open? 1: https://github.com/jvolkman/intellij-protobuf-editor/blob/master/core/src/main/java/idea/plugin/protoeditor/lang/psi/util/PbCommentUtil.java https://github.com/jvolkman/intellij-protobuf-editor/blob/ma... 2: https://github.com/jvolkman/intellij-protobuf-editor/blob/master/core/src/itest/java/idea/plugin/protoeditor/lang/comments/PbCommentUtilTest.java https://github.com/jvolkman/intellij-protobuf-editor/blob/ma...
  - jhumphries131 4y ago
    I agree! Protoc allows comments anywhere, but it doesn't bother preserving them all in the descriptor. At one point I thought this was a bug, since comments _could_ be preserved in far greater contexts (though definitely not all). But then I realized that the descriptor.proto comments do actually state the places where comments are retained: > If this SourceCodeInfo represents a complete declaration, these are any > comments appearing before and after the declaration which appear to be > attached to the declaration. https://github.com/protocolbuffers/protobuf/blob/v21.5/src/google/protobuf/descriptor.proto#L844 https://github.com/protocolbuffers/protobuf/blob/v21.5/src/g... After re-reading with this caveat in mind -- that comments are only preserved before and after complete declarations -- I realized that it does retain the comments that are expected. Long story short: the descriptor is bad as an AST if you care about recovering the original source. The descriptor is lossy. I'd recommend using a real AST for use cases that want something non-lossy: https://pkg.go.dev/github.com/jhump/protoreflect/desc/protoparse#Parser.ParseToAST https://pkg.go.dev/github.com/jhump/protoreflect/desc/protop...
jjtheblunt 4y ago
if comfortable answering, why protobuf instead of gob?
- whacker 4y ago
  proto predates gob by quite a bit. gob was introduced with golang, and it's not really used anywhere else.
  - erik_seaberg 4y ago
    My takeaway from Java serialization (which became passé) was that a schema-driven encoding that’s supported in many languages is a lot more useful.
    - adra 4y ago
      Java serialization is passe largely because it has huge gaping holes in how one can abuse the serialization loaders to perform very unsafe security holes. And it's incredibly slow, and tied to JVMs, but sadly only really tied to JVM that have the exact same version and serialized objects, etc.. did I mention java serialization sucks? (Spoken by a true java lover that that subjected himself to serialization way more than he wanted to)
- jhumphries131 4y ago
  Gob is Go-specific. Our mission is to make schema-driven APIs easy, regardless of what language you use. Protobuf already has official support for nearly a dozen languages, and unofficial support for many more. Protobuf also has a compiler with a plugin model, which facilitates supporting even more in the future. Furthermore, Protobuf is an IDL, not a full-blown programming language. This makes it ideal for this use case, for describing APIs and data structures. Gob-encoded data structures are described with Go. While Go is great for writing server-side business logic, it is not as well-suited as a description language for data that you need to share with non-Go systems.
  - jjtheblunt 4y ago

by kyrra 4y ago

jhumphries131 4y ago

It's possible that the internal version of protoc is very different from the open-source version. (I know there are numerous differences, but not sure how pervasive they are in the parser.) The open-source version has a hand-written tokenizer and recursive descent parser that is not too difficult to translate to EBNF. You'll notice that the section on numeric literals is a little wonky, because the tokenizer does a check that is hard to describe in EBNF. But it isn't too bad. Also, some of the constraints of the language are in prose in this spec because they are easier to enforce using a semantic validation pass, instead of trying to model purely with a CFG. (Optionality of the colon in the text format, used in message literals, comes to mind.) There are some things that technically _could_ be handled in the grammar, but they would make the grammar much more cumbersome to read and understand. So those things are also extracted into prose. > Definitely interesting for this company to create an EBNF definition for protobuf. For what it's worth, Google has also published an EBNF definition (the subject blog post contains links to those specs). But they are incomplete and not entirely accurate, which is a non-trivial part of what led us to writing and publishing this spec.

kyrra 4y ago
One place protoc doesn't align well is the descriptor object. https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.descriptor https://developers.google.com/protocol-buffers/docs/referenc... Comment placement is basically allowed anywhere by protoc, but how to get those comments within a Descriptor object for a proto is not well defined (there are places where you can put comments that are not available within Descriptor). It provides leading/trailing comments, but there are many other cases that are missed today (like comments embedded within a list of items in an array). Maybe this is a mismatch between what protoc allows and what Descriptor presents, but it's definitely annoying.
- jvolkman 4y ago
  I totally forgot about .proto comments. I spent a while re-implementing that [1] for the IntelliJ editor and even longer testing it [2]. Apparently I opened a bug while doing so; is b/33539835 still open? 1: https://github.com/jvolkman/intellij-protobuf-editor/blob/master/core/src/main/java/idea/plugin/protoeditor/lang/psi/util/PbCommentUtil.java https://github.com/jvolkman/intellij-protobuf-editor/blob/ma... 2: https://github.com/jvolkman/intellij-protobuf-editor/blob/master/core/src/itest/java/idea/plugin/protoeditor/lang/comments/PbCommentUtilTest.java https://github.com/jvolkman/intellij-protobuf-editor/blob/ma...
- jhumphries131 4y ago
  I agree! Protoc allows comments anywhere, but it doesn't bother preserving them all in the descriptor. At one point I thought this was a bug, since comments _could_ be preserved in far greater contexts (though definitely not all). But then I realized that the descriptor.proto comments do actually state the places where comments are retained: > If this SourceCodeInfo represents a complete declaration, these are any > comments appearing before and after the declaration which appear to be > attached to the declaration. https://github.com/protocolbuffers/protobuf/blob/v21.5/src/google/protobuf/descriptor.proto#L844 https://github.com/protocolbuffers/protobuf/blob/v21.5/src/g... After re-reading with this caveat in mind -- that comments are only preserved before and after complete declarations -- I realized that it does retain the comments that are expected. Long story short: the descriptor is bad as an AST if you care about recovering the original source. The descriptor is lossy. I'd recommend using a real AST for use cases that want something non-lossy: https://pkg.go.dev/github.com/jhump/protoreflect/desc/protoparse#Parser.ParseToAST https://pkg.go.dev/github.com/jhump/protoreflect/desc/protop...

jjtheblunt 4y ago

if comfortable answering, why protobuf instead of gob?

whacker 4y ago
proto predates gob by quite a bit. gob was introduced with golang, and it's not really used anywhere else.
- erik_seaberg 4y ago
  My takeaway from Java serialization (which became passé) was that a schema-driven encoding that’s supported in many languages is a lot more useful.
  - adra 4y ago
    Java serialization is passe largely because it has huge gaping holes in how one can abuse the serialization loaders to perform very unsafe security holes. And it's incredibly slow, and tied to JVMs, but sadly only really tied to JVM that have the exact same version and serialized objects, etc.. did I mention java serialization sucks? (Spoken by a true java lover that that subjected himself to serialization way more than he wanted to)
jhumphries131 4y ago
Gob is Go-specific. Our mission is to make schema-driven APIs easy, regardless of what language you use. Protobuf already has official support for nearly a dozen languages, and unofficial support for many more. Protobuf also has a compiler with a plugin model, which facilitates supporting even more in the future. Furthermore, Protobuf is an IDL, not a full-blown programming language. This makes it ideal for this use case, for describing APIs and data structures. Gob-encoded data structures are described with Go. While Go is great for writing server-side business logic, it is not as well-suited as a description language for data that you need to share with non-Go systems.
- jjtheblunt 4y ago