11 ms·

The Protobuf Language Specification

194 points by akshayshah 4y ago

mmastrac 4y ago
(removing my unfair characterization)
- Master_Odin 4y ago
  A large corporate sponsor that has done a terrible job of shepherding the protocol, maintaining docs, etc. I'm all for a community push to divorce the protobuf specification/implementation from Google and to have it be much more community maintained, as it's clear that Google doesn't seem to care to.
- jhumphries131 4y ago
  Google actually requested that the community contribute to a real specification. https://github.com/protocolbuffers/protobuf/issues/6188#issuecomment-497885237 https://github.com/protocolbuffers/protobuf/issues/6188#issu... So we've taken the initiative. No rent-seeking.
  - prashantv 4y ago
    bufdev brought up some good concerns on that issue: > This as a community project defeats the purpose of creating a definitive grammar/spec that protocolbuffers/protobuf holds itself to - without that component, having a community-written spec is no improvement over the current situation from what I can see, and in fact could create even more fragmentation and confusion, just my two cents. It looks like the protobuf team created a repo intended for this (https://github.com/protocolbuffers/protobuf-grammar https://github.com/protocolbuffers/protobuf-grammar), perhaps the spec could be contributed there, including a review/blessing from the official team.
CobrastanJorji 4y ago
This is interesting. So one company invents and maintains a compiler, then a different company found that documentation to be insufficient (big surprise), so wrote out a lengthy standard that conformed to what the first company's compiler happened to do? Seems very useful, but also seems risky and hard to maintain. What happens when Google tightens a constraint or adds a new feature next week?
- mmastrac 4y ago
  (deleted)
  - CobrastanJorji 4y ago
    Ahhh, is that the play? That makes a lot of sense.
  - jhumphries131 4y ago
    Matt, that is not our intention. But we are trying to build a business around making schema-driven APIs easy, and protobuf is at the core of our current products. So we are trying to improve the ecosystem around protobuf, and a critical aspect of that in our esteem is having a spec. While `protoc` remains the source of truth, this spec captures the syntax and rules accepted and enforced by `protoc` in a far more detailed way than the official developer site.
    - mmastrac 4y ago
      Ah, hey Josh. I didn't realize you were involved in this effort. FWIW, seeing your name associated with this definitely gives it a bit more authority than I had originally assumed.
      - kentonv 4y ago
        If it helps, I (original author of proto2) have been advising Buf and like what they're doing. (Disclosure: I also invested a small amount.) Buf is founded by engineers who spent a LOT of time working with Protobufs outside of Google. I was always the one saying "please don't write your own .proto parser" but I am convinced Buf actually knows what they are doing here and probably have all the details right. Our industry has a whole lot of tooling and infrastructure built around JSON, and almost every piece of it could be way better if it were operating with well-defined types instead, in the same way that TypeScript tooling benefits vs. JavaScript. Google has had an all-protobuf ecosystem internally for a long time, but much of it will never be released publicly. So that leaves someone like Buf to really build it out. I'm pretty interested to see where they're able to take it.
jeffparsons 4y ago
> As of today, Protobuf is now a fully-defined language: (Etc.) I'm not sure what this is meant to achieve in reality. There is still only one implementation that defines what the language actually is, and that is Google's protoc. I'm my experience working with and writing alternative parsers for the '.proto' language, I've found that time and again Google's documentation for the format is either woefully vague, or directly contradicts the actual implementation. I don't see the value in a third party "spec" if what I have to do in practice will always be "whatever Google did in protoc".
- jhumphries131 4y ago
  The intent of this spec is to actually put "whatever Google did in protoc" into a readable format, so you don't have to read the C++ code. The official docs fall short on providing much of the details that are included.
  - lhorie 4y ago
    That sounds great, but what's the governance story? Are the authors of the spec document committing to keeping the document up to date here henceforth? Are the protoc maintainers committing to have these folks involved in project direction decisions?
    - jhumphries131 4y ago
      As of right now, the former. That is currently required for our tools to remain functioning (https://buf.build https://buf.build). If/when changes are made to the language, we update our tools (and this spec) to continue to be accurate.
  - morelisp 4y ago
    Protobuf isn't too complicated, I've found the wire format docs to be some of the best among the avro/msgpack/thrift/etc competitors. Maybe you mean something besides the wire format. In that case, good luck, because that shit ain't protobuf.
jupp0r 4y ago
> Protobuf is the most stable and widely adopted IDL today I've run into this misconception so many times over the last decade. Protobuf is much less than an IDL (intentionally so). It's used to describe the data of an interface but is completely unopinionated about all other aspects of an IDL. GRPC is a great example of how to use ProtoBuf in an IDL, but it could be used for other categories of interfaces (object oriented, etc). People treating ProtoBuf as an IDL make the mistake of concentrating too much on the data format and not about (imho) more important aspects like pre and postconditions etc, that make an interface an interface.
marsven_422 4y ago
advisedwang 4y ago
Is there a license on this spec?
- akshayshah 4y ago
  Apache 2.0: https://github.com/bufbuild/protobuf.com/blob/main/LICENSE https://github.com/bufbuild/protobuf.com/blob/main/LICENSE
IshKebab 4y ago
A good reference. I don't think it was really needed in the same way that e.g. a JSON or C++ spec was, since the language is so simple there's not much room for ambiguity. Definitely nice to have anyway.
- jhumphries131 4y ago
  It is a simple language, being just an IDL (no expressions, no logic, no state, no memory model, etc). However, you might be surprised about the room for ambiguity. For example, there are several mistakes in the grammars on the official developer site. And there is no place on the developer site that, for example, clearly explains how option names are formulated and interpreted. Even the way that relative references are resolved is not thoroughly described; it is probably the most complicated part of the spec because coherence/consistency wasn't keenly considered when the reference implementation in protoc was devised (https://www.protobuf.com/docs/language-spec#relative-references https://www.protobuf.com/docs/language-spec#relative-referen...). So if you wanted to write a tool for the language, one that could correctly parse and understand a source file, without the details in this spec that tool will almost certainly be incorrect and disagree with how protoc parses and understands the same source.
  - IshKebab 4y ago
    It does make me hope that some protobuf libraries will integrate their own compilers. The official one can be a bit of a pain to install. This will definitely make that easier!
jvolkman 4y ago
This seems like a great resource. Kudos. > But most of them are based on the incomplete specs from Google's developer site. None of them can correctly predict what source files protoc will actually accept or reject 100% of the time. I'd like to think I got pretty close with the plugin that now ships with IntelliJ. It even supports the 65-bit integer literal [1] that protoc happens to accept for proto2-style float and double default values. With this as a starting point, it'd be nice to fix some of the pecularities that arise from "implementation as spec", such as that literal value, and the fact that colon optionality in text format is based on value type, not syntax. 1: https://github.com/jvolkman/intellij-protobuf-editor/blob/6e1f946ef24b19b97a777c7eb0f07d67dcbefb09/core/src/main/java/idea/plugin/protoeditor/lang/psi/ProtoNumberValue.java#L234-L243 https://github.com/jvolkman/intellij-protobuf-editor/blob/6e...
- miohtama 4y ago
  Out of curiosity, why does Protobuf allow a negative 64-bit value in the first place? No CPU architecture supports such as far as I know.
  - kentonv 4y ago
    It's been 15 years since I wrote this so I don't remember exactly, but I think it's just an implementation quirk. See: https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/compiler/parser.cc#L1321 https://github.com/protocolbuffers/protobuf/blob/main/src/go... The parser consumes a "-", then consumes a number. The number can be any 64-bit unsigned integer. It is then converted to a double. Finally, if a "-" was seen earlier, it is negated. So by accident, it ends up allowing the range of a 65-bit signed integer.
  - jvolkman 4y ago
    It's just a long-standing quirk in the parser. It parses the numeric part as an unsigned 64-bit number, then applies the sign afterwards. And the result can be approximately stuffed into a floating point value. The behavior for integer fields is different; compilation will fail with an out of range error.
peteradio 4y ago
Never having had the opportunity to put protobuf into action but having some interest I've had these questions: 1) What would you say is the best use case 2) and what unfortunate misusecases have you come across
- jhumphries131 4y ago
  The key "killer" use case is for describing RPC schemas. By describing the schema in an IDL, you can then generate client stubs and server interfaces in a variety of implementation languages, allowing interop between heterogenous clients and servers. They are also useful for describing domain models. This isn't surprising since domain models usually find their way into RPC schemas (since RPCs will often query or define model data). But they can also be used in other cases, such as for persistence and structured logging. Some misuse I have seen involves trying to make a protobuf model the _only_ representation of a domain model: it is almost inevitable that a physical model (a representation of your data in a SQL database, for example) will need to vary from a logical model, and trying to make a single protobuf representation serve double duty can be a source of problems. Making protobuf schemas conform to physical storage constraints often makes for worse abstraction. The model becomes de-normalized (which can make constraints and relationships harder to model/enforce), and can even leak details that consumers shouldn't know about or care about. Another misuse is using it for data that never leaves a process -- a program's private, internal state. If a data structure never needs to be serialized (to persist or send to another process), then you're better off using native data structures in the implementation language (which generally have far greater flexibility/expressibility as well as better performance).
- shaftway 4y ago
  I do most of my work with protos in Java, and it's nice to have a schema like this that will build a bunch of immutable POJOs with builder classes and enough infrastructure to be able to do some interesting reflection-style stuff on top of the serialization / deserialization. The wide variety of client languages is really nice. I'm fairly certain that I can parse a proto in any language I'm ever going to use. The binary wire format is fairly straightforward, and is pretty tight without using compression. Fields are byte-aligned, and if you wanted to generate a binary proto message by concatenating a few things together it isn't very hard. And then you can use your proto definitions in whatever language you want to parse it. You can even parse your proto definition into a proto (Google provides the proto proto definitions, I think I got that grammar right) and write tools that generate whatever code you want easily. I think the text format is under-utilized. It's my go-to for configuration files. You create the proto definitions with whatever structures you want to structure your config settings, and then use the official parser to parse a text file. It supports comments (I'm throwing shade at you, JSON), and is simpler than YAML, while adding that structure. You can also use command line tools to validate the file as a pre-commit, or translate it into a binary format if you don't want to rely on the text format.
kyrra 4y ago
Googler, opinions are my own. I don't work on protobuf at all, just use it all the time (like most Googlers) I haven't dug into this in great detail yet, but the hard thing about the proto "spec" is that there isn't one, and protoc lets you do all kinds of crazy things that are really hard to model in languages like Antlr. There were some poor choices in protoc dating back to when proto1 was first created that have been carried forward. Having 20 years of proto definitions lets people come up with some crazy use cases. Definitely interesting for this company to create an EBNF definition for protobuf.
- jhumphries131 4y ago
  It's possible that the internal version of protoc is very different from the open-source version. (I know there are numerous differences, but not sure how pervasive they are in the parser.) The open-source version has a hand-written tokenizer and recursive descent parser that is not too difficult to translate to EBNF. You'll notice that the section on numeric literals is a little wonky, because the tokenizer does a check that is hard to describe in EBNF. But it isn't too bad. Also, some of the constraints of the language are in prose in this spec because they are easier to enforce using a semantic validation pass, instead of trying to model purely with a CFG. (Optionality of the colon in the text format, used in message literals, comes to mind.) There are some things that technically _could_ be handled in the grammar, but they would make the grammar much more cumbersome to read and understand. So those things are also extracted into prose. > Definitely interesting for this company to create an EBNF definition for protobuf. For what it's worth, Google has also published an EBNF definition (the subject blog post contains links to those specs). But they are incomplete and not entirely accurate, which is a non-trivial part of what led us to writing and publishing this spec.
  - kyrra 4y ago
    One place protoc doesn't align well is the descriptor object. https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.descriptor https://developers.google.com/protocol-buffers/docs/referenc... Comment placement is basically allowed anywhere by protoc, but how to get those comments within a Descriptor object for a proto is not well defined (there are places where you can put comments that are not available within Descriptor). It provides leading/trailing comments, but there are many other cases that are missed today (like comments embedded within a list of items in an array). Maybe this is a mismatch between what protoc allows and what Descriptor presents, but it's definitely annoying.
silasdavis 4y ago
Golang is an example of a language defined by a spec not an implementation? Discuss.
- wrs 4y ago
  I see your point, but the implementers of Go do pay a lot more attention than most "defined by an implementation" languages to specifying what they're doing before they do it. And if you find a difference between the specification and the implementation, generally the specification will prevail.
season2episode3 4y ago
It appears Google has released a spec as of 11 days ago: https://github.com/protocolbuffers/protobuf/issues/6188#issuecomment-1234641108 https://github.com/protocolbuffers/protobuf/issues/6188#issu...
- bufbuild 4y ago
  That is for the text format, which is a serialized representation of Protobuf data. As they specify in the linked doc, it is not the format for the actual language: > This format is distinct from the format of text within a .proto schema.
rigelbm 4y ago
Echoing some of the sentiment here: although this was certainly a great effort and the result is awesome, that is NOT The Protobuf Language Specification, for as long as the maintainers of the Protobuf (protoc) project don't agree to follow it. This is certainly The Buf Language Specification, which is useful in itself. Specs are like contracts. If I were to build a tool to be compatible with Buf, I would definitely aim it to work with this spec. The problem is that the Protobuf project simply didn't sign this contract. Whatever it says is, sorry for the choice of word, a bit pointless if I'm trying to build a tool compatible with Protobuf, specially around forward compatibility. The industry does need better tooling around protobuf/efficient RPC, and being dependent on a single company (i.e. Google) is definitely not healthy. I hope you guys succeed in what your are trying to doing.
- rigelbm 4y ago
  Actually, I thought about it twice and I retract what I said about this specification being pointless for building tools compatible with protobuf. Reasons: * The language itself is unlikely to change much given it's been public for so long. A non-official spec that captures the current implementation is probably going to survive for some time. * There's no official spec (which I would prefer) for me to base my tool on. This spec is about my only choice. The more tools targeting this spec, the hardest would be for Google to break compatibility with it, reinforcing my previous point. I will keep the parent comment for context, and I don't retract the fact that I think the title is misleading. Otherwise, great work!!
  - staticassertion 4y ago
    Agreed. If IDEs and alternative compilers are all building off of the spec because it's the path of least resistance, and there are no bugs for a while, the defacto standard impl is going to face serious scrutiny for parting from it. And, as you said, proto isn't in a great position to be making crazy changes anyway.
- mook 4y ago
  That's basically the equivalent of RubySpec — reverse engineered from MRI (the original Ruby implementation) for use by Rubinius. It was adopted by the other alternative Ruby implementations too. It looks like the original is now gone, but a fork has taken over. Looking at some comments, fighting to get MRI to adopt it may have burnt out the people behind it.
habitue 4y ago
> we are standing on the shoulders of giants, those who have built and battle-tested it, and brought it to its current mature state I would rewrite this maybe to: > we are making Google's internal problems into everyone's problems There are benefits to an IDL in the abstract, but an IDL for everyone should be built with the benefit of hindsight looking at the lessons of protobuf, ion, thrift, etc. Not just baking Google's internal backwards compatibility obligations into a formal spec everyone should follow. I think any time google takes an internal tool and flips the "open source" bit on it, it turns out to be a bad match for the rest of the world. When they instead take the time to build a new system that learns from the internal tool, like Kubernetes learned from Borg, I think the end result is significantly more valuable.
- orf 4y ago
  I quite like Protobuf definitions. I find them very easy to read and I love the fact I can distribute them to a bunch of different languages via a library. Are these Google’s internal problems? Or, what google-internal problems do protobufs solve that nobody else needs to care about? Edit: to your edit, I find it hard to see a different way to do things.
  - vitiral 4y ago
    A different way: learn from protobufs, then design a new language from the ground up and define a spec very early.
    - saalweachter 4y ago
      Eh, I've been corrupted by a decade of working with protobufs, but I've ultimately settled on "protos are good enough". Maybe not perfect, maybe there are use cases and patterns where they totally suck, but they exist and are already being used and they generally get the job done. While a part of me will always want to spend time optimizing and carefully crafting the perfect system from the bits up, there are more interesting things to build on top of your serialization and RPC frameworks so I don't see any real value in spending more time creating a 15th standard.
cpurdy 4y ago
oh .. cool .. pricing for protobuf
elcritch 4y ago
Lately I've been reading up on Amazon's Ion format and think it's underappreciated. It's got a decent spec for both binary and text serialization formats, under an Apache-2.0 license even. Maybe I'm weird but I just don't like using serialization formats that don't include a specification for the binary format. I also like where Ion goes with it's symbol tables. They provide much of the space efficiency that Protobufs provide with schemas, but gives you much more freedom and flexibility. I'm a longtime fan of msgpack but it becomes inefficient if you transport field names with it. The Ion schema also seems nice as an alternative to Protobuf schemas, but able to describe arbitrary data constraints too.
Cloudef 4y ago
Here is protobuf parser I wrote for a client w/ ragel/c++. We needed way to parse the proto specification to generate our own code, since the google's own C++ runtimes were way too huge for our use case especially with reflection. The nanopb in addition to the code generation w/ this parser was used in the end to keep the binary bloat in check. https://gist.github.com/Cloudef/7a69eba054f61ee63d6ee59262e1b695 https://gist.github.com/Cloudef/7a69eba054f61ee63d6ee59262e1... I also did this benchmark for cap'n'proto, flatbuffers, protobuf and raw structs. This is already years old so it might not be accurate anymore, but what surprised me most was how different flatbuffers was compared to raw struct, as the design sounds like it should be merely extension to what is essentially raw struct + making sure the data can be portably encoded and decoded. https://cloudef.pw/protobug.png https://cloudef.pw/protobug.png
elromulous 4y ago
Paging kentonv for his thoughts on this
- kentonv 4y ago
  Hello! See: https://news.ycombinator.com/item?id=32816319 https://news.ycombinator.com/item?id=32816319
rad_gruchalski 4y ago
Question: if there’s no ebnf protobuf spec, what’s this then? https://developers.google.com/protocol-buffers/docs/reference/proto3-spec https://developers.google.com/protocol-buffers/docs/referenc... The text at the top of the page says: > This is a language specification reference for version 3 of the Protocol Buffers language (proto3). The syntax is specified using Extended Backus-Naur Form (EBNF) …

194 points by akshayshah 4y ago

mmastrac 4y ago

(removing my unfair characterization)

Master_Odin 4y ago
A large corporate sponsor that has done a terrible job of shepherding the protocol, maintaining docs, etc. I'm all for a community push to divorce the protobuf specification/implementation from Google and to have it be much more community maintained, as it's clear that Google doesn't seem to care to.
jhumphries131 4y ago
Google actually requested that the community contribute to a real specification. https://github.com/protocolbuffers/protobuf/issues/6188#issuecomment-497885237 https://github.com/protocolbuffers/protobuf/issues/6188#issu... So we've taken the initiative. No rent-seeking.
- prashantv 4y ago
  bufdev brought up some good concerns on that issue: > This as a community project defeats the purpose of creating a definitive grammar/spec that protocolbuffers/protobuf holds itself to - without that component, having a community-written spec is no improvement over the current situation from what I can see, and in fact could create even more fragmentation and confusion, just my two cents. It looks like the protobuf team created a repo intended for this (https://github.com/protocolbuffers/protobuf-grammar https://github.com/protocolbuffers/protobuf-grammar), perhaps the spec could be contributed there, including a review/blessing from the official team.

CobrastanJorji 4y ago

This is interesting. So one company invents and maintains a compiler, then a different company found that documentation to be insufficient (big surprise), so wrote out a lengthy standard that conformed to what the first company's compiler happened to do? Seems very useful, but also seems risky and hard to maintain. What happens when Google tightens a constraint or adds a new feature next week?

mmastrac 4y ago
(deleted)
- CobrastanJorji 4y ago
  Ahhh, is that the play? That makes a lot of sense.
- jhumphries131 4y ago
  Matt, that is not our intention. But we are trying to build a business around making schema-driven APIs easy, and protobuf is at the core of our current products. So we are trying to improve the ecosystem around protobuf, and a critical aspect of that in our esteem is having a spec. While `protoc` remains the source of truth, this spec captures the syntax and rules accepted and enforced by `protoc` in a far more detailed way than the official developer site.
  - mmastrac 4y ago
    Ah, hey Josh. I didn't realize you were involved in this effort. FWIW, seeing your name associated with this definitely gives it a bit more authority than I had originally assumed.
    - kentonv 4y ago
      If it helps, I (original author of proto2) have been advising Buf and like what they're doing. (Disclosure: I also invested a small amount.) Buf is founded by engineers who spent a LOT of time working with Protobufs outside of Google. I was always the one saying "please don't write your own .proto parser" but I am convinced Buf actually knows what they are doing here and probably have all the details right. Our industry has a whole lot of tooling and infrastructure built around JSON, and almost every piece of it could be way better if it were operating with well-defined types instead, in the same way that TypeScript tooling benefits vs. JavaScript. Google has had an all-protobuf ecosystem internally for a long time, but much of it will never be released publicly. So that leaves someone like Buf to really build it out. I'm pretty interested to see where they're able to take it.

jeffparsons 4y ago

> As of today, Protobuf is now a fully-defined language: (Etc.) I'm not sure what this is meant to achieve in reality. There is still only one implementation that defines what the language actually is, and that is Google's protoc. I'm my experience working with and writing alternative parsers for the '.proto' language, I've found that time and again Google's documentation for the format is either woefully vague, or directly contradicts the actual implementation. I don't see the value in a third party "spec" if what I have to do in practice will always be "whatever Google did in protoc".

jhumphries131 4y ago
The intent of this spec is to actually put "whatever Google did in protoc" into a readable format, so you don't have to read the C++ code. The official docs fall short on providing much of the details that are included.
- lhorie 4y ago
  That sounds great, but what's the governance story? Are the authors of the spec document committing to keeping the document up to date here henceforth? Are the protoc maintainers committing to have these folks involved in project direction decisions?
  - jhumphries131 4y ago
    As of right now, the former. That is currently required for our tools to remain functioning (https://buf.build https://buf.build). If/when changes are made to the language, we update our tools (and this spec) to continue to be accurate.
- morelisp 4y ago
  Protobuf isn't too complicated, I've found the wire format docs to be some of the best among the avro/msgpack/thrift/etc competitors. Maybe you mean something besides the wire format. In that case, good luck, because that shit ain't protobuf.

jupp0r 4y ago

> Protobuf is the most stable and widely adopted IDL today I've run into this misconception so many times over the last decade. Protobuf is much less than an IDL (intentionally so). It's used to describe the data of an interface but is completely unopinionated about all other aspects of an IDL. GRPC is a great example of how to use ProtoBuf in an IDL, but it could be used for other categories of interfaces (object oriented, etc). People treating ProtoBuf as an IDL make the mistake of concentrating too much on the data format and not about (imho) more important aspects like pre and postconditions etc, that make an interface an interface.

marsven_422 4y ago

advisedwang 4y ago

Is there a license on this spec?

akshayshah 4y ago
Apache 2.0: https://github.com/bufbuild/protobuf.com/blob/main/LICENSE https://github.com/bufbuild/protobuf.com/blob/main/LICENSE

IshKebab 4y ago

A good reference. I don't think it was really needed in the same way that e.g. a JSON or C++ spec was, since the language is so simple there's not much room for ambiguity. Definitely nice to have anyway.

jhumphries131 4y ago
It is a simple language, being just an IDL (no expressions, no logic, no state, no memory model, etc). However, you might be surprised about the room for ambiguity. For example, there are several mistakes in the grammars on the official developer site. And there is no place on the developer site that, for example, clearly explains how option names are formulated and interpreted. Even the way that relative references are resolved is not thoroughly described; it is probably the most complicated part of the spec because coherence/consistency wasn't keenly considered when the reference implementation in protoc was devised (https://www.protobuf.com/docs/language-spec#relative-references https://www.protobuf.com/docs/language-spec#relative-referen...). So if you wanted to write a tool for the language, one that could correctly parse and understand a source file, without the details in this spec that tool will almost certainly be incorrect and disagree with how protoc parses and understands the same source.
- IshKebab 4y ago
  It does make me hope that some protobuf libraries will integrate their own compilers. The official one can be a bit of a pain to install. This will definitely make that easier!

jvolkman 4y ago

This seems like a great resource. Kudos. > But most of them are based on the incomplete specs from Google's developer site. None of them can correctly predict what source files protoc will actually accept or reject 100% of the time. I'd like to think I got pretty close with the plugin that now ships with IntelliJ. It even supports the 65-bit integer literal [1] that protoc happens to accept for proto2-style float and double default values. With this as a starting point, it'd be nice to fix some of the pecularities that arise from "implementation as spec", such as that literal value, and the fact that colon optionality in text format is based on value type, not syntax. 1: https://github.com/jvolkman/intellij-protobuf-editor/blob/6e1f946ef24b19b97a777c7eb0f07d67dcbefb09/core/src/main/java/idea/plugin/protoeditor/lang/psi/ProtoNumberValue.java#L234-L243 https://github.com/jvolkman/intellij-protobuf-editor/blob/6e...

miohtama 4y ago
Out of curiosity, why does Protobuf allow a negative 64-bit value in the first place? No CPU architecture supports such as far as I know.
- kentonv 4y ago
  It's been 15 years since I wrote this so I don't remember exactly, but I think it's just an implementation quirk. See: https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/compiler/parser.cc#L1321 https://github.com/protocolbuffers/protobuf/blob/main/src/go... The parser consumes a "-", then consumes a number. The number can be any 64-bit unsigned integer. It is then converted to a double. Finally, if a "-" was seen earlier, it is negated. So by accident, it ends up allowing the range of a 65-bit signed integer.
- jvolkman 4y ago
  It's just a long-standing quirk in the parser. It parses the numeric part as an unsigned 64-bit number, then applies the sign afterwards. And the result can be approximately stuffed into a floating point value. The behavior for integer fields is different; compilation will fail with an out of range error.

peteradio 4y ago

Never having had the opportunity to put protobuf into action but having some interest I've had these questions: 1) What would you say is the best use case 2) and what unfortunate misusecases have you come across

jhumphries131 4y ago
The key "killer" use case is for describing RPC schemas. By describing the schema in an IDL, you can then generate client stubs and server interfaces in a variety of implementation languages, allowing interop between heterogenous clients and servers. They are also useful for describing domain models. This isn't surprising since domain models usually find their way into RPC schemas (since RPCs will often query or define model data). But they can also be used in other cases, such as for persistence and structured logging. Some misuse I have seen involves trying to make a protobuf model the _only_ representation of a domain model: it is almost inevitable that a physical model (a representation of your data in a SQL database, for example) will need to vary from a logical model, and trying to make a single protobuf representation serve double duty can be a source of problems. Making protobuf schemas conform to physical storage constraints often makes for worse abstraction. The model becomes de-normalized (which can make constraints and relationships harder to model/enforce), and can even leak details that consumers shouldn't know about or care about. Another misuse is using it for data that never leaves a process -- a program's private, internal state. If a data structure never needs to be serialized (to persist or send to another process), then you're better off using native data structures in the implementation language (which generally have far greater flexibility/expressibility as well as better performance).
shaftway 4y ago
I do most of my work with protos in Java, and it's nice to have a schema like this that will build a bunch of immutable POJOs with builder classes and enough infrastructure to be able to do some interesting reflection-style stuff on top of the serialization / deserialization. The wide variety of client languages is really nice. I'm fairly certain that I can parse a proto in any language I'm ever going to use. The binary wire format is fairly straightforward, and is pretty tight without using compression. Fields are byte-aligned, and if you wanted to generate a binary proto message by concatenating a few things together it isn't very hard. And then you can use your proto definitions in whatever language you want to parse it. You can even parse your proto definition into a proto (Google provides the proto proto definitions, I think I got that grammar right) and write tools that generate whatever code you want easily. I think the text format is under-utilized. It's my go-to for configuration files. You create the proto definitions with whatever structures you want to structure your config settings, and then use the official parser to parse a text file. It supports comments (I'm throwing shade at you, JSON), and is simpler than YAML, while adding that structure. You can also use command line tools to validate the file as a pre-commit, or translate it into a binary format if you don't want to rely on the text format.

kyrra 4y ago

Googler, opinions are my own. I don't work on protobuf at all, just use it all the time (like most Googlers) I haven't dug into this in great detail yet, but the hard thing about the proto "spec" is that there isn't one, and protoc lets you do all kinds of crazy things that are really hard to model in languages like Antlr. There were some poor choices in protoc dating back to when proto1 was first created that have been carried forward. Having 20 years of proto definitions lets people come up with some crazy use cases. Definitely interesting for this company to create an EBNF definition for protobuf.

jhumphries131 4y ago
It's possible that the internal version of protoc is very different from the open-source version. (I know there are numerous differences, but not sure how pervasive they are in the parser.) The open-source version has a hand-written tokenizer and recursive descent parser that is not too difficult to translate to EBNF. You'll notice that the section on numeric literals is a little wonky, because the tokenizer does a check that is hard to describe in EBNF. But it isn't too bad. Also, some of the constraints of the language are in prose in this spec because they are easier to enforce using a semantic validation pass, instead of trying to model purely with a CFG. (Optionality of the colon in the text format, used in message literals, comes to mind.) There are some things that technically _could_ be handled in the grammar, but they would make the grammar much more cumbersome to read and understand. So those things are also extracted into prose. > Definitely interesting for this company to create an EBNF definition for protobuf. For what it's worth, Google has also published an EBNF definition (the subject blog post contains links to those specs). But they are incomplete and not entirely accurate, which is a non-trivial part of what led us to writing and publishing this spec.
- kyrra 4y ago
  One place protoc doesn't align well is the descriptor object. https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.descriptor https://developers.google.com/protocol-buffers/docs/referenc... Comment placement is basically allowed anywhere by protoc, but how to get those comments within a Descriptor object for a proto is not well defined (there are places where you can put comments that are not available within Descriptor). It provides leading/trailing comments, but there are many other cases that are missed today (like comments embedded within a list of items in an array). Maybe this is a mismatch between what protoc allows and what Descriptor presents, but it's definitely annoying.

silasdavis 4y ago

Golang is an example of a language defined by a spec not an implementation? Discuss.

wrs 4y ago
I see your point, but the implementers of Go do pay a lot more attention than most "defined by an implementation" languages to specifying what they're doing before they do it. And if you find a difference between the specification and the implementation, generally the specification will prevail.

season2episode3 4y ago

It appears Google has released a spec as of 11 days ago: https://github.com/protocolbuffers/protobuf/issues/6188#issuecomment-1234641108 https://github.com/protocolbuffers/protobuf/issues/6188#issu...

bufbuild 4y ago
That is for the text format, which is a serialized representation of Protobuf data. As they specify in the linked doc, it is not the format for the actual language: > This format is distinct from the format of text within a .proto schema.

rigelbm 4y ago

Echoing some of the sentiment here: although this was certainly a great effort and the result is awesome, that is NOT The Protobuf Language Specification, for as long as the maintainers of the Protobuf (protoc) project don't agree to follow it. This is certainly The Buf Language Specification, which is useful in itself. Specs are like contracts. If I were to build a tool to be compatible with Buf, I would definitely aim it to work with this spec. The problem is that the Protobuf project simply didn't sign this contract. Whatever it says is, sorry for the choice of word, a bit pointless if I'm trying to build a tool compatible with Protobuf, specially around forward compatibility. The industry does need better tooling around protobuf/efficient RPC, and being dependent on a single company (i.e. Google) is definitely not healthy. I hope you guys succeed in what your are trying to doing.

rigelbm 4y ago
Actually, I thought about it twice and I retract what I said about this specification being pointless for building tools compatible with protobuf. Reasons: * The language itself is unlikely to change much given it's been public for so long. A non-official spec that captures the current implementation is probably going to survive for some time. * There's no official spec (which I would prefer) for me to base my tool on. This spec is about my only choice. The more tools targeting this spec, the hardest would be for Google to break compatibility with it, reinforcing my previous point. I will keep the parent comment for context, and I don't retract the fact that I think the title is misleading. Otherwise, great work!!
- staticassertion 4y ago
  Agreed. If IDEs and alternative compilers are all building off of the spec because it's the path of least resistance, and there are no bugs for a while, the defacto standard impl is going to face serious scrutiny for parting from it. And, as you said, proto isn't in a great position to be making crazy changes anyway.
mook 4y ago
That's basically the equivalent of RubySpec — reverse engineered from MRI (the original Ruby implementation) for use by Rubinius. It was adopted by the other alternative Ruby implementations too. It looks like the original is now gone, but a fork has taken over. Looking at some comments, fighting to get MRI to adopt it may have burnt out the people behind it.

habitue 4y ago

> we are standing on the shoulders of giants, those who have built and battle-tested it, and brought it to its current mature state I would rewrite this maybe to: > we are making Google's internal problems into everyone's problems There are benefits to an IDL in the abstract, but an IDL for everyone should be built with the benefit of hindsight looking at the lessons of protobuf, ion, thrift, etc. Not just baking Google's internal backwards compatibility obligations into a formal spec everyone should follow. I think any time google takes an internal tool and flips the "open source" bit on it, it turns out to be a bad match for the rest of the world. When they instead take the time to build a new system that learns from the internal tool, like Kubernetes learned from Borg, I think the end result is significantly more valuable.

orf 4y ago
I quite like Protobuf definitions. I find them very easy to read and I love the fact I can distribute them to a bunch of different languages via a library. Are these Google’s internal problems? Or, what google-internal problems do protobufs solve that nobody else needs to care about? Edit: to your edit, I find it hard to see a different way to do things.
- vitiral 4y ago
  A different way: learn from protobufs, then design a new language from the ground up and define a spec very early.
  - saalweachter 4y ago
    Eh, I've been corrupted by a decade of working with protobufs, but I've ultimately settled on "protos are good enough". Maybe not perfect, maybe there are use cases and patterns where they totally suck, but they exist and are already being used and they generally get the job done. While a part of me will always want to spend time optimizing and carefully crafting the perfect system from the bits up, there are more interesting things to build on top of your serialization and RPC frameworks so I don't see any real value in spending more time creating a 15th standard.

cpurdy 4y ago

oh .. cool .. pricing for protobuf

elcritch 4y ago

Lately I've been reading up on Amazon's Ion format and think it's underappreciated. It's got a decent spec for both binary and text serialization formats, under an Apache-2.0 license even. Maybe I'm weird but I just don't like using serialization formats that don't include a specification for the binary format. I also like where Ion goes with it's symbol tables. They provide much of the space efficiency that Protobufs provide with schemas, but gives you much more freedom and flexibility. I'm a longtime fan of msgpack but it becomes inefficient if you transport field names with it. The Ion schema also seems nice as an alternative to Protobuf schemas, but able to describe arbitrary data constraints too.

Cloudef 4y ago

Here is protobuf parser I wrote for a client w/ ragel/c++. We needed way to parse the proto specification to generate our own code, since the google's own C++ runtimes were way too huge for our use case especially with reflection. The nanopb in addition to the code generation w/ this parser was used in the end to keep the binary bloat in check. https://gist.github.com/Cloudef/7a69eba054f61ee63d6ee59262e1b695 https://gist.github.com/Cloudef/7a69eba054f61ee63d6ee59262e1... I also did this benchmark for cap'n'proto, flatbuffers, protobuf and raw structs. This is already years old so it might not be accurate anymore, but what surprised me most was how different flatbuffers was compared to raw struct, as the design sounds like it should be merely extension to what is essentially raw struct + making sure the data can be portably encoded and decoded. https://cloudef.pw/protobug.png https://cloudef.pw/protobug.png

elromulous 4y ago

Paging kentonv for his thoughts on this

kentonv 4y ago
Hello! See: https://news.ycombinator.com/item?id=32816319 https://news.ycombinator.com/item?id=32816319

rad_gruchalski 4y ago

Question: if there’s no ebnf protobuf spec, what’s this then? https://developers.google.com/protocol-buffers/docs/reference/proto3-spec https://developers.google.com/protocol-buffers/docs/referenc... The text at the top of the page says: > This is a language specification reference for version 3 of the Protocol Buffers language (proto3). The syntax is specified using Extended Backus-Naur Form (EBNF) …