4 ms·

This is interesting. So one company invents and maintains a compiler, then a different company found that documentation to be insufficient (big surprise), so wr

This is interesting. So one company invents and maintains a compiler, then a different company found that documentation to be insufficient (big surprise), so wrote out a lengthy standard that conformed to what the first company's compiler happened to do? Seems very useful, but also seems risky and hard to maintain. What happens when Google tightens a constraint or adds a new feature next week?

mmastrac 4y ago
(deleted)
- CobrastanJorji 4y ago
  Ahhh, is that the play? That makes a lot of sense.
- jhumphries131 4y ago
  Matt, that is not our intention. But we are trying to build a business around making schema-driven APIs easy, and protobuf is at the core of our current products. So we are trying to improve the ecosystem around protobuf, and a critical aspect of that in our esteem is having a spec. While `protoc` remains the source of truth, this spec captures the syntax and rules accepted and enforced by `protoc` in a far more detailed way than the official developer site.
  - mmastrac 4y ago
    Ah, hey Josh. I didn't realize you were involved in this effort. FWIW, seeing your name associated with this definitely gives it a bit more authority than I had originally assumed.
    - kentonv 4y ago
      If it helps, I (original author of proto2) have been advising Buf and like what they're doing. (Disclosure: I also invested a small amount.) Buf is founded by engineers who spent a LOT of time working with Protobufs outside of Google. I was always the one saying "please don't write your own .proto parser" but I am convinced Buf actually knows what they are doing here and probably have all the details right. Our industry has a whole lot of tooling and infrastructure built around JSON, and almost every piece of it could be way better if it were operating with well-defined types instead, in the same way that TypeScript tooling benefits vs. JavaScript. Google has had an all-protobuf ecosystem internally for a long time, but much of it will never be released publicly. So that leaves someone like Buf to really build it out. I'm pretty interested to see where they're able to take it.
Master_Odin 4y ago
I think the hope is that this could be a situation like CommonMark and Markdown, where google's implementation will continue to exist, but that the community just totally moves over to this new specification / tooling, and that anytime someone says "protobuf", they don't even necessarily realize that Google has a thing, they just know this specification.
- numbsafari 4y ago
  ... except for whenever you've got to communicate or exist inside the Google ecosystem. Now you've got "protobuf as per Google" and "protobuf as per internet randos" with twice the dependency graph. It's unfortunate, and not surprising, that Google hasn't made a protobuf spec. I realize there are a ton of protobuf fans out there but, personally, it just feels like a massive, awkwardly maintained mess that I'm forced to live with for (mostly) their benefit.
- smcl 4y ago
  Trouble is that I don’t think everyone’s moved on with Markdown, I still encounter different dialects that are subtly different. Even within one suite of tools from a single vendor … [angry stares in the direction of Atlassian]
eklitzke 4y ago
Generally new features in protobuf are just new features, so if a new feature is added then the worst case is the documentation will be out of date/incomplete for a short period of time. For the most part the "constraint tightening" thing doesn't affect the language specification, at least in my experience. For example, there have been some changes in protobuf that affect things like serialization order. A change like that can break brittle tests that do things like checking that a function produces some exact serialized string/byte sequence, but they don't affect the semantics of the language.
jhumphries131 4y ago
We keep an eye on the protobuf repo, so if a change is made we can both incorporate it into our products (https://buf.build https://buf.build) and into this spec. A great outcome for the ecosystem would be that Google chooses to engage with the community before making language changes. And the best possible outcome would be that the authority is eventually inverted: a specification doc becomes the definitive source of truth on the language and `protoc` is updated to conform to it, instead of vice versa.
wrs 4y ago
This is a similar situation to Ruby and Rubinius (an alternative implementation of Ruby). Because there was no Ruby specification other than the original MRI implementation, the Rubinius project (a new alternative implementation) created their own test suite to codify expected Ruby behavior. However, the MRI developers didn't use it, and the behavior diverged. The original creator gave up on the idea [0] but it was immediately taken over by others and is still maintained [1]. In case of conflict, though, Matz (lead developer on MRI), not the specification, is the source of truth [2]. [0] https://github.com/rubinius/rubinius-website-archive/blob/87fe9d13c39789ecd9140bfd825e745c6026e7b5/_posts/2014-12-31-matz-s-ruby-developers-don-t-use-rubyspec.markdown https://github.com/rubinius/rubinius-website-archive/blob/87... [1] https://github.com/ruby/spec https://github.com/ruby/spec [2] http://ruby.github.io/rubyspec.github.io/bugs_found/ http://ruby.github.io/rubyspec.github.io/bugs_found/

mmastrac 4y ago

(deleted)

CobrastanJorji 4y ago
Ahhh, is that the play? That makes a lot of sense.
jhumphries131 4y ago
Matt, that is not our intention. But we are trying to build a business around making schema-driven APIs easy, and protobuf is at the core of our current products. So we are trying to improve the ecosystem around protobuf, and a critical aspect of that in our esteem is having a spec. While `protoc` remains the source of truth, this spec captures the syntax and rules accepted and enforced by `protoc` in a far more detailed way than the official developer site.
- mmastrac 4y ago
  Ah, hey Josh. I didn't realize you were involved in this effort. FWIW, seeing your name associated with this definitely gives it a bit more authority than I had originally assumed.
  - kentonv 4y ago
    If it helps, I (original author of proto2) have been advising Buf and like what they're doing. (Disclosure: I also invested a small amount.) Buf is founded by engineers who spent a LOT of time working with Protobufs outside of Google. I was always the one saying "please don't write your own .proto parser" but I am convinced Buf actually knows what they are doing here and probably have all the details right. Our industry has a whole lot of tooling and infrastructure built around JSON, and almost every piece of it could be way better if it were operating with well-defined types instead, in the same way that TypeScript tooling benefits vs. JavaScript. Google has had an all-protobuf ecosystem internally for a long time, but much of it will never be released publicly. So that leaves someone like Buf to really build it out. I'm pretty interested to see where they're able to take it.

Master_Odin 4y ago

I think the hope is that this could be a situation like CommonMark and Markdown, where google's implementation will continue to exist, but that the community just totally moves over to this new specification / tooling, and that anytime someone says "protobuf", they don't even necessarily realize that Google has a thing, they just know this specification.

numbsafari 4y ago
... except for whenever you've got to communicate or exist inside the Google ecosystem. Now you've got "protobuf as per Google" and "protobuf as per internet randos" with twice the dependency graph. It's unfortunate, and not surprising, that Google hasn't made a protobuf spec. I realize there are a ton of protobuf fans out there but, personally, it just feels like a massive, awkwardly maintained mess that I'm forced to live with for (mostly) their benefit.
smcl 4y ago
Trouble is that I don’t think everyone’s moved on with Markdown, I still encounter different dialects that are subtly different. Even within one suite of tools from a single vendor … [angry stares in the direction of Atlassian]

eklitzke 4y ago

Generally new features in protobuf are just new features, so if a new feature is added then the worst case is the documentation will be out of date/incomplete for a short period of time. For the most part the "constraint tightening" thing doesn't affect the language specification, at least in my experience. For example, there have been some changes in protobuf that affect things like serialization order. A change like that can break brittle tests that do things like checking that a function produces some exact serialized string/byte sequence, but they don't affect the semantics of the language.

jhumphries131 4y ago

We keep an eye on the protobuf repo, so if a change is made we can both incorporate it into our products (https://buf.build https://buf.build) and into this spec. A great outcome for the ecosystem would be that Google chooses to engage with the community before making language changes. And the best possible outcome would be that the authority is eventually inverted: a specification doc becomes the definitive source of truth on the language and `protoc` is updated to conform to it, instead of vice versa.

wrs 4y ago

This is a similar situation to Ruby and Rubinius (an alternative implementation of Ruby). Because there was no Ruby specification other than the original MRI implementation, the Rubinius project (a new alternative implementation) created their own test suite to codify expected Ruby behavior. However, the MRI developers didn't use it, and the behavior diverged. The original creator gave up on the idea [0] but it was immediately taken over by others and is still maintained [1]. In case of conflict, though, Matz (lead developer on MRI), not the specification, is the source of truth [2]. [0] https://github.com/rubinius/rubinius-website-archive/blob/87fe9d13c39789ecd9140bfd825e745c6026e7b5/_posts/2014-12-31-matz-s-ruby-developers-don-t-use-rubyspec.markdown https://github.com/rubinius/rubinius-website-archive/blob/87... [1] https://github.com/ruby/spec https://github.com/ruby/spec [2] http://ruby.github.io/rubyspec.github.io/bugs_found/ http://ruby.github.io/rubyspec.github.io/bugs_found/