3 ms·

Protobuf 3 was bending over backwards to try to make the Go API make sense, but in the process it screwed up the API for C++, with many compromises. Then they c

Protobuf 3 was bending over backwards to try to make the Go API make sense, but in the process it screwed up the API for C++, with many compromises. Then they changed course and made presence explicit again in proto 3.1. Now they are saying Go gets a C++-like API. What I'd like is to rewind the time machine and undo all the path-dependent brain damage.

ein0p 2y ago
Nice to see my comments on their proto3 design doc vindicated, lol. There were a lot of comments on that doc, far more than what you'd usually see. Some of those comments dealt with the misguided decision to basically drop nullability (that is, the `has_` methods) that proto2 had. The team then just deleted all the comments and disabled commenting on the doc and proceeded with their original design much to the consternation of their primary stakeholders.
deleted 2y ago
[deleted]
sa46 2y ago
When I was at Google around 2016, there was a significant push to convince folks that the proto3 implicit presence was superior to explicit presence. Is there a design doc with the rationale for switching back to explicit presence for Edition 2023? The closest docs I've found are https://buf.build/blog/protobuf-editions-are-here https://buf.build/blog/protobuf-editions-are-here and https://github.com/protocolbuffers/protobuf/tree/main/docs/design/editions https://github.com/protocolbuffers/protobuf/tree/main/docs/d....
- jeffbee 2y ago
  I was only there for the debate you mentioned and not there for the reversal, so I dunno.
  - IX-103 2y ago
    I wasn't there for the debate, but was there for the reversal. I don't remember there being anything explicitly said about it. The only thing I can think of is that I know of some important projects that couldn't migrate to proto3 because of this implicit field issue. So some people were still writing new code with proto2.
    - summerlight 2y ago
      And... most of the important projects are still using proto2 and there is no realistic way to do interop between those two formats. IIRC, you cannot use proto2 enum in proto3 for some obscure technical reason! This creates a backsliding issue; you cannot migrate old protos with thousands of different messages, fields and enums where new proto2 messages to use them are added everyday. This is an important distinction from the proto1->proto2 migration, which was in a much better compatibility situation yet still took years to complete. AFAIK, this is the main reason why the proto team decided to create a superset approach (edition) so migration can be handled as a flag at each message level.
- akshayshah 2y ago
  Best bet is likely https://github.com/protocolbuffers/protobuf/blob/main/docs/field_presence.md https://github.com/protocolbuffers/protobuf/blob/main/docs/f..., which predates editions.
jcdavis 2y ago
> it screwed up the API for C++, with many compromises The implicit presence garbage screwed up the API for many languages, not just C++ What is wild is how obviously silly it was at the time, too - no hindsight was needed.
- kubb 2y ago
  It was but when the wrong fool gets a say, they will mess a perfectly good thing up for everyone. Organizations often promote fools who don’t second guess their beliefs and think they have it all figured out.
dekhn 2y ago
I work mainly in Python, it's always seemed really bad that there are 3 main implementations of Protobufs, instead of the C++ being the real implementation and other platforms just dlopen'ing and using it (there are a million software engineering arguments around this; I've heard them all before, have my own opinions, and have listened to the opinions of people I disagree with). It seems like the velocity of a project is the reciprocal of the number of independent implementations of a spec because any one of the implementations can slow down all the implementations (like what happened with proto3 around required and optional). From what I can tell, a major source of the problem was that protobuf field semantics were absolutely critical to the scaling of google in the early days (as an inter-server protocol for rapidly evolving things like the search stack), but it's also being used as a data modelling toolkit (as a way of representing data with a high level of fidelity). And those two groups- along with the multiple language developers who don't want to deal with native code- do not see eye to eye, and want to drive the spec in their preferred direction. (FWIW nowadays I use pydantic for type descriptions and JSON for transport, but I really prefer having an external IDL unrelated to any specific programming language)
- charleslmunger 2y ago
  You'll be thrilled to hear about upb then, which was designed to be embeddable to power other languages without a from-scratch implementation - and now powers python protos. https://github.com/protocolbuffers/protobuf/tree/main/upb https://github.com/protocolbuffers/protobuf/tree/main/upb
- lmm 2y ago
  > It seems like the velocity of a project is the reciprocal of the number of independent implementations of a spec because any one of the implementations can slow down all the implementations (like what happened with proto3 around required and optional). Velocity and stability/maturity are in tension, sure. I think for a foundational protocol like protobuf you want the stability and reliability that come from multiple independent implementations more than you want it to be moving fast and breaking things.
  - elcritch 2y ago
sbrother 2y ago
I still use proto2 if possible. The syntactic sugar around `oneof` wasn't nice enough to merit dealing with proto3's implicit presence -- maybe it is just because I learned proto2 with C++ and don't use Go, but proto3 just seemed like a big step back and introduced footguns that weren't there before. Happy to hear they are reverting some of those finally.
boulos 2y ago
That's not how I remember it. I thought proto3 was all about JSON compatibility. No?

ein0p 2y ago

Nice to see my comments on their proto3 design doc vindicated, lol. There were a lot of comments on that doc, far more than what you'd usually see. Some of those comments dealt with the misguided decision to basically drop nullability (that is, the `has_` methods) that proto2 had. The team then just deleted all the comments and disabled commenting on the doc and proceeded with their original design much to the consternation of their primary stakeholders.

deleted 2y ago

[deleted]

sa46 2y ago

When I was at Google around 2016, there was a significant push to convince folks that the proto3 implicit presence was superior to explicit presence. Is there a design doc with the rationale for switching back to explicit presence for Edition 2023? The closest docs I've found are https://buf.build/blog/protobuf-editions-are-here https://buf.build/blog/protobuf-editions-are-here and https://github.com/protocolbuffers/protobuf/tree/main/docs/design/editions https://github.com/protocolbuffers/protobuf/tree/main/docs/d....

jeffbee 2y ago
I was only there for the debate you mentioned and not there for the reversal, so I dunno.
- IX-103 2y ago
  I wasn't there for the debate, but was there for the reversal. I don't remember there being anything explicitly said about it. The only thing I can think of is that I know of some important projects that couldn't migrate to proto3 because of this implicit field issue. So some people were still writing new code with proto2.
  - summerlight 2y ago
    And... most of the important projects are still using proto2 and there is no realistic way to do interop between those two formats. IIRC, you cannot use proto2 enum in proto3 for some obscure technical reason! This creates a backsliding issue; you cannot migrate old protos with thousands of different messages, fields and enums where new proto2 messages to use them are added everyday. This is an important distinction from the proto1->proto2 migration, which was in a much better compatibility situation yet still took years to complete. AFAIK, this is the main reason why the proto team decided to create a superset approach (edition) so migration can be handled as a flag at each message level.
akshayshah 2y ago
Best bet is likely https://github.com/protocolbuffers/protobuf/blob/main/docs/field_presence.md https://github.com/protocolbuffers/protobuf/blob/main/docs/f..., which predates editions.

jcdavis 2y ago

> it screwed up the API for C++, with many compromises The implicit presence garbage screwed up the API for many languages, not just C++ What is wild is how obviously silly it was at the time, too - no hindsight was needed.

kubb 2y ago
It was but when the wrong fool gets a say, they will mess a perfectly good thing up for everyone. Organizations often promote fools who don’t second guess their beliefs and think they have it all figured out.

dekhn 2y ago

I work mainly in Python, it's always seemed really bad that there are 3 main implementations of Protobufs, instead of the C++ being the real implementation and other platforms just dlopen'ing and using it (there are a million software engineering arguments around this; I've heard them all before, have my own opinions, and have listened to the opinions of people I disagree with). It seems like the velocity of a project is the reciprocal of the number of independent implementations of a spec because any one of the implementations can slow down all the implementations (like what happened with proto3 around required and optional). From what I can tell, a major source of the problem was that protobuf field semantics were absolutely critical to the scaling of google in the early days (as an inter-server protocol for rapidly evolving things like the search stack), but it's also being used as a data modelling toolkit (as a way of representing data with a high level of fidelity). And those two groups- along with the multiple language developers who don't want to deal with native code- do not see eye to eye, and want to drive the spec in their preferred direction. (FWIW nowadays I use pydantic for type descriptions and JSON for transport, but I really prefer having an external IDL unrelated to any specific programming language)

charleslmunger 2y ago
You'll be thrilled to hear about upb then, which was designed to be embeddable to power other languages without a from-scratch implementation - and now powers python protos. https://github.com/protocolbuffers/protobuf/tree/main/upb https://github.com/protocolbuffers/protobuf/tree/main/upb
lmm 2y ago
> It seems like the velocity of a project is the reciprocal of the number of independent implementations of a spec because any one of the implementations can slow down all the implementations (like what happened with proto3 around required and optional). Velocity and stability/maturity are in tension, sure. I think for a foundational protocol like protobuf you want the stability and reliability that come from multiple independent implementations more than you want it to be moving fast and breaking things.
- elcritch 2y ago

sbrother 2y ago

I still use proto2 if possible. The syntactic sugar around `oneof` wasn't nice enough to merit dealing with proto3's implicit presence -- maybe it is just because I learned proto2 with C++ and don't use Go, but proto3 just seemed like a big step back and introduced footguns that weren't there before. Happy to hear they are reverting some of those finally.

boulos 2y ago

That's not how I remember it. I thought proto3 was all about JSON compatibility. No?