3 ms·

As an engineer at Google, my opinion on Protocol Buffers changed massively. Pre-Google I found them awkward, the language bindings in Python sucked, and I didn'

As an engineer at Google, my opinion on Protocol Buffers changed massively. Pre-Google I found them awkward, the language bindings in Python sucked, and I didn't really see the point. I knew a schema for services was a good idea, but protobufs didn't seem like the best option. The thing is that at Google protobufs are used everywhere. Like, absolutely everywhere. Think of all the places they could be, and it's way more. All the tooling understands them, code search and go-to-reference works on them everywhere, they are truly transformational on how many different services with many different implementations interact. Are they perfect? Far from it. But a Go-only implementation misses almost all the value of protobufs. If I was inventing them for Python I could do better (pickle? maybe not) but the whole point is that they aren't language/ecosystem/use-case specific. If this was Rob Pike's frustration then I can't help but feel he missed the point, or this post is a little disingenuous as to the benefits. I've not seen Gobs used in Google, but I'd imagine an engineer would need to make a very strong case against using protos between services, regardless of if both services are in Go.

knorker 3y ago
In my opinion this is pretty on-brand for Pike. Back when he did his best work, it was possible for one person to "just write the new thing", without making it fit with anything else. There was nothing else to fit it with. You could invent everything from scratch, and not only was it not a waste of time, if you were good enough it had a chance of being the best fit for purpose. You could take shortcuts. You could have every part of your system be "odd", because nothing was "even". That's not true anymore. And the way I see it Pike has not moved on. Science in general had this switch at one point, too. There was a point where one could know all of science. But it's long gone.
- danpalmer 3y ago
  Interesting hypothesis. I can't comment on Pike's history here, but at Google there's certainly a noticeable difference between old special cases that were built pre-2015 ish, and the modern world where everything is very cohesive. I get the impression that there was a big push to achieve that, led from around that time by various products. It's hit some areas more than others, but there do seem to be almost no new products building in the way you've described now.
- hellozomo 3y ago
  This is an unnecessary ad hominem attack. He wrote this over a decade ago, in the midst of doing his best work creating Go itself. gob was his opinionated way of doing a Go-specific encoding, while also supporting any number of other encodings in the language. Go has incredibly good support for almost every popular encoding there is. gob has also been used successfully by a number of projects. In many cases, it's a perfectly good way to encode a piece of data that is completely local to a Go program.
  - knorker 3y ago
    > This is an unnecessary ad hominem attack I didn't mean it as such. > in the midst of doing his best work creating Go itself. Not to go too far off topic, but Go is another example. It famously ignores decades of language theory, and they wrote their own assembler, linker, etc. Now, much of that has been undone and rewritten, as Go became more adopted, requiring playing well with the rest of the ecosystem. (but much of it we're unfortunately stuck with, because it's part of the language) 30 years ago there was no ecosystem to play well with, and compared to now we were just banging rocks together. Back then you could be a CS polymath as one person. Well, I couldn't, but Pike could. It was the old days of John Carmack starting every game engine with an empty directory. I'm saying that today nobody can. Even John Carmack could not on his own write a AAA game. (I know ID had other coders, but my point stands)
    - hellozomo 3y ago
      Well, to go off-topic with you, the idea that Go "ignores decades of language theory" is just an opinion that you hold. A factual statement would be to say that the Go designers omitted a great many features other languages have included. The idea that they did so out of ignorance is ridiculous, given the background of the Go design team. They made considered decisions about what to include in Go. A fixed-gear bicycle is not "ignoring" decades of bicycle design theory. Creating a programming language is an engineering task not a theoretical task. Which means there are major trade-offs to be made. And they chose their trade-offs. The wild success of Go should at least make you consider whether or not they're better at making these trade-offs than you, and most language designers, are. Maybe they were wrong about some choices, which is why Go is still evolving, but they were self-evidently mostly correct.
- geodel 3y ago
  Well, it is 2011 post. Now Pike is retired, I don't think he cares or matters anyway.
  - knorker 3y ago
    I think he matters, for computer history. I never aimed to have him care what I say, though.
randomdata 3y ago
> I can't help but feel he missed the point Did he miss the point, or was the point to light a fire under the protobuf team to improve their product? Which they eventually did (e.g. required fields were removed after this post was made).
- danpalmer 3y ago
  It's possible that was an aim, and I'm not familiar with the timelines here, however this does again focus more on the mechanics of protos which I still don't think matter as much as them being fully integrated into a company like they are at Google. The mechanics of them are still not great. Maybe this was a blocker to that happening though?
- _ZeD_ 3y ago
  FYI: required fields were re-introduced in protobuf. why? because despite what the gobs team and others thinks, that is a necessary feature to work with data structures. If "everything is optional", what is the value of any structure, over sending an untyped associative array of stuff?
  - morelisp 3y ago
    "Optional" fields, as in fields you can tell if they were explicitly set or not, were re-introduced. Depending on your definition either required fields were not reintroduced, or required fields with a mandatory default value were until recently the only kind of field in proto3.
Timon3 3y ago
As awkward as Protobufs might be - is there any similar format with so many client bindings? I tried comparing the message formats I could find (e.g. Cap'n'Proto, Protobuf, msgpack and others), and still Protobuf seems to have the most supported languages. I'd be happy for any other suggestions!
yegle 3y ago
It's called "Larry & Sergey Protobuf Moving Co." for a reason. See the t-shirt design in the video screenshot here: https://isocpp.org/blog/2020/07/cppcon-2019-there-are-no-zero-cost-abstractions-chandler-carruth https://isocpp.org/blog/2020/07/cppcon-2019-there-are-no-zer... Disclaimer: an employee at the said moving company.

knorker 3y ago

In my opinion this is pretty on-brand for Pike. Back when he did his best work, it was possible for one person to "just write the new thing", without making it fit with anything else. There was nothing else to fit it with. You could invent everything from scratch, and not only was it not a waste of time, if you were good enough it had a chance of being the best fit for purpose. You could take shortcuts. You could have every part of your system be "odd", because nothing was "even". That's not true anymore. And the way I see it Pike has not moved on. Science in general had this switch at one point, too. There was a point where one could know all of science. But it's long gone.

danpalmer 3y ago
Interesting hypothesis. I can't comment on Pike's history here, but at Google there's certainly a noticeable difference between old special cases that were built pre-2015 ish, and the modern world where everything is very cohesive. I get the impression that there was a big push to achieve that, led from around that time by various products. It's hit some areas more than others, but there do seem to be almost no new products building in the way you've described now.
hellozomo 3y ago
This is an unnecessary ad hominem attack. He wrote this over a decade ago, in the midst of doing his best work creating Go itself. gob was his opinionated way of doing a Go-specific encoding, while also supporting any number of other encodings in the language. Go has incredibly good support for almost every popular encoding there is. gob has also been used successfully by a number of projects. In many cases, it's a perfectly good way to encode a piece of data that is completely local to a Go program.
- knorker 3y ago
  > This is an unnecessary ad hominem attack I didn't mean it as such. > in the midst of doing his best work creating Go itself. Not to go too far off topic, but Go is another example. It famously ignores decades of language theory, and they wrote their own assembler, linker, etc. Now, much of that has been undone and rewritten, as Go became more adopted, requiring playing well with the rest of the ecosystem. (but much of it we're unfortunately stuck with, because it's part of the language) 30 years ago there was no ecosystem to play well with, and compared to now we were just banging rocks together. Back then you could be a CS polymath as one person. Well, I couldn't, but Pike could. It was the old days of John Carmack starting every game engine with an empty directory. I'm saying that today nobody can. Even John Carmack could not on his own write a AAA game. (I know ID had other coders, but my point stands)
  - hellozomo 3y ago
    Well, to go off-topic with you, the idea that Go "ignores decades of language theory" is just an opinion that you hold. A factual statement would be to say that the Go designers omitted a great many features other languages have included. The idea that they did so out of ignorance is ridiculous, given the background of the Go design team. They made considered decisions about what to include in Go. A fixed-gear bicycle is not "ignoring" decades of bicycle design theory. Creating a programming language is an engineering task not a theoretical task. Which means there are major trade-offs to be made. And they chose their trade-offs. The wild success of Go should at least make you consider whether or not they're better at making these trade-offs than you, and most language designers, are. Maybe they were wrong about some choices, which is why Go is still evolving, but they were self-evidently mostly correct.
geodel 3y ago
Well, it is 2011 post. Now Pike is retired, I don't think he cares or matters anyway.
- knorker 3y ago
  I think he matters, for computer history. I never aimed to have him care what I say, though.

randomdata 3y ago

> I can't help but feel he missed the point Did he miss the point, or was the point to light a fire under the protobuf team to improve their product? Which they eventually did (e.g. required fields were removed after this post was made).

danpalmer 3y ago
It's possible that was an aim, and I'm not familiar with the timelines here, however this does again focus more on the mechanics of protos which I still don't think matter as much as them being fully integrated into a company like they are at Google. The mechanics of them are still not great. Maybe this was a blocker to that happening though?
_ZeD_ 3y ago
FYI: required fields were re-introduced in protobuf. why? because despite what the gobs team and others thinks, that is a necessary feature to work with data structures. If "everything is optional", what is the value of any structure, over sending an untyped associative array of stuff?
- morelisp 3y ago
  "Optional" fields, as in fields you can tell if they were explicitly set or not, were re-introduced. Depending on your definition either required fields were not reintroduced, or required fields with a mandatory default value were until recently the only kind of field in proto3.

Timon3 3y ago

As awkward as Protobufs might be - is there any similar format with so many client bindings? I tried comparing the message formats I could find (e.g. Cap'n'Proto, Protobuf, msgpack and others), and still Protobuf seems to have the most supported languages. I'd be happy for any other suggestions!

yegle 3y ago

It's called "Larry & Sergey Protobuf Moving Co." for a reason. See the t-shirt design in the video screenshot here: https://isocpp.org/blog/2020/07/cppcon-2019-there-are-no-zero-cost-abstractions-chandler-carruth https://isocpp.org/blog/2020/07/cppcon-2019-there-are-no-zer... Disclaimer: an employee at the said moving company.