8 ms·

This was like 2 or 3 sentences worth of good points sprinkled in between a bunch of stuff that I'm pretty sure isn't an actual problem for anyone. If anything,

This was like 2 or 3 sentences worth of good points sprinkled in between a bunch of stuff that I'm pretty sure isn't an actual problem for anyone. If anything, the things pointed out are issues for people trying to write code that manipulates protos generically, which is not what most people spend their time writing and is probably exactly the wrong thing to optimize. The main good point: Google's problems are probably not your problems, don't just blindly adopt Google tech for no reason. Also: calling people amateurs without really substantiating is a huge smell IMO. The average Google engineer isn't a genius or particularly amazing by any stretch, but especially for something as core/foundational as protobuf, the answer is much more likely something like "these decisions made more sense for Google internally, especially when weighing against the cost of significantly re-architecting how proto works". The ad-hominem at the beginning reeks of someone who had an email chain that went like: "You guys are doing proto wrong, don't you realize protos should obviously be like XYZ?" "Well actually we'd like to do X but it would've been too hard, I'm not actually sure Y is a net-win, ..." (omg what amateurs...)

msla 7y ago
Using the word "amateur" to mean "inexperienced" or "unskilled" is a smell, or, more precisely, an equivocation: It lumps not being paid to do something with doing that thing poorly, which definitely aren't the same thing.
kentonv 7y ago
> calling people amateurs without really substantiating is a huge smell IMO. The average Google engineer isn't a genius or particularly amazing by any stretch Note that Jeff Dean and Sanjay Ghemawat -- the original creators of Protobuf -- aren't average Google engineers. They are literally the highest-ranked engineers at Google (Level 11, "Senior Fellow", a title assigned only to the two of them last I heard), and they basically invented MapReduce, BigTable, Spanner, and a variety of other foundational distributed systems technologies. Jeff now leads the AI division while Sanjay continues to focus on systems infrastructure. So yeah, "amateurs". (Disclosure: I wrote Protobuf v2, but it was just a fresh implementation of the same design.)
- wollsvagwn 7y ago
  Actually, Jeff Dean is an idiot now. I think that's what happens when you get to level 11. > Jeff Dean: That’s obviously a very broad space, and there’s a lot of potential for using machine learning to help tackle climate change-related topics or mitigate some of the effects. So we’re pretty excited about this. I think Google and the [AI] community in general [are] excited because it’s a serious problem, and also … one that has a lot of technical meat behind it. Like, how can we actually apply machine learning [to] some of these subproblems? That answer contains zero information. It's word salad. So ya, smartest guys at Google.
  - bradhe 7y ago
    That's like 100 words to say "Yes."
  - tedunangst 7y ago
    Does that retroactively make protobuffers stupid too?
    - mironic 7y ago
      It doesn't but I don't get why saying someone is smart makes their work better when smart people obviously make stupid decisions and give stupid answers. I've used protobuffs, they're fine. I was riffing on Kenton's justification.
      - lobo_tuerto 7y ago
        Because being smart is related to making less stupid decisions? Think about it in terms of probability. Smarter -> less probability of making stupid decisions and giving stupid answers. Dumber -> more probability of making stupid decisions and giving stupid answers.
      - kentonv 7y ago
        My only point was that it's absurd to call them "amateurs", and that the author damages his own credibility by doing so.
kudokatz 7y ago
> sprinkled in between a bunch of stuff that I'm pretty sure isn't an actual problem for anyone I'd push back at this. After starting to use protos (even at Google) these issues smack pretty much everybody right in the face.
cmrdporcupine 7y ago
As a Googler who has seen gPRC and Protobufs continually rammed into places where they clearly don't belong (ahem, embedded systems), I actually have a lot of sympathy with this article. I wouldn't call the authors of protobufs amateurs, but I do think there's flaws there, many of them around the type system, as this article points out, but also a lot around the language APIs. My biggest concern with protobuf though is that it ends becoming the proverbial "I have a hammer, now everything looks like a nail" scenario. Every Googler goes through orientation with protobufs and gRPC, and then they proceed to stamp it everywhere... including places it may not belong. And then take it with them when they leave Google. I think the article tone is inflammatory but most of the points are solid.
- alecbenzer 7y ago
  > many of them around the type system, as this article points out, but also a lot around the language APIs. I think there's a lot not to love about gRPC/protobufs (used them at Google and again now at a startup) but I don't feel like this did a good job highlighting those issues.
- jhallenworld 7y ago
  I wrote my own serialization library for embedded systems recently. It serializes data that can be defined and initialized natively in C. The intention is for data section only data, not heap data. So for example, you can have variable length arrays, but you must have a maximum length to match the preallocated C array declaration. The serialized format mimicks JSON, except that it supports tables- arrays of structs. This saves space compared with actual JSON, since the column names only have to be given once. Also tables where the columns are primitive types can be loaded directly into common spreadsheet applications. This is useful for my specific application. The application has a CLI that allows you to browse the data with xpath like expressions. You can set or get any field of the hierarchy (using the serialization format), and since the type is known it can parse and schema check the user input. I use the serialized format for storing a copy of the data in flash memory, so that it can be restored on boot up. The metadata with the type information is all marked as const. In embedded systems, this data is placed in flash memory instead of precious RAM.
otabdeveloper2 7y ago
Pretty much all Google libraries are reliably amateur and very low quality. No offense, but they give off that inimitable H1B 'code this shit and get out of here' smell. Protobufs are no exception.
joatmon-snoo 7y ago
Also worth calling out: protos have evolved a lot since they were originally created, and there are very few people that are actually aware of the deep, dark corners of protos (think extensions, JS, Android...). (Got a complaint about what Google's open sourced for protos? You don't even want to imagine some of the batshit crazy stuff we did with it before that...)
- colanderman 7y ago
  > extensions As someone who has done extensive work with Protobuf extension fields in Python, I can only echo that the API is nightmarish.
coldtea 7y ago
>This was like 2 or 3 sentences worth of good points sprinkled in between a bunch of stuff that I'm pretty sure isn't an actual problem for anyone. If anything, the things pointed out are issues for people trying to write code that manipulates protos generically, which is not what most people spend their time writing and is probably exactly the wrong thing to optimize. Handling a serialization scheme "generically" is the "wrong thing to optimize"? Sounds like the #1 thing anybody would want from it...
- mooted1 7y ago
  Have you ever worked in a polyglot ecosystem with rapidly evolving schemas? Tools like protobuf and thrift were designed to facilitate schema evolution since interfaces in these ecosystems evolve quickly and independently. Generics undermine this by creating strict dependencies on a few types, making it difficult to evolve a single type without breaking things. Poorly implemented generics would undermine one of the design goals of this project. In addition, there aren't nearly as many opportunities for generics in an IDL as in a programming language, so what would the upside even be?
- alecbenzer 7y ago
  Obviously you need certain core pieces of infra that handle protos generically (like serializing and deserializing) but a) total SLOC for that logic is much, much less than SLOC for code that works with _specific_ protocol buffers b) "consumers" of protocol buffers as a tool/technology mostly don't worry about what's going on under the hood of the generic serialization, etc. code So it will often make sense to to make that core generic logic even significantly more complicated if it means making the stuff that everyone has to write over and over again even a little bit easier.

msla 7y ago

Using the word "amateur" to mean "inexperienced" or "unskilled" is a smell, or, more precisely, an equivocation: It lumps not being paid to do something with doing that thing poorly, which definitely aren't the same thing.

kentonv 7y ago

> calling people amateurs without really substantiating is a huge smell IMO. The average Google engineer isn't a genius or particularly amazing by any stretch Note that Jeff Dean and Sanjay Ghemawat -- the original creators of Protobuf -- aren't average Google engineers. They are literally the highest-ranked engineers at Google (Level 11, "Senior Fellow", a title assigned only to the two of them last I heard), and they basically invented MapReduce, BigTable, Spanner, and a variety of other foundational distributed systems technologies. Jeff now leads the AI division while Sanjay continues to focus on systems infrastructure. So yeah, "amateurs". (Disclosure: I wrote Protobuf v2, but it was just a fresh implementation of the same design.)

wollsvagwn 7y ago
Actually, Jeff Dean is an idiot now. I think that's what happens when you get to level 11. > Jeff Dean: That’s obviously a very broad space, and there’s a lot of potential for using machine learning to help tackle climate change-related topics or mitigate some of the effects. So we’re pretty excited about this. I think Google and the [AI] community in general [are] excited because it’s a serious problem, and also … one that has a lot of technical meat behind it. Like, how can we actually apply machine learning [to] some of these subproblems? That answer contains zero information. It's word salad. So ya, smartest guys at Google.
- bradhe 7y ago
  That's like 100 words to say "Yes."
- tedunangst 7y ago
  Does that retroactively make protobuffers stupid too?
  - mironic 7y ago
    It doesn't but I don't get why saying someone is smart makes their work better when smart people obviously make stupid decisions and give stupid answers. I've used protobuffs, they're fine. I was riffing on Kenton's justification.
    - lobo_tuerto 7y ago
      Because being smart is related to making less stupid decisions? Think about it in terms of probability. Smarter -> less probability of making stupid decisions and giving stupid answers. Dumber -> more probability of making stupid decisions and giving stupid answers.
    - kentonv 7y ago
      My only point was that it's absurd to call them "amateurs", and that the author damages his own credibility by doing so.

kudokatz 7y ago

> sprinkled in between a bunch of stuff that I'm pretty sure isn't an actual problem for anyone I'd push back at this. After starting to use protos (even at Google) these issues smack pretty much everybody right in the face.

cmrdporcupine 7y ago

As a Googler who has seen gPRC and Protobufs continually rammed into places where they clearly don't belong (ahem, embedded systems), I actually have a lot of sympathy with this article. I wouldn't call the authors of protobufs amateurs, but I do think there's flaws there, many of them around the type system, as this article points out, but also a lot around the language APIs. My biggest concern with protobuf though is that it ends becoming the proverbial "I have a hammer, now everything looks like a nail" scenario. Every Googler goes through orientation with protobufs and gRPC, and then they proceed to stamp it everywhere... including places it may not belong. And then take it with them when they leave Google. I think the article tone is inflammatory but most of the points are solid.

alecbenzer 7y ago
> many of them around the type system, as this article points out, but also a lot around the language APIs. I think there's a lot not to love about gRPC/protobufs (used them at Google and again now at a startup) but I don't feel like this did a good job highlighting those issues.
jhallenworld 7y ago
I wrote my own serialization library for embedded systems recently. It serializes data that can be defined and initialized natively in C. The intention is for data section only data, not heap data. So for example, you can have variable length arrays, but you must have a maximum length to match the preallocated C array declaration. The serialized format mimicks JSON, except that it supports tables- arrays of structs. This saves space compared with actual JSON, since the column names only have to be given once. Also tables where the columns are primitive types can be loaded directly into common spreadsheet applications. This is useful for my specific application. The application has a CLI that allows you to browse the data with xpath like expressions. You can set or get any field of the hierarchy (using the serialization format), and since the type is known it can parse and schema check the user input. I use the serialized format for storing a copy of the data in flash memory, so that it can be restored on boot up. The metadata with the type information is all marked as const. In embedded systems, this data is placed in flash memory instead of precious RAM.

otabdeveloper2 7y ago

Pretty much all Google libraries are reliably amateur and very low quality. No offense, but they give off that inimitable H1B 'code this shit and get out of here' smell. Protobufs are no exception.

joatmon-snoo 7y ago

Also worth calling out: protos have evolved a lot since they were originally created, and there are very few people that are actually aware of the deep, dark corners of protos (think extensions, JS, Android...). (Got a complaint about what Google's open sourced for protos? You don't even want to imagine some of the batshit crazy stuff we did with it before that...)

colanderman 7y ago
> extensions As someone who has done extensive work with Protobuf extension fields in Python, I can only echo that the API is nightmarish.

coldtea 7y ago

>This was like 2 or 3 sentences worth of good points sprinkled in between a bunch of stuff that I'm pretty sure isn't an actual problem for anyone. If anything, the things pointed out are issues for people trying to write code that manipulates protos generically, which is not what most people spend their time writing and is probably exactly the wrong thing to optimize. Handling a serialization scheme "generically" is the "wrong thing to optimize"? Sounds like the #1 thing anybody would want from it...

mooted1 7y ago
Have you ever worked in a polyglot ecosystem with rapidly evolving schemas? Tools like protobuf and thrift were designed to facilitate schema evolution since interfaces in these ecosystems evolve quickly and independently. Generics undermine this by creating strict dependencies on a few types, making it difficult to evolve a single type without breaking things. Poorly implemented generics would undermine one of the design goals of this project. In addition, there aren't nearly as many opportunities for generics in an IDL as in a programming language, so what would the upside even be?
alecbenzer 7y ago
Obviously you need certain core pieces of infra that handle protos generically (like serializing and deserializing) but a) total SLOC for that logic is much, much less than SLOC for code that works with _specific_ protocol buffers b) "consumers" of protocol buffers as a tool/technology mostly don't worry about what's going on under the hood of the generic serialization, etc. code So it will often make sense to to make that core generic logic even significantly more complicated if it means making the stuff that everyone has to write over and over again even a little bit easier.