7 ms·

The main point that people are missing is that experienced engineers don’t want to work with people who think like the author of this article. Protocol Buffers

The main point that people are missing is that experienced engineers don’t want to work with people who think like the author of this article. Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages. No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases. If you are building a system where your data can be described by protobufs, it may be a good choice. If your data structures don’t mash up well and you have to manipulate them heavily, protobufs may be a bad choice. Instead of pointing out use cases where a different serialization format may be better than protobufs and use cases where protobufs are better, and why, the author is spouting dogma about how protobufs are bad for every use case. Be wary of working with developers who prefer to argue about why they hate certain technologies instead of providing useful data and ways to solve problems. You don’t always have to solve a problem just because you are aware of it but don’t go shouting from the rooftops that a technology sucks for every use case under the planet when that’s obviously not the case. The author’s opinion is more of: I don’t like protobufs. Not: protobufs are wrong

derefr 8y ago
The author isn’t trying to say either of those things, really. I think their real point was something like: many companies that have an Enterprise Service Bus architecture for their pile of polyglot microservices, have a dogma for the encoding of the data flowing over the Bus. And Protobufs, though good for some use-cases, are a particularly bad dogma to be stuck with. That is, when they don’t work for a use-case, they really don’t work for that use-case—and an ESB means having to try to shove a wire format into pretty much every use-case. So “when it’s bad, it’s really bad” is a bad property for a format aimed specifically at being an enterprise’s ESB-interchange-format dogma to have. (Contrast to, say, JSON-RPC, which I’d place on the opposite end of the scale being described here. JSON-RPC is mediocre at best in terms of data fidelity, performance, etc., but it’s inoffensive—when it’s bad, it’s no worse than when it’s good. That makes it a good choice of ESB dogma... even though, in engineering terms, it sucks!)
- ender7 8y ago
  No, the author is showing off how smart they are, and in an unprofessional and nasty manner to boot. It's kind of you to gift the article with some genuinely thoughtful conclusions, but you're doing all the work there, not the original. As the grandparent says, it's easy to poke holes in something, especially when disregarding important requirements that influenced its design. The fact that the OP can't point to any implementation of the "right" way to do things is telling.
  - pm 8y ago
    The poster might be being kind, but their conclusions are spot on.
  - derefr 8y ago
    I mean, all the arguments to support my conclusions, are their arguments. If someone spends 10 pages giving you facts and anecdotes and other types of data, and then uses them to support a "dumb" conclusion, they've still done useful work. They don't realize what exactly it is their data proved, but they did do the work required to prove something. Those facts and anecdotes support a conclusion, whether or not they feed it to you at the end/in the abstract. And, that being said, I think they're right about the conclusion, too. A shorter way to say "Protocol Buffers are a bad choice of common ESB-bus format, even though being an ESB-bus format is the primary thing they're for and what everyone tries to use them for" is "Protocol Buffers are a wrong design." If something doesn't work when used to do the thing it's advertised to do, then it's broken, even if it can do something else. The only difference between "Protobuffers Are Wrong" and my conclusion is that I'm making the implicit context of their argument explicit. Read between the lines of their argument—they are talking about the use of protocol buffers (specifically, gRPC) in an ESB-bus common-format scenario. None of their arguments make sense if they aren't.
  - whateveracct 8y ago
    How is the author showing off exactly? Because he said “coproduct” and “Prism” (both of which have more well-defined meaning than profobufs btw lmao)
- 8y ago
lucio 8y ago
I see protobuf as being designed to be serialized fast and with a low footprint by not too complex assembly or C or Go. All the trade-offs are correct with that mindset. Maybe the author is looking at this from an academic type-theory perspective, and that's why he can't see the "why" for each one of the trade-offs.
- candiodari 8y ago
  Nope. There are similar formats designed to be rapidly serializable and back that outperform protobufs on this front. * cap'n proto https://capnproto.org/ https://capnproto.org/ * flatbuffers https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/ * hdf5 (for machine learning/numerical analysis/finance) https://support.hdfgroup.org/HDF5/ https://support.hdfgroup.org/HDF5/ You're just not going to beat these formats in serialization speed with anything (just mmap the file, use. Good luck beating it, certainly not with protobuf). Seriously, use hdf5 for machine learning and you can restart experiments and ... well they start. They don't spend the first minute or 2-3 reading their data back in. Protobufs focus on: * standard (meaning it's perhaps a bad standard but it's standard) (also meaning of course you don't get to choose formats, or perhaps I should say your company will lose a lot if it lets developers choose alternatives to protobuf anywhere) (this is where the frustration comes from) * streaming support (meaning you can write out and read in protobufs without having to keep the whole thing in memory) * extendable (including sort-of kind-of backwards-forward-compatibility. Meaning old code can read in a new version protobuf, change something and write out a proto that still has the fields it didn't understand) * language agnostic * composability (meaning cat protobuf1 protobuf2 > resultbuf means resultbuf is deserializable) * an attempt to efficiently store integers (meaning it's integer encoding format is bloody complex, but "space efficient", except not quite so efficient it doesn't need compression at which point, why bother ?) And in this protobuf will beat the above formats (except cap'n proto). None of the alternatives to protobuf existed when it was invented though. The only alternative in existence was ASN.1. And it beat that, by a LOT.
  - 21 8y ago
    HDF5 is quite bad and corruption prone. Parquet is gaining a lot of traction lately for this use case.
schizoidboy 8y ago
If we suppose that his conclusion is using boolean logic, then what you're saying is a strawman because of his last claim; namely, protobufs are bad if "[...] && !Google": > They're clearly written by amateurs, unbelievably ad-hoc, mired in gotchas, tricky to compile, and solve a problem that nobody but Google really has. This dovetails with other arguments that I've seen recently that are becoming more frequent: Have we entered a new world where the lessons of companies working at massive scales are not only generally superfluous for smaller scales, but are actively harmful?
- hodgesrm 8y ago
  No. But you have to examine use cases carefully including the assumptions. I think this has always been the case but people (in my experience at least) get a little dazzled by the massive scale of companies like Facebook and try to apply their solutions to problems for which they are simply not applicable.
- mcphage 8y ago
  > Have we entered a new world where the lessons of companies working at massive scales are not only generally superfluous for smaller scales, but are actively harmful? I think so, yeah. Microservices turn out to have a lot of negative consequences, and their positives work best when you have dozens or hundreds of developers. If you've got a handful of developers... not so great.
  - hodgesrm 8y ago
    Your argument really depends on the use case. If you have discrete, well-factored operations microservices can make even small systems easier to deploy and manage. For example, you do the front-end API in Java (easier to build secure, debuggable systems with good RDBMS access) and backend analytic services in Python (easier to scrape data out of XML/JSON). Splitting them up into 2 or more microservices can simplify development, CI/CD, and deployment. Whenever I see large numbers of microservices anywhere my null hypothesis is that some organizational disfunction is leading teams to factor applications into unnecessarily small pieces.
dman 8y ago
Arent you doing exactly what you are accusing the author of? One could say - "People are not wrong, they simply have constraints, advantages, and disadvantages". Drawing a conclusion about whether all experienced engineers want to work with the author based on a single article or opinion appears just as myopic.
finnthehuman 8y ago
>experienced engineers don’t want to work with people who think like the author of this article - has opinions on interface design - isn't afraid to be wrong publicly - is brash on a personal blog I dunno, this is mostly positive. I'd have to see how well they'd adapt to the much different context and goals of one of our design reviews, but this isn't an immediate red flag. There's a million ways to be bad at a job. I'll give them enough rope to hang themselves in a serious conversation rather than invent the idea that someone would be an unworkable perfectionist professionally just from being a type purist on the internet.
- anothergoogler 8y ago
  Brash on a personal blog is tricky. In this post's case, I'd worry that the blogger has a hard time separating technical deficiencies from professional incompetence. "Designed by amateurs" is an over-the-top and dubious claim.
  - mnsc 8y ago
    As an experienced engineer I can hear the pain behind that dubiuos claim and no, I wouldn't want to work with someone who behaves like this all the time but I wouldn't mind someone who once in a while gets fed up show some emotions. It's all about how you handle the aftermath, ie. can you apologize to people you accidentally hurt along the way.
    - ionforce 8y ago
      How about don't hurt people to begin with?
  - whateveracct 8y ago
    The Googlers who designed Protobuffers’ type system were undeniably amateurs in that realm without knowledge of the state-of-the-art.
    - anothergoogler 8y ago
alexashka 8y ago
Maybe that's the main point, for you. I don't think I'm missing anything :) You can take your logic and apply it to anything. For example killing innocent babies: "Killing babies has constraints, advantage and disadvantages. No other action in the universe is free from having advantages and disadvantages. They all have different use cases... I don't get why the author is 'spouting dogma' about how killing innocent babies is bad. Be wary of people who argue about killing babies etc etc." This relativism you're expressing can be seen as ridiculous when applied to killing babies - it can actually seem to make sense when you use it to justify your own biases - which is what you're doing. Of course doing that has it's advantages and disadvantages, etc etc no opinion is better or worse everything is relative :)
hodgesrm 8y ago
Protocol buffers are definitely not perfect. As of 2014 in Java the biggest issue was that you could not have different versions of protobuf [easily] running in the same VM due to the fact that protobuf generated version-dependent implementation code for IDL. So if one library depended version X of protobufs and another library depended on version Y, you had a problem because Java will only load one protobuf jar file. (Maybe it's been fixed but it was a problem then.) That said...I used protobufs to encode the log of Tungsten Replicator, a replicator for MySQL. We did not have a single version related error while I worked on it over a period of years. That was essential given the fact that replication logs are an on-disk format that must remain stable across version upgrades. Our logs in aggregate contained billions of transactions--you can't not read an older log file just because you are running a new software version. The claim that protobufs do not handle versioning is nonsense.
- angry_octet 8y ago
  Is that a problem with ProtoBufs or Java?
  - hodgesrm 8y ago
    Java only allows one version of any class within a single class loader. That's just how it works. So the problem was at the very least a limitation of the protobufs Java implementation. Speaking of which, the generated code was incredibly convoluted with what seemed like tangled dependencies on the underlying protobufs library. I used to dread debugging it--sometimes I would have to go into it to figure out what the upper layers were doing wrong. Luckily it did not happen very often. And I don't recall ever hitting a bug in protobufs itself for all that the code was virtually unreadable.
  - ionforce 8y ago
    I'm an outsider, but maybe it's a problem of their interaction? Most likely, the vast majority of Protobuf use is with code generated to a single, canonical location (which leads to the problem described above). But hypothetically, it would be possible for code to generate and refer to two different versions. That doesn't sound impossible. Maybe just improbably given the way things are typically set up.
whateveracct 8y ago
He did give a way to solve the problem though -.- the author isn’t spouting dogma and in fact uses a lot of factual arguments against their design.

derefr 8y ago

The author isn’t trying to say either of those things, really. I think their real point was something like: many companies that have an Enterprise Service Bus architecture for their pile of polyglot microservices, have a dogma for the encoding of the data flowing over the Bus. And Protobufs, though good for some use-cases, are a particularly bad dogma to be stuck with. That is, when they don’t work for a use-case, they really don’t work for that use-case—and an ESB means having to try to shove a wire format into pretty much every use-case. So “when it’s bad, it’s really bad” is a bad property for a format aimed specifically at being an enterprise’s ESB-interchange-format dogma to have. (Contrast to, say, JSON-RPC, which I’d place on the opposite end of the scale being described here. JSON-RPC is mediocre at best in terms of data fidelity, performance, etc., but it’s inoffensive—when it’s bad, it’s no worse than when it’s good. That makes it a good choice of ESB dogma... even though, in engineering terms, it sucks!)

ender7 8y ago
No, the author is showing off how smart they are, and in an unprofessional and nasty manner to boot. It's kind of you to gift the article with some genuinely thoughtful conclusions, but you're doing all the work there, not the original. As the grandparent says, it's easy to poke holes in something, especially when disregarding important requirements that influenced its design. The fact that the OP can't point to any implementation of the "right" way to do things is telling.
- pm 8y ago
  The poster might be being kind, but their conclusions are spot on.
- derefr 8y ago
  I mean, all the arguments to support my conclusions, are their arguments. If someone spends 10 pages giving you facts and anecdotes and other types of data, and then uses them to support a "dumb" conclusion, they've still done useful work. They don't realize what exactly it is their data proved, but they did do the work required to prove something. Those facts and anecdotes support a conclusion, whether or not they feed it to you at the end/in the abstract. And, that being said, I think they're right about the conclusion, too. A shorter way to say "Protocol Buffers are a bad choice of common ESB-bus format, even though being an ESB-bus format is the primary thing they're for and what everyone tries to use them for" is "Protocol Buffers are a wrong design." If something doesn't work when used to do the thing it's advertised to do, then it's broken, even if it can do something else. The only difference between "Protobuffers Are Wrong" and my conclusion is that I'm making the implicit context of their argument explicit. Read between the lines of their argument—they are talking about the use of protocol buffers (specifically, gRPC) in an ESB-bus common-format scenario. None of their arguments make sense if they aren't.
- whateveracct 8y ago
  How is the author showing off exactly? Because he said “coproduct” and “Prism” (both of which have more well-defined meaning than profobufs btw lmao)
8y ago

lucio 8y ago

I see protobuf as being designed to be serialized fast and with a low footprint by not too complex assembly or C or Go. All the trade-offs are correct with that mindset. Maybe the author is looking at this from an academic type-theory perspective, and that's why he can't see the "why" for each one of the trade-offs.

candiodari 8y ago
Nope. There are similar formats designed to be rapidly serializable and back that outperform protobufs on this front. * cap'n proto https://capnproto.org/ https://capnproto.org/ * flatbuffers https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/ * hdf5 (for machine learning/numerical analysis/finance) https://support.hdfgroup.org/HDF5/ https://support.hdfgroup.org/HDF5/ You're just not going to beat these formats in serialization speed with anything (just mmap the file, use. Good luck beating it, certainly not with protobuf). Seriously, use hdf5 for machine learning and you can restart experiments and ... well they start. They don't spend the first minute or 2-3 reading their data back in. Protobufs focus on: * standard (meaning it's perhaps a bad standard but it's standard) (also meaning of course you don't get to choose formats, or perhaps I should say your company will lose a lot if it lets developers choose alternatives to protobuf anywhere) (this is where the frustration comes from) * streaming support (meaning you can write out and read in protobufs without having to keep the whole thing in memory) * extendable (including sort-of kind-of backwards-forward-compatibility. Meaning old code can read in a new version protobuf, change something and write out a proto that still has the fields it didn't understand) * language agnostic * composability (meaning cat protobuf1 protobuf2 > resultbuf means resultbuf is deserializable) * an attempt to efficiently store integers (meaning it's integer encoding format is bloody complex, but "space efficient", except not quite so efficient it doesn't need compression at which point, why bother ?) And in this protobuf will beat the above formats (except cap'n proto). None of the alternatives to protobuf existed when it was invented though. The only alternative in existence was ASN.1. And it beat that, by a LOT.
- 21 8y ago
  HDF5 is quite bad and corruption prone. Parquet is gaining a lot of traction lately for this use case.

schizoidboy 8y ago

If we suppose that his conclusion is using boolean logic, then what you're saying is a strawman because of his last claim; namely, protobufs are bad if "[...] && !Google": > They're clearly written by amateurs, unbelievably ad-hoc, mired in gotchas, tricky to compile, and solve a problem that nobody but Google really has. This dovetails with other arguments that I've seen recently that are becoming more frequent: Have we entered a new world where the lessons of companies working at massive scales are not only generally superfluous for smaller scales, but are actively harmful?

hodgesrm 8y ago
No. But you have to examine use cases carefully including the assumptions. I think this has always been the case but people (in my experience at least) get a little dazzled by the massive scale of companies like Facebook and try to apply their solutions to problems for which they are simply not applicable.
mcphage 8y ago
> Have we entered a new world where the lessons of companies working at massive scales are not only generally superfluous for smaller scales, but are actively harmful? I think so, yeah. Microservices turn out to have a lot of negative consequences, and their positives work best when you have dozens or hundreds of developers. If you've got a handful of developers... not so great.
- hodgesrm 8y ago
  Your argument really depends on the use case. If you have discrete, well-factored operations microservices can make even small systems easier to deploy and manage. For example, you do the front-end API in Java (easier to build secure, debuggable systems with good RDBMS access) and backend analytic services in Python (easier to scrape data out of XML/JSON). Splitting them up into 2 or more microservices can simplify development, CI/CD, and deployment. Whenever I see large numbers of microservices anywhere my null hypothesis is that some organizational disfunction is leading teams to factor applications into unnecessarily small pieces.

dman 8y ago

Arent you doing exactly what you are accusing the author of? One could say - "People are not wrong, they simply have constraints, advantages, and disadvantages". Drawing a conclusion about whether all experienced engineers want to work with the author based on a single article or opinion appears just as myopic.

finnthehuman 8y ago

>experienced engineers don’t want to work with people who think like the author of this article - has opinions on interface design - isn't afraid to be wrong publicly - is brash on a personal blog I dunno, this is mostly positive. I'd have to see how well they'd adapt to the much different context and goals of one of our design reviews, but this isn't an immediate red flag. There's a million ways to be bad at a job. I'll give them enough rope to hang themselves in a serious conversation rather than invent the idea that someone would be an unworkable perfectionist professionally just from being a type purist on the internet.

anothergoogler 8y ago
Brash on a personal blog is tricky. In this post's case, I'd worry that the blogger has a hard time separating technical deficiencies from professional incompetence. "Designed by amateurs" is an over-the-top and dubious claim.
- mnsc 8y ago
  As an experienced engineer I can hear the pain behind that dubiuos claim and no, I wouldn't want to work with someone who behaves like this all the time but I wouldn't mind someone who once in a while gets fed up show some emotions. It's all about how you handle the aftermath, ie. can you apologize to people you accidentally hurt along the way.
  - ionforce 8y ago
    How about don't hurt people to begin with?
- whateveracct 8y ago
  The Googlers who designed Protobuffers’ type system were undeniably amateurs in that realm without knowledge of the state-of-the-art.
  - anothergoogler 8y ago

alexashka 8y ago

Maybe that's the main point, for you. I don't think I'm missing anything :) You can take your logic and apply it to anything. For example killing innocent babies: "Killing babies has constraints, advantage and disadvantages. No other action in the universe is free from having advantages and disadvantages. They all have different use cases... I don't get why the author is 'spouting dogma' about how killing innocent babies is bad. Be wary of people who argue about killing babies etc etc." This relativism you're expressing can be seen as ridiculous when applied to killing babies - it can actually seem to make sense when you use it to justify your own biases - which is what you're doing. Of course doing that has it's advantages and disadvantages, etc etc no opinion is better or worse everything is relative :)

hodgesrm 8y ago

Protocol buffers are definitely not perfect. As of 2014 in Java the biggest issue was that you could not have different versions of protobuf [easily] running in the same VM due to the fact that protobuf generated version-dependent implementation code for IDL. So if one library depended version X of protobufs and another library depended on version Y, you had a problem because Java will only load one protobuf jar file. (Maybe it's been fixed but it was a problem then.) That said...I used protobufs to encode the log of Tungsten Replicator, a replicator for MySQL. We did not have a single version related error while I worked on it over a period of years. That was essential given the fact that replication logs are an on-disk format that must remain stable across version upgrades. Our logs in aggregate contained billions of transactions--you can't not read an older log file just because you are running a new software version. The claim that protobufs do not handle versioning is nonsense.

angry_octet 8y ago
Is that a problem with ProtoBufs or Java?
- hodgesrm 8y ago
  Java only allows one version of any class within a single class loader. That's just how it works. So the problem was at the very least a limitation of the protobufs Java implementation. Speaking of which, the generated code was incredibly convoluted with what seemed like tangled dependencies on the underlying protobufs library. I used to dread debugging it--sometimes I would have to go into it to figure out what the upper layers were doing wrong. Luckily it did not happen very often. And I don't recall ever hitting a bug in protobufs itself for all that the code was virtually unreadable.
- ionforce 8y ago
  I'm an outsider, but maybe it's a problem of their interaction? Most likely, the vast majority of Protobuf use is with code generated to a single, canonical location (which leads to the problem described above). But hypothetically, it would be possible for code to generate and refer to two different versions. That doesn't sound impossible. Maybe just improbably given the way things are typically set up.

whateveracct 8y ago

He did give a way to solve the problem though -.- the author isn’t spouting dogma and in fact uses a lot of factual arguments against their design.