4 ms·

It seems easy to square the practical experience of protobufs at Google versus the tradeoffs encountered in the wire format when you take into account how many

It seems easy to square the practical experience of protobufs at Google versus the tradeoffs encountered in the wire format when you take into account how many Google engineers have directly contributed to the diaspora of related formats such as CapnProto, flatbuffers, msgpack, etc with different tradeoffs (especially for smaller scales than Google). It seems clear enough that protobufs were optimized in a scale that involved a lot of time/cost-sensitive reads and far fewer time/cost-sensitive writes. That's a valid tradeoff for Google scale, that comes at a cost of other potential applications (such as telemetry that can be very time/cost-sensitive at write-time but has much more relaxed read time/cost-sensitivity). > Sure, having to recursively precompute lengths before serialization is a bit of a hassle, but I wouldn’t call it expensive. It's expensive in memory (needing to prebuild all the subcomponents of the message, instead of streaming them as needed), and potentially time (prebuilding then sending, versus streaming as soon as data is available to write). Of course, in the scheme of things it may not be expensive to particular use cases. (Which seems why the author seems to particularly highlight this specific use case where such things really are expensive overhead necessary to avoid.)

kentonv 6y ago
> It seems clear enough that protobufs were optimized in a scale that involved a lot of time/cost-sensitive reads and far fewer time/cost-sensitive writes. Nah, you're assuming too much. Protobuf was thrown together in a fairly ad hoc way by a couple (brilliant!) engineers (Jeff and Sanjay) to help make the Google search index protocol easier to maintain. The specific design decisions in Protobuf were not carefully tested or weighed against other possibilities. They just did something that worked well enough, and it worked well enough that it was rapidly adopted by the rest of the company. It was then too late to change anything. Yes, the fact that a variable-width size must be written before the data is a kind of big problem, which essentially requires you to make two passes over the message tree, one to compute sizes and one to write the data. There are some clever optimizations that can reduce the impact, but I don't think the designers would make the same decision if starting from scratch with no need to support legacy. But there was no point in history where it was worth breaking compatibility to fix this issue, so that's how it remains. It's a problem, just not that big a problem. (I maintained Protobuf for several years, including writing version 2 and open sourcing it.)
- haberman 6y ago
  Hey that reminds me, I've always wondered: how did it come to be that delimited encoding "won" for sub-messages and group encoding was deprecated? A lot of these problems would go away if things had gone the other way. Groups can be encoded in one pass, they are efficient to decode (unless you were trying to skip the sub-message, a la LazyField), and they don't have the string/message ambiguity in UnknownFieldSet that messages have. Do you remember how that came to pass? Maybe some of this happened during the evolution of proto1, which was before my time.
  - kentonv 6y ago
    That decision predates me. I think it was basically because early versions of protobuf didn't actually support using a message type as field type, so instead people would declare "string" fields and then manually encode/decode another protobuf type into that field. When the ability to explicitly use message types as field types was added to the language, they wanted to use it in those existing protocols without breaking compatibility, so the design was fit to the pre-existing practice. I argued for switching to group encoding for submessages when working on proto2, but was shot down. It was a long time ago, but I think the counter-argument was some combination of "it's not worth the breakage" and "the ability to lazily parse sub-messages is too valuable".
    - cbsmith 6y ago
      Your recollection sounds about right to me.
- kentonv 6y ago
  > The specific design decisions in Protobuf were not carefully tested or weighed against other possibilities. And just to be clear, I don't think this is bad. On the contrary, I think Protobuf won because it did a wide variety of things "pretty well" while moving quickly and solving real problems. This is how the best technologies are usually made, not by academically trying to perfect everything, but by banging out something that works and running with it to solve real problems. If you try to carefully design everything perfectly upfront, you'll spend a huge amount of time on decisions that don't really matter.

kentonv 6y ago

> It seems clear enough that protobufs were optimized in a scale that involved a lot of time/cost-sensitive reads and far fewer time/cost-sensitive writes. Nah, you're assuming too much. Protobuf was thrown together in a fairly ad hoc way by a couple (brilliant!) engineers (Jeff and Sanjay) to help make the Google search index protocol easier to maintain. The specific design decisions in Protobuf were not carefully tested or weighed against other possibilities. They just did something that worked well enough, and it worked well enough that it was rapidly adopted by the rest of the company. It was then too late to change anything. Yes, the fact that a variable-width size must be written before the data is a kind of big problem, which essentially requires you to make two passes over the message tree, one to compute sizes and one to write the data. There are some clever optimizations that can reduce the impact, but I don't think the designers would make the same decision if starting from scratch with no need to support legacy. But there was no point in history where it was worth breaking compatibility to fix this issue, so that's how it remains. It's a problem, just not that big a problem. (I maintained Protobuf for several years, including writing version 2 and open sourcing it.)

haberman 6y ago
Hey that reminds me, I've always wondered: how did it come to be that delimited encoding "won" for sub-messages and group encoding was deprecated? A lot of these problems would go away if things had gone the other way. Groups can be encoded in one pass, they are efficient to decode (unless you were trying to skip the sub-message, a la LazyField), and they don't have the string/message ambiguity in UnknownFieldSet that messages have. Do you remember how that came to pass? Maybe some of this happened during the evolution of proto1, which was before my time.
- kentonv 6y ago
  That decision predates me. I think it was basically because early versions of protobuf didn't actually support using a message type as field type, so instead people would declare "string" fields and then manually encode/decode another protobuf type into that field. When the ability to explicitly use message types as field types was added to the language, they wanted to use it in those existing protocols without breaking compatibility, so the design was fit to the pre-existing practice. I argued for switching to group encoding for submessages when working on proto2, but was shot down. It was a long time ago, but I think the counter-argument was some combination of "it's not worth the breakage" and "the ability to lazily parse sub-messages is too valuable".
  - cbsmith 6y ago
    Your recollection sounds about right to me.
kentonv 6y ago
> The specific design decisions in Protobuf were not carefully tested or weighed against other possibilities. And just to be clear, I don't think this is bad. On the contrary, I think Protobuf won because it did a wide variety of things "pretty well" while moving quickly and solving real problems. This is how the best technologies are usually made, not by academically trying to perfect everything, but by banging out something that works and running with it to solve real problems. If you try to carefully design everything perfectly upfront, you'll spend a huge amount of time on decisions that don't really matter.