3 ms·

I'd say the main arguments are: 1. Many transports that you might use to transmit a Protobuf already have their own length tracking, making a length prefix red

I'd say the main arguments are: 1. Many transports that you might use to transmit a Protobuf already have their own length tracking, making a length prefix redundant. E.g. HTTP has Content-Length. Having two lengths feels wrong and forces you to decide what to do if they don't agree. 2. As others note, a length prefix makes it infeasible to serialize incrementally, since computing the serialized size requires most of the work of actually serializing it. With that said, TBH the decision was probably not carefully considered, it just evolved that way and the protocol was in wide use in Google before anyone could really change their mind. In practice, this did turn out to be a frequent source of confusion for users of the library, who often expected that the parser would just know where to stop parsing without them telling it explicitly. Especially when people used the functions that parse from an input stream of some sort, it surprised them that the parser would always consume the entire input rather than stopping at the end of the message. People would write two messages into a file and then find when they went to parse it, only one message would come out, with some weird combination of the data from the two inputs. Based on that experience, Cap'n Proto chose to go the other way, and define a message format that is explicitly self-delimiting, so the parser does in fact know where the message ends. I think this has proven to be the right choice in practice. (I maintained protobuf for a while and created Cap'n Proto.)

marcosdumay 3y ago
About #2, it seems that every time someone creates some format with a prefix serialized size, the next required step is creating a chunked format without that prefix. What IMO is perfectly ok. Chunking things is a perfectly fine way to allow for incremental communication. Just do it up-front, and you will not need the complexity of 2 different sizing formats.
- kentonv 3y ago
  In truth, the vast majority of protobuf users do not construct messages in a streaming way anyway; they have the whole message tree in memory upfront and the serialize it all at once. "Streaming" in gRPC (or Cap'n Proto) involves sending multiple messages. I think this is probably the right approach. Otherwise it seems very hard to use a message as its streaming in as you can never know which fields are not present vs. just haven't arrived yet. But representing a stream as multiple messages leaves it up to the application to decide what it wants in each chunk, which makes sense.
  - saghm 3y ago
    Yeah, that's another good point I forgot to mention; it's often hard to know the difference between "I haven't received this info yet but it might still come" and "I know this info definitely isn't coming at all". I've dealt with this more often in the context of a specific schema rather than in the encoding format itself (e.g. sending back a simple "success" response when relaying a message where the response will be delivered asynchronously so that the other side can tell the difference between the message or response getting lost and the message being successfully sent and the response just not having arrived yet), but it's definitely possible for an encoding format to be designed in a way where it's not clear where the message should end, and having a length prefix is an effective way to deal with that as well. I also fully agree with the "streaming can be done as multiple messages" approach; from the discussion here, it sounds like there may be some nice use cases where having a length prefix would be prohibitive (e.g. compression being generated on-the-fly), but these don't sound like typical use cases for encoding formats intended to be used generally; if anything, I'd expect something like a gzip response to be sent back as the entirety of a response (e.g. an HTTP get request for a specific file) rather than a part of a message in some custom protocol using protobuf or something similar.
mannyv 3y ago
About #1, FYI you should never trust the content-length of an HTTP request. It actually says that in one of the spec versions or another. The problem with writing the length out at the beginning of the message is that you need to know the length before you write it out. For large objects that may cause memory issues/be problematic. In many cases it works just fine. I doubt any protocol puts a "0" as the length for a dynamic length, but I can see a many-months long technical fight about that particular design decision.
- kentonv 3y ago
  > FYI you should never trust the content-length of an HTTP request. It actually says that in one of the spec versions or another. That's not quite right. When a Content-Length is present, it is always correct by definition (it is what determines the body size; there's no way for the body to be a different size). What you probably mean is that you shouldn't ever assume that a Content-Length is available, because an HTTP message can alternatively be encoded using Transfer-Encoding: chunked, in which case the length is not known upfront. That is true, but doesn't change my point: Either Content-Length or chunked transfer encoding serve to delimit the HTTP entity-body, which makes any other in-band delimiter redundant.
crabbone 3y ago
> serialize incrementally You cannot do it in Protobuf anyways (you need to allocate memory, remember? and you need to know how much to allocate, so you need to calculate the length anyways, you just throw it away after you calculate it, fun, fun fun!).
- kentonv 3y ago
  You don't necessarily have to allocate space for the whole message upfront. You can write some of it into a buffer, write that out to the socket, fill the buffer with more stuff, etc. There are actually programs that do this, although it's true that it's rarely worth the complexity.
  - crabbone 3y ago
    Not going to work really. The way it works in real life if, say, you want to serialize a list of varint is that you'd need some small memory chunk (let's call it staging) where you write individual integers (although, this is a bad idea for long lists, as you'd really want to write multiple elements at once if you have enough elements to justify spawning more threads). So, in this staging area you write those integers, more or less a byte at a time. You know they aren't going to take more than 8 bytes in the worst case, so you can have your staging area be 8 bytes. Then you need to keep track of how many bytes in total you wrote. And, at this point you may start writing the field with the serialized list (in bytes). The field will contain the length of the list. So, you've already calculated the length even before you started writing. Also, you need to store those varints somewhere before you start writing the field with the list... Protobuf isn't designed to do streaming. Well, really, it isn't designed at all. Like I wrote elsewhere, it was implemented first and then there was an attempt to describe what was implemented and call that "design". Having implemented several formats (eg. FLV and MP4) that were designed for streaming, I'm very confident Protobuf authors never concerned themselves with this aspect.

marcosdumay 3y ago

About #2, it seems that every time someone creates some format with a prefix serialized size, the next required step is creating a chunked format without that prefix. What IMO is perfectly ok. Chunking things is a perfectly fine way to allow for incremental communication. Just do it up-front, and you will not need the complexity of 2 different sizing formats.

kentonv 3y ago
In truth, the vast majority of protobuf users do not construct messages in a streaming way anyway; they have the whole message tree in memory upfront and the serialize it all at once. "Streaming" in gRPC (or Cap'n Proto) involves sending multiple messages. I think this is probably the right approach. Otherwise it seems very hard to use a message as its streaming in as you can never know which fields are not present vs. just haven't arrived yet. But representing a stream as multiple messages leaves it up to the application to decide what it wants in each chunk, which makes sense.
- saghm 3y ago
  Yeah, that's another good point I forgot to mention; it's often hard to know the difference between "I haven't received this info yet but it might still come" and "I know this info definitely isn't coming at all". I've dealt with this more often in the context of a specific schema rather than in the encoding format itself (e.g. sending back a simple "success" response when relaying a message where the response will be delivered asynchronously so that the other side can tell the difference between the message or response getting lost and the message being successfully sent and the response just not having arrived yet), but it's definitely possible for an encoding format to be designed in a way where it's not clear where the message should end, and having a length prefix is an effective way to deal with that as well. I also fully agree with the "streaming can be done as multiple messages" approach; from the discussion here, it sounds like there may be some nice use cases where having a length prefix would be prohibitive (e.g. compression being generated on-the-fly), but these don't sound like typical use cases for encoding formats intended to be used generally; if anything, I'd expect something like a gzip response to be sent back as the entirety of a response (e.g. an HTTP get request for a specific file) rather than a part of a message in some custom protocol using protobuf or something similar.

mannyv 3y ago

About #1, FYI you should never trust the content-length of an HTTP request. It actually says that in one of the spec versions or another. The problem with writing the length out at the beginning of the message is that you need to know the length before you write it out. For large objects that may cause memory issues/be problematic. In many cases it works just fine. I doubt any protocol puts a "0" as the length for a dynamic length, but I can see a many-months long technical fight about that particular design decision.

kentonv 3y ago
> FYI you should never trust the content-length of an HTTP request. It actually says that in one of the spec versions or another. That's not quite right. When a Content-Length is present, it is always correct by definition (it is what determines the body size; there's no way for the body to be a different size). What you probably mean is that you shouldn't ever assume that a Content-Length is available, because an HTTP message can alternatively be encoded using Transfer-Encoding: chunked, in which case the length is not known upfront. That is true, but doesn't change my point: Either Content-Length or chunked transfer encoding serve to delimit the HTTP entity-body, which makes any other in-band delimiter redundant.

crabbone 3y ago

> serialize incrementally You cannot do it in Protobuf anyways (you need to allocate memory, remember? and you need to know how much to allocate, so you need to calculate the length anyways, you just throw it away after you calculate it, fun, fun fun!).

kentonv 3y ago
You don't necessarily have to allocate space for the whole message upfront. You can write some of it into a buffer, write that out to the socket, fill the buffer with more stuff, etc. There are actually programs that do this, although it's true that it's rarely worth the complexity.
- crabbone 3y ago
  Not going to work really. The way it works in real life if, say, you want to serialize a list of varint is that you'd need some small memory chunk (let's call it staging) where you write individual integers (although, this is a bad idea for long lists, as you'd really want to write multiple elements at once if you have enough elements to justify spawning more threads). So, in this staging area you write those integers, more or less a byte at a time. You know they aren't going to take more than 8 bytes in the worst case, so you can have your staging area be 8 bytes. Then you need to keep track of how many bytes in total you wrote. And, at this point you may start writing the field with the serialized list (in bytes). The field will contain the length of the list. So, you've already calculated the length even before you started writing. Also, you need to store those varints somewhere before you start writing the field with the list... Protobuf isn't designed to do streaming. Well, really, it isn't designed at all. Like I wrote elsewhere, it was implemented first and then there was an attempt to describe what was implemented and call that "design". Having implemented several formats (eg. FLV and MP4) that were designed for streaming, I'm very confident Protobuf authors never concerned themselves with this aspect.