4 ms·

You don't necessarily have to allocate space for the whole message upfront. You can write some of it into a buffer, write that out to the socket, fill the buffe

You don't necessarily have to allocate space for the whole message upfront. You can write some of it into a buffer, write that out to the socket, fill the buffer with more stuff, etc. There are actually programs that do this, although it's true that it's rarely worth the complexity.

crabbone 3y ago
Not going to work really. The way it works in real life if, say, you want to serialize a list of varint is that you'd need some small memory chunk (let's call it staging) where you write individual integers (although, this is a bad idea for long lists, as you'd really want to write multiple elements at once if you have enough elements to justify spawning more threads). So, in this staging area you write those integers, more or less a byte at a time. You know they aren't going to take more than 8 bytes in the worst case, so you can have your staging area be 8 bytes. Then you need to keep track of how many bytes in total you wrote. And, at this point you may start writing the field with the serialized list (in bytes). The field will contain the length of the list. So, you've already calculated the length even before you started writing. Also, you need to store those varints somewhere before you start writing the field with the list... Protobuf isn't designed to do streaming. Well, really, it isn't designed at all. Like I wrote elsewhere, it was implemented first and then there was an attempt to describe what was implemented and call that "design". Having implemented several formats (eg. FLV and MP4) that were designed for streaming, I'm very confident Protobuf authors never concerned themselves with this aspect.