7 ms·
> author decided it was a good idea to prepend the message with the message length encoded as a varint. > WHY? Oh, why?! Uh oh. Is this my HN moment? This is
by jimmytucson 3y ago
> author decided it was a good idea to prepend the message with the message length encoded as a varint.
> WHY? Oh, why?!
Uh oh. Is this my HN moment?
This is exactly how I implemented it at my company. We had to write many protobuf messages to one file in bulk (in parallel). I did a fair amount of research before designing this and didn’t find any standard for separating protobuf messages (in fact, found that there explicitly isn’t a standard in that protobuf doesn’t care). So I thought rather than using some “special” control character, like a null byte, which would inevitably be not-so-special and collide with somebody else’s (like Schema Registry’s “magic byte”), I’d use something meaningful like the number of bytes the following record is.
As for why I chose varint instead of just picking an interger size, well for one I got nerd-sniped by varint encoding and thought it would be cool to try and implement it in Scala. Secondly, I thought if I chose a fixed size integer, no matter what size I pick, my users will always surprise me and exceed it at least once, and when that happens, kaboom! I wanted to future proof this without wasting 64 goddamn bytes in front of each message, and also I got nerd-sniped, OK?!?
Someone on my team recently shared one of these files outside the company and so I really hope she’s not talking about me but that’s a crazy coincidence if not!
- lifthrasiir 3y agoYou can also define a wrapper message with a single repeated field. The resulting encoding would be `varint(field id * 8 + 1) varint(length)` plus the actual message, so it can be also easily generated from and parsed back into a raw byte sequence without protoc.
- xyzzy_plugh 3y agoThis approach is problematic because, as far as I know, all implementations expect to deserialize from a complete buffer -- at least all official implementations. That means the entire "stream" would have to be buffered in memory. You can, of course, simply read field by field, but few libraries expose the ability to do that simply. And the naive operation then becomes quite problematic indeed.
- pavlov 3y ago> “I wanted to future proof this without wasting 64 goddamn bytes in front of each message” 64 bytes would be a 512-bit integer. That seems like excessive future proofing for the length of any message that would be transmitted before the Sun runs out of fuel.
- xlogout 3y agoFun fact: just 103 more bits and it would be enough to address every point in the observable universe distinguishible by Planck length [1]. Not my thought originally, I heard it from somewhere else but can't find it. Possibly from Foone Turing. [1] https://www.wolframalpha.com/input?i=log2%28+volume+of+universe+%2F+planck+length%5E3+%29 https://www.wolframalpha.com/input?i=log2%28+volume+of+unive...
- benj111 3y agoTbf though, I'm sure the number of cat pics floating about the internet dwarfs this number, so it depends on the data I suppose. Plus rather simple things like pi could create rather a long message.
- gumby 3y agoI suppose pi has a cheat code: if you made the largest possible circle in the universe you only need enough digits to distinguish points on its circumference that are a Planck distance apart. Then you can either ignore any digits beyond that or even simply make them up, as there would be no way to measure the difference.
- spacebanana7 3y agoI wonder whether there’s an upper bound on the largest number that can be expressed in the observable universe. All digital representations rely on discrete states in hardware, and there’s a finite number of those in the observable universe, so there should a finite maximum number for computers.
- 3y ago
- NavinF 3y ago> didn’t find any standard for separating protobuf messages The fact that protobufs are not self-delimiting is an endless source of frustration, but I know of 2 standards for doing this: - SerializeDelimited* is part of the protobuf library: https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/util/delimited_message_util.cc https://github.com/protocolbuffers/protobuf/blob/main/src/go... - Riegeli is "a file format for storing a sequence of string records, typically serialized protocol buffers. It supports dense compression, fast decoding, seeking, detection and optional skipping of data corruption, filtering of proto message fields for even faster decoding, and parallel encoding": https://github.com/google/riegeli https://github.com/google/riegeli
- harvie 3y agohttps://msgpack.org/ https://msgpack.org/
- vore 3y agoHuh? This is an entirely different serialization format altogether.
- twic 3y agoCreate a MessagePack message containing a byte array, put the protobuf in that. Of course, the enterprise solution would be to hex encode it and put it in a CDATA section in an XML document.
- formerly_proven 3y agoJust stick this in front of your frames: 81 A2 70 62 C6 <32 bit be length> <protobuf data>
- KMag 3y agoYou got me. What's that format / magic number? At first I thought it was the zlib header for incompressible data, but that's not right. Is that just the MAC address of some NIC you own?
- blauditore 3y agoPrepending the message with a delimiter (size varint) is pretty common, even part of the reference Java implementation: https://protobuf.dev/reference/java/api-docs/com/google/protobuf/MessageLite.html#writeDelimitedTo-java.io.OutputStream- https://protobuf.dev/reference/java/api-docs/com/google/prot...
- catiopatio 3y agoYour length-prefixed message frames represent a distinct serialization scheme that provides external framing for embedded messages; it’s a standalone format, and you can use whatever frame header you want without worrying about collision. The only issue is if you were to ship a “protobuf” library that emits/consumes your (very much not protobuf) framing format. Also, a 64 bit frame length would only be 8 bytes, not 64 :-)
- tetrep 3y agoIf you want to future proof it you need to version it. It sounds like you're trying to pack many things into a single file, so having the first few bits of the file represent a version allows you to use fixed length integers without fear of them being too small (in the future). You can reserve the "last" version for varint if you truly need it. In general, I find adding versions to things allows for much more graceful future redesigns and that is, IMO, invaluable if you're concerned about longevity and are not confident in your ability to perfectly design something for the indefinite future.
- paulddraper 3y agoYAGNI The idea is so simple that any change would be a misfeature.
- stravant 3y agoI'm a big YAGNI proponent but format version numbers are not something I would dare to YAGNI.
- paulddraper 3y agoEven if it's varint separated messages?
- catiopatio 3y agoYou stick a magic number and a version number at the top of the file and call it done. It’s trivial and buys you a great deal. The magic means it’s possible to identify the file type. Maybe you’ll add a tool later that operates on multiple types of files. The version means you can evolve the contents in non-backwards-compatible ways, while maintaining the ability to read/parse the old version. It’s a pain in the ass to add a magic or version number later; there’s a reason why nearly every file format on the planet has both.
- paulddraper 3y ago
- paulddraper 3y agoDo you mean 64 bits I assume? Because that's all you would need.
- paulddraper 3y agoCongrats :) you perfectly re-created the actual Protobuf stream format. [1] https://protobuf.dev/reference/java/api-docs/com/google/protobuf/Parser#parseDelimitedFrom-java.io.InputStream- https://protobuf.dev/reference/java/api-docs/com/google/prot...
- zX41ZdbW 3y agoThis is the same as it's implemented in ClickHouse. In fact, it has two formats: `Protobuf` (with length-delimited messages) and `ProtobufSingle` (no delimiters, but only a single message can be read/written). And it is fairly common: https://clickhouse.com/docs/en/sql-reference/formats#protobuf https://clickhouse.com/docs/en/sql-reference/formats#protobu...
- catiopatio 3y agoConforming implementations will fail to interoperate with their “protobuf” serialization; it’s absolutely incorrect for them to call their length-prefixed framing protobuf.
- jimvdv 3y agoFrom my experimentation with writing my own push notification client for android, this is exactly what Google does with their push messages.
- seanw444 3y agoIf it was for fun and to learn how, that's fair. But are you aware of https://ntfy.sh https://ntfy.sh?
- jimvdv 3y agoAh I was just experimenting with what was possible, essentially I wanted to intercept notifications from an app to use for an automation.
- xyzzy_plugh 3y agoCan you share more?
- rkangel 3y agoYou've basically got two ways of handling this: Using a delimiter to mark end of packet. For text protocols we usually use "\n" to be the delimiter. This usually requires some escaping in the packet to make it unambiguous. Two standard protocols for this are SLIP and HDLC. Length encoding - like what you did. The downside of the delimiter approach is that it changes the length of the packet - when you are escaping one byte becomes two and that's sometimes a pain if you're doing it in place in memory (less of a problem if you're streaming byte by byte). The big advantage is that it allows for resynchronisation - if you lose a single byte from your stream, or your length byte ever gets corrupted then you're permanently out of sync - the receiver will never again know where the start or end of a packet is. With the delimiter approach, you just lose one packet. So if you're ever doing this for a UART or network stream or something, always do the delimiter approach!
- hiimkeks 3y ago> So if you're ever doing this for a UART or network stream or something, always do the delimiter approach! I'm confused. I mean for UART, sure. But network streams is usually sent over a protocol that recovers lost data. Am I reckless for sending length-prefixed data chunks over TCP?
- rkangel 3y agoNo, you're right that this approach is ok for TCP streams. I shouldn't have thrown that in there without clarification that I meant something more generic. I actually use SLIP for packetisation over TCP anyway, because then syncing for any other logging or diagnostic thing that joins halfway through is easy. Basically I find it to just be a more robust system.
- mannyv 3y ago"I actually use SLIP for packetisation over TCP anyway" Whoa, that's an acronym I haven't heard in a long time. SLIP/TCP seems redundant. Your serial encoding IP then sending it over TCP/IP? Is this a tunnel?
- brandmeyer 3y ago> isn’t a standard in that protobuf doesn’t care Shove protobuf into Something Else that does packet delimitation for you. I'm fond of SQLite for offline cases as a richer alternative to sstable.
- benreesman 3y agoOh who hasn’t written some LengthPrefixedProtobufRecord class. I get why Rachel would be annoyed if someone claimed it was protobuf, but even then it’s like, 8 seconds in xxd to see what’s going on. I shudder to think how well shit must be going for this to merit a Rachel post.
- bobmcn 3y agoWhy not just define a higher level proto that contains all possible (maybe repeated) protos you might want to include? Then if one of the included protos is not present, the higher level proto will efficiently encode that, and nothing gets broken.