3 ms·
If I opened my comment with personal attacks on the authors competency I hope people would downvote me. This has 137 points right now and I don’t even think it
by jchw 7y ago
If I opened my comment with personal attacks on the authors competency I hope people would downvote me. This has 137 points right now and I don’t even think it makes much sense; it sounds as though they stopped short of understanding the reasoning behind many of the limitations and just assume they are mistakes, when I’d argue it makes a ton of sense from a PoV of how protos work.
Like why can’t you repeat a oneof? Imo because it stops acting like a oneof. A oneof is treated like a union in the generated code, and you can expect that only one of the oneof message tags will appear in binary for that message. If you want a repeated oneof, it’s actually no different than if you had all of the fields be repeated and outside of a oneof. It gives a different interface in generated code but it’s the exact same thing you’d want in the underlying proto binaries: multiple of whichever message tag. The distinction of oneof is not useful here.
I think the proto design is quite smart, OTOH. Like the format is designed to allow backwards and forwards compatibility provided you follow some rules that you can easily enforce via linting.
Yes, there are some slightly odd side effects. Many things in proto are special cased. Like Map can’t be repeated because Map is already repeated; maps are sugar for repeated pairs, and you can’t have a repeated repeated field. You can of course just make a quick submessage with a map and repeat that. It doesn’t seem like that big of a deal.
OTOH, for how simple protobufs wire format is, the sugar features like Map help make it feel a bit richer from the PoV of the generated code, whereas the simple wire format makes it more predictable, easier to understand under the hood, and helps to future proof for new features.
Seriously, binary protobuf is so simple anyone can parse it trivially. It’s just a flat sequence of pairs, of a message tag and the corresponding data, with 5 different wire types that do not specify any typing but instead only how to interpret the wire data (IIRC: variable length w length prefix, 4 byte intrger, variable integer, group start and group end, where ‘group’ refers to nested messages.) The wire type is encoded in the lowest 3 bits of the message tag. The message tag is written as a base128 variable length quantity, which is just an integer where each byte has a high bit specifying if there are more bytes, and low bits specifying binary data in least significant first order. The remainder of the bits are just the tag of the field from the proto file. The length prefix variant uses a second base128 vlq to specify length.
Protos are clever: the protobuf compiler itself compiles protobuf definitions into protobuf messages called descriptors that can be passed to a languages own protobuf compiler through standard in. These descriptors also get encoded into the resulting output because they can then be used to perform reflection.
Speaking of reflection, you can also have a bunch of metadata in the form of extensions and message/field/enum/etc. extension options. You can use these at runtime, or you can write custom protobuf plugins. I am doing both of these things simultaneously for different purposes in some projects; it helps me organize schema information and couple it with metadata.
I don’t think protos are perfect, but I do think they are clever and useful for what they are used for at Google. I actually personally suspect they are a bit underrated because outside of Google it’s not always completely obvious how to use protobufs to their fullest. That said, nothing’s perfect and protos are certainly full of weird quirks. But if you embrace them, I think there’s a lot of elegance to be found lurking beneath.
That is all. Disclaimer: I do work for Google, and admittedly I did not like protobuf until I started working here. But, now I sincerely like protobuf.