4 ms·
MS-RPC NDR IDL, ONC/RPC XDR, ASN.1, and many, many others. There aren't too many variations. You can have "self-describing" formats, which is generally short
by cryptonector 6y ago
MS-RPC NDR IDL, ONC/RPC XDR, ASN.1, and many, many others. There aren't too many variations.
You can have "self-describing" formats, which is generally short for tag-length-value (TLV) formats where the tag identifies the type of the encoded value, and the value is an octet string of the given length encoding a value of the given type tag. This includes ASN.1's BER/DER/CER. Protobufs is a TLV encoding.
Or you can have a more "binary" format with no tags and lengths elided for fields with fixed-sized encodings or fields whose variable-length value encodings are self-describing as to length (e.g., various varint formats). This includes NDR, XDR, ASN.1's PER and OER. Flatbuffers is a variation on this theme.
Or you can have textual encodings like XML and JSON.
There's not a lot of other options.
Each has its pros and cons, but, really, the packed encoding type (NDR, XDR, PER, OER, flatbuffers) is generally superior. That's because TLV schemes are redundant and therefore bloated, and because you really need to have the schema at hand (or have code generated from the schema, which is a different way of having the schema at hand) no matter what, so "self-describing" just isn't terribly useful. If you've ever seen the output of "asn1print" type programs you'll understand that you still need to know the schema in your head to understand the output, so might as well have a dumper/pretty-printer that can use a schema to really pretty-print...
Textual encodings generally end up getting binary / optimized variants. E.g., XML has FastInfoSet (which is essentially a transcoding to ASN.1 with PER!). JSON has many binary formats, including several "JSONB" schemes (e.g., PostgreSQL's), and CBOR. This is because at the end of the day, textual formats are hard to parse -- they really suck. No one needs to build a SIMD parser for NDR/XDR/PER/OER/flatbuffers/CBOR/JSONB, but JSON is so hard to parse quickly that SIMD parsers for it exist.
Besides the many standards, I've seen quite a few ad-hoc schemes, such as Lustre's RPC format (basically flatbuffers, but without an IDL compiler and code generator), many Perl5 DataDumper-type modules, Lisp S-expressions (the grand-daddy of data encoding formats!), and many others.
In a few years we might see a new round of bad reinventions of this wheel. This happens because inexperienced people find that scheme X "sucks" or there's no tooling for scheme X in the hot new programming language yet, and rather than read the specs for X and implement their own tooling they go off and make up a new scheme as they implement it.
ASN.1 gets a lot of hate, but mostly the issue is with the TLV-type encoding rules, but there are many more encoding rules than just BER (and the DER and CER variants of BER): there's binary-style non-TLV ERs like PER and OER, there's XER (XML!), JER (JSON!), and even GSER (a textual encoding just for LDAP), and anyone can create new encoding rules if they really want to. Another reason ASN.1 gets hated on is lack of tooling, but every time anyone invents a new scheme... they have to write new tooling anyways -- what a waste.
Note: I'm not saying we should only be using ASN.1 or anything of the sort. I'm saying we need to really understand what came before so we don't make the same mistakes again. Most importantly we need to understand that the T and often the L in TLV are just a lame crutch that bloats data on the wire and makes it easy to hand-roll bug-riddled codecs.
- Serow225 6y agoNice summary!! :)
- cryptonector 6y agoThanks!
- athenot 6y agoWhat I do appreciate about TLV-style formats is general-purpose debugging tools. When all you have is a bitstream where field lengths are not multiples of 8 bits and are variable based on the actual value being encoded (looking at you, MPEG), the toolset to debug streams ends up being non-trivial.
- cryptonector 6y agoIf the IDL compiler is more like a library then you can have an "asn1print" that takes a schema and dumps the data correctly with all the metadata you need and want and none of the crappy TLV.
- svachalek 6y agoHaven’t used protobufs for years but as I remember it you can use an old schema to read a newer version object, discarding the new fields. I don’t think that would be possible without the TLV.
- mumblemumble 6y agoflatbuffers also support this, but use offset tables instead of TLV. It may be that TLV is necessary if you want to do this in a format that is designed for streaming rather than random access.
- cryptonector 6y agoThis is called "extensibility". In ASN.1 people were doing the same thing unofficially in the mid-80s, then explicit support was added in the form of "extensibility markers". There's nothing new in protobufs. EDIT: I should add that the non-TLV encoding rules for ASN.1 all support extensibility. Also, there are good engineering practices and security reasons why one should want explicit documentation of what structures/enums/choices/typeranges are expected to be extensible. Relying on decoders to ignore unexpected bits is not a sound practice.