4 ms·

The type of protocol (message type, bitmap to define fields, followed by a set of fixed and variable length values) is pretty normal for the time it was develop

The type of protocol (message type, bitmap to define fields, followed by a set of fixed and variable length values) is pretty normal for the time it was developed in. Many low level things are basically packed C-structs with this type of protocol. It comes with some pitfalls on the receiver side to be careful validating dynamic field length and refusing to read past end of message or allocate an infinite buffer. But all of those are well understood by now. What I find baffling is that this "standard" does not specify how to encode the fields or even the message type. Pick anything, binary, ASCII, BCD, EBCDIC. That doesn't work as a standard, every implementation could send you nearly any random set of bytes with no way for the receiver to make sense of them.

mananaysiempre 2y ago
A bitmap to define field presence doesn’t seem so offensive, as far as serialization formats go. FlatBuffers[1] use a list of offsets instead, but that’s in the context of expecting to see many identically-shaped records. One could argue that Cap’n Proto with zero-packing[2] amounts to the same thing if you squint, just with the bitmap smeared all over the message. I mean, this specific thing sounds like it should have been a fatter but much more unexciting TLV affair instead. But given it’s from 1987, it’d probably have ended up as ASN.1 BER in that case (ETA: ah, and for extensions, it mostly did, what fun), instead of a simpler design like Protobufs or MessagePack/CBOR, so maybe the bitmaps are a blessing in disguise. [1] https://google.github.io/flatbuffers/flatbuffers_internals.html https://google.github.io/flatbuffers/flatbuffers_internals.h... [2] https://capnproto.org/encoding.html#packing https://capnproto.org/encoding.html#packing
- lxgr 2y ago
  I'd trade the field layer of ISO 8583 for some ASN.1 any day! Luckily, there's a bit of everything in the archeological site that is ISO 8583, and field 55 (where EMV chip data goes, and EMV itself is quite ASN.1-heavy, presumably for historical reasons) and some others in fact contain something very close to it :)
- grishka 2y ago
  Telegram's "TL" serialization, that's part of its network protocol, also uses a bitmap for optional fields. It's an okay protocol overall. The only problem is that the official documentation[1] was written by Nikolay Durov who is a mathematician. He just loves to overgeneralize everything to a ridiculous degree and spend two screens worth of outstandingly obtuse text to say what amounts to, for example, "some types have type IDs before them and some don't because the type is obvious form the schema". [1] https://core.telegram.org/mtproto https://core.telegram.org/mtproto
lxgr 2y ago
> Many low level things are basically packed C-structs with this type of protocol. Not really: C structs notably don't have variable-length fields, but ISO 8583 very much does. To add insult to injury, it does not offer a standard way to determine field lengths. This means that in order to ignore a given field, you'll need to be able to parse it (at least at the highest level)! Even ASN.1, not exactly the easiest format to deal with, is one step up from that (in a TLV structure, you can always skip unknown types by just skipping "length" bytes).
- david-gpu 2y ago
  > Not really: C structs notably don't have variable-length fields Feast your eyes: C99 introduced an ~~abomination~~ feature called flexible array members (FAM), which allows the last member of a struct to be a variable length array. If you want to ~~gouge you eyes~~ learn more, see section 6.7.2.1.16 [0]. > To add insult to injury, it does not offer a standard way to determine field lengths That's awful. You can sort of say the same about variable length struts in C, but at least the strict tupe definition usually has a field that tell you the length of the variable length array at the end. [0] https://rgambord.github.io/c99-doc/sections/6/7/2/1/index.html#p16 https://rgambord.github.io/c99-doc/sections/6/7/2/1/index.ht...
  - j16sdiz 2y ago
    FAM was a (not so successful) attempt to standardise some existing usage
  - lxgr 2y ago
    > [...] feature called flexible array members (FAM), which allows the last member of a struct to be a variable length array. Oh, ISO 8583 has these too! Sometimes they're even combined with the "feature" described in the article where there's a variable number of fixed-length elements, except for the last element, which is a variable-length string (or sometimes the last field type repeated n times). That's always "fun" to work with. ISO 8583 really is a living museum of all ideas people had about binary encoding in the last half century or so.
quotemstr 2y ago
As far as I'm concerned, we solved binary formats with ASN.1 and its various encodings. Everything afterwards has been NIH, ignorance, and square wheel reinvention.
- wbl 2y ago
  ASN.1 DER, BER, or OER? Implicit and optional can really break compat in surprising ways. Then there are the machine unfriendly roster of available types. XDR was more tuned for that. Finally free tooling doesn't really exist. The connection to the OSI model also didn't help.
  - quotemstr 2y ago
    > ASN.1 DER, BER, or OER? Or XER or JER! One of the brilliant things about ASN.1 is that it decouples the data model from the serialization format. Of the major successor systems, only protobuf does something similar, and the text proto format barely counts. > Implicit and optional can really break compat in surprising ways Any implementation of any spec can be broken. You could argue that the spec should be simpler and require, e.g., explicit tagging everywhere, like protobuf. Sure. But the complexity enables efficiencies, and it's sometimes worth making a foundational library a bit more complex to enable simplifications and optimizations throughout the ecosystem. > Then there are the machine unfriendly roster of available type Protobuf's variable-length integers are machine-friendly now? :-) We can always come up with better encoding rules without changing the fundamental data structures. > Finally free tooling doesn't really exist. What do you mean? You use ASN.1 to talk to every server talking SNMP, LDAP, or the X.509 PKI. Every programming environment has a way to talk about ASN.1. > The connection to the OSI model also didn't help. Agreed. The legacy string types aren't great either. You can, of course, do ASN.1 better. No technology is perfect. But what we don't need, IMHO, is more investment in "simple" technologies like varlink that end up being too simple and shunting complexity and schema-ness that belongs in a universal foundation layer into every single application using the new "simple" thing.
    - devman0 2y ago
      XML also decouples the data model and serialization with the XML Infoset specification.
immibis 2y ago
Something similar is TLV which is extremely common in binary network protocols, because it's very flexible for compatibility.

mananaysiempre 2y ago

A bitmap to define field presence doesn’t seem so offensive, as far as serialization formats go. FlatBuffers[1] use a list of offsets instead, but that’s in the context of expecting to see many identically-shaped records. One could argue that Cap’n Proto with zero-packing[2] amounts to the same thing if you squint, just with the bitmap smeared all over the message. I mean, this specific thing sounds like it should have been a fatter but much more unexciting TLV affair instead. But given it’s from 1987, it’d probably have ended up as ASN.1 BER in that case (ETA: ah, and for extensions, it mostly did, what fun), instead of a simpler design like Protobufs or MessagePack/CBOR, so maybe the bitmaps are a blessing in disguise. [1] https://google.github.io/flatbuffers/flatbuffers_internals.html https://google.github.io/flatbuffers/flatbuffers_internals.h... [2] https://capnproto.org/encoding.html#packing https://capnproto.org/encoding.html#packing

lxgr 2y ago
I'd trade the field layer of ISO 8583 for some ASN.1 any day! Luckily, there's a bit of everything in the archeological site that is ISO 8583, and field 55 (where EMV chip data goes, and EMV itself is quite ASN.1-heavy, presumably for historical reasons) and some others in fact contain something very close to it :)
grishka 2y ago
Telegram's "TL" serialization, that's part of its network protocol, also uses a bitmap for optional fields. It's an okay protocol overall. The only problem is that the official documentation[1] was written by Nikolay Durov who is a mathematician. He just loves to overgeneralize everything to a ridiculous degree and spend two screens worth of outstandingly obtuse text to say what amounts to, for example, "some types have type IDs before them and some don't because the type is obvious form the schema". [1] https://core.telegram.org/mtproto https://core.telegram.org/mtproto

lxgr 2y ago

> Many low level things are basically packed C-structs with this type of protocol. Not really: C structs notably don't have variable-length fields, but ISO 8583 very much does. To add insult to injury, it does not offer a standard way to determine field lengths. This means that in order to ignore a given field, you'll need to be able to parse it (at least at the highest level)! Even ASN.1, not exactly the easiest format to deal with, is one step up from that (in a TLV structure, you can always skip unknown types by just skipping "length" bytes).

david-gpu 2y ago
> Not really: C structs notably don't have variable-length fields Feast your eyes: C99 introduced an ~~abomination~~ feature called flexible array members (FAM), which allows the last member of a struct to be a variable length array. If you want to ~~gouge you eyes~~ learn more, see section 6.7.2.1.16 [0]. > To add insult to injury, it does not offer a standard way to determine field lengths That's awful. You can sort of say the same about variable length struts in C, but at least the strict tupe definition usually has a field that tell you the length of the variable length array at the end. [0] https://rgambord.github.io/c99-doc/sections/6/7/2/1/index.html#p16 https://rgambord.github.io/c99-doc/sections/6/7/2/1/index.ht...
- j16sdiz 2y ago
  FAM was a (not so successful) attempt to standardise some existing usage
- lxgr 2y ago
  > [...] feature called flexible array members (FAM), which allows the last member of a struct to be a variable length array. Oh, ISO 8583 has these too! Sometimes they're even combined with the "feature" described in the article where there's a variable number of fixed-length elements, except for the last element, which is a variable-length string (or sometimes the last field type repeated n times). That's always "fun" to work with. ISO 8583 really is a living museum of all ideas people had about binary encoding in the last half century or so.

quotemstr 2y ago

As far as I'm concerned, we solved binary formats with ASN.1 and its various encodings. Everything afterwards has been NIH, ignorance, and square wheel reinvention.

wbl 2y ago
ASN.1 DER, BER, or OER? Implicit and optional can really break compat in surprising ways. Then there are the machine unfriendly roster of available types. XDR was more tuned for that. Finally free tooling doesn't really exist. The connection to the OSI model also didn't help.
- quotemstr 2y ago
  > ASN.1 DER, BER, or OER? Or XER or JER! One of the brilliant things about ASN.1 is that it decouples the data model from the serialization format. Of the major successor systems, only protobuf does something similar, and the text proto format barely counts. > Implicit and optional can really break compat in surprising ways Any implementation of any spec can be broken. You could argue that the spec should be simpler and require, e.g., explicit tagging everywhere, like protobuf. Sure. But the complexity enables efficiencies, and it's sometimes worth making a foundational library a bit more complex to enable simplifications and optimizations throughout the ecosystem. > Then there are the machine unfriendly roster of available type Protobuf's variable-length integers are machine-friendly now? :-) We can always come up with better encoding rules without changing the fundamental data structures. > Finally free tooling doesn't really exist. What do you mean? You use ASN.1 to talk to every server talking SNMP, LDAP, or the X.509 PKI. Every programming environment has a way to talk about ASN.1. > The connection to the OSI model also didn't help. Agreed. The legacy string types aren't great either. You can, of course, do ASN.1 better. No technology is perfect. But what we don't need, IMHO, is more investment in "simple" technologies like varlink that end up being too simple and shunting complexity and schema-ness that belongs in a universal foundation layer into every single application using the new "simple" thing.
  - devman0 2y ago
    XML also decouples the data model and serialization with the XML Infoset specification.

immibis 2y ago

Something similar is TLV which is extremely common in binary network protocols, because it's very flexible for compatibility.