4 ms·

protobufs / asn.1 / structs ... Edit: hell even XML is better than this!

by nbbnbb 2y ago

baq 2y ago
Thank goodness they didn’t pick YAML though.
- nbbnbb 2y ago
  Yet!
RandomThoughts3 2y ago
Structs are a part of C semantic. They are not an ipc format. You can somewhat use them like one if you take a lot of precaution about how they are laid out in memory including padding and packing but it’s very brittle. Asn.1 is both quite complicated and not very efficient. They could certainly have gone with protobufs or another binary serialisation format but would it really be better than the current choice? I don’t think the issue they are trying to solve is related to serialisation anyway. Seems to me they are unhappy about the bus part not the message format part.
- dfox 2y ago
  ASN.1 BER/DER is more or less the same thing as CBOR. The perceived complexity of ASN.1 comes from the schema language and specifications written in the convoluted telco/ITU-T style (and well, the 80's type system that has ~8 times two different types for “human readable string”).
  - RandomThoughts3 2y ago
    > The perceived complexity of ASN.1 comes from the schema language and specifications written in the convoluted telco/ITU-T style (and well, the 80's type system that has ~8 times two different types for “human readable string”). I can’t resist pointing that it’s basically a longer way of saying quite complicated and not very efficient.
    - cryptonector 2y ago
      > I can’t resist pointing that it’s basically a longer way of saying quite complicated and not very efficient. That's very wrong. ASN.1 is complicated because it's quite complete by comparison to other syntaxes, but it's absolutely not inefficient unless you mean BER/DER/CER, but those are just _some_ of the encoding rules available for use with ASN.1. To give just one example of "complicated", ASN.1 lets you specify default values for optional members (fields) of SEQUENCEs and SETs (structures), whereas Protocol Buffers and XDR (to give some examples) only let you specify optional fields but not default values. Another example of "complicated" is that ASN.1 has extensibility rules because the whole "oh TLV encodings are inherently extensible" thing turned out to be a Bad Idea (tm) when people decided that TLV encodings were unnecessarily inefficient (true!) so they designed efficient, non-TLV encodings. Well guess what: Protocol Buffers suffers from extensibility issues that ASN.1 does not, and that is a serious problem. Basically, with a subset of ASN.1 you can do everything that you can do with MSFT RPC's IDL, with XDR, with Protocol Buffers, etc. But if you stick to a simple subset of ASN.1, or to any of those other IDLs, then you end up having to write _normative_ natural language text (typically English) in specifications to cover all the things not stated in the IDL part of the spec. The problem with that is that it's easy to miss things or get them wrong, or to be ambiguous. ASN.1 in its full flower of "complexity" (all of X.680 plus all of X.681, X.682, and X.683) lets you express much more of your protocols in a _formal_ language. I maintain an ASN.1 compiler. I've implemented parts of X.681, X.682, and X.683 so that I could have the compiler generate code for the sorts of typed holes you see in PKI -all the extensions, all the SANs, and so on- so that the programmer can do much less of the work of having to invoke a codec for each of those extensions. A lot of the complexity in ASN.1 is optional, but it's very much worth at least knowing about it. Certainly it's worth not repeating mistakes of the past. Protocol Buffers is infuriating. Not only is PB a TLV encoding (why? probably because "extensibility is easy with TLV!!1!, but that's not quite true), but the IDL requires manual assignment of tag values, which makes uses of the PB IDL very ugly. ASN.1 originally also had the manual assignment of tags problem, but eventually ASN.1 was extended to not require that anymore. Cavalier attitudes like "ASN.1 is too complicated" lead to bad results.
      - RandomThoughts3 2y ago
  - cryptonector 2y ago
    That "convoluted telco/ITU-T style" yields amazingly high quality specifications. I'll take X.68x / X.69x any day over most Internet RFCs (and I've written a number of Internet RFCs). The ITU-T puts a great deal of effort into its specs, or at least the ASN.1 working group did. ASN.1 is not that complicated. Pity that fools who thought ASN.1 was complicated re-invented the wheel quite poorly (Protocol Buffers I'm looking at you).
    - quotemstr 2y ago
      For our sins, our industry is doomed to suffer under unbearable weight of endless reinvented wheels. Of course it would have been better to stick with ASN.1. Of course we didn't, because inexperience and hubris. We'll never learn.
      - cryptonector 2y ago
        It sure seems that way. Sad. It's not just hubris nor inexperience -- it's cognitive load. It's often easier to wing something that later grows a lot than it is to go find a suitable technology that already exists.
      - jabl 2y ago
        One thing I liked about a Vernor Vinge sci-fi novel I read once was the concept of "computer archeologist". Spool the evolution of software forwards a few centuries, and we'll have layers upon layers of software where instead of solving problems with existing tooling, we just plaster on yet another NIH layer. Rinse and repeat, and soon enough we'll need a separate profession of people who are capable of digging down into those old layers and figure out how they work.
- cryptonector 2y ago
  > Structs are a part of C semantic. Uh, no, structs, records, whatever you want to call them, are in many, if not most programming languages. "Structs" is not just "C structs" -- it's just shorthand for "structured data types" (same as in C!). > Asn.1 is both quite complicated and not very efficient. Every rich encoding system is complicated. As for efficiency, ASN.1 has many encoding rules, some of which are quite bad (BER/DER/CER, which are the first family of ERs, and so many thing ASN.1 == BER/DER/CER, but that's not the case), and some of which are very efficient (PER, OER). Heck, you can use XML and JSON as ASN.1 encoding rules (XER, JER). > They could certainly have gone with protobufs or another binary serialisation format but would it really be better than the current choice? Protocol buffers is a tag-length-value (TLV) encoding, same as BER/DER/CER. Having to have a tag and length for every value encoded is very inefficient, both in terms of encoding size as well as in terms of computation. The better ASN.1 ERs -PER and OER- are much more akin to XDR and Flat buffers than to protobufs. > I don’t think the issue they are trying to solve is related to serialisation anyway. Seems to me they are unhappy about the bus part not the message format part. This definitely seems to be be the case.
  - Netch 2y ago
    It seems you posit taglessness to be a universal crucial merit of any encoding scheme. This is good in an ideal world, heh. I have had a misfortune to work for 5G which is full of PER-encoded protocols. Dealing with discrepancies in them - incompatible changes in 3GPP standard versions, different vendorsʼ errors, combined with usually low level of developers and managers in a typical corporation - was an utter nightmare. IETF, in general, provides a good policy combining truly fixed binary protocols when they are unavoidable (IP/TCP/UDP levels) and flexible, often text, protocols where there is no substantial overhead from their use. Their early moves, well, suffered from over-grammaticalization (as RFC822). CBOR is nice here because it combines tagness and compactness. 3-bit basic tag combined with value (if fit) or length, it is commensurable with OER in efficiency but is decodable without scheme - and it is extremely useful in practice.
  - RandomThoughts3 2y ago
    > Uh, no, structs, records, whatever you want to call them It's plenty clear from discussion context that OP is talking about C struct but yes, replace C with any languages which suit you. It will still be part of the language semantic and not an IPC specification. The point is you can't generally use memory layout as an IPC protocol because you generally have no guarantee that it will be the same for all architectures.
otabdeveloper4 2y ago
You don't quite understand how this works. One requirement is being able to strace a misbehaving service and figure out quickly what it's sending and receiving. This is a system-level protocol, not just a random user app.

protobufs / asn.1 / structs ... Edit: hell even XML is better than this!

by nbbnbb 2y ago

protobufs / asn.1 / structs ... Edit: hell even XML is better than this!

baq 2y ago

Thank goodness they didn’t pick YAML though.

nbbnbb 2y ago
Yet!

RandomThoughts3 2y ago

Structs are a part of C semantic. They are not an ipc format. You can somewhat use them like one if you take a lot of precaution about how they are laid out in memory including padding and packing but it’s very brittle. Asn.1 is both quite complicated and not very efficient. They could certainly have gone with protobufs or another binary serialisation format but would it really be better than the current choice? I don’t think the issue they are trying to solve is related to serialisation anyway. Seems to me they are unhappy about the bus part not the message format part.

dfox 2y ago
ASN.1 BER/DER is more or less the same thing as CBOR. The perceived complexity of ASN.1 comes from the schema language and specifications written in the convoluted telco/ITU-T style (and well, the 80's type system that has ~8 times two different types for “human readable string”).
- RandomThoughts3 2y ago
  > The perceived complexity of ASN.1 comes from the schema language and specifications written in the convoluted telco/ITU-T style (and well, the 80's type system that has ~8 times two different types for “human readable string”). I can’t resist pointing that it’s basically a longer way of saying quite complicated and not very efficient.
  - cryptonector 2y ago
    > I can’t resist pointing that it’s basically a longer way of saying quite complicated and not very efficient. That's very wrong. ASN.1 is complicated because it's quite complete by comparison to other syntaxes, but it's absolutely not inefficient unless you mean BER/DER/CER, but those are just _some_ of the encoding rules available for use with ASN.1. To give just one example of "complicated", ASN.1 lets you specify default values for optional members (fields) of SEQUENCEs and SETs (structures), whereas Protocol Buffers and XDR (to give some examples) only let you specify optional fields but not default values. Another example of "complicated" is that ASN.1 has extensibility rules because the whole "oh TLV encodings are inherently extensible" thing turned out to be a Bad Idea (tm) when people decided that TLV encodings were unnecessarily inefficient (true!) so they designed efficient, non-TLV encodings. Well guess what: Protocol Buffers suffers from extensibility issues that ASN.1 does not, and that is a serious problem. Basically, with a subset of ASN.1 you can do everything that you can do with MSFT RPC's IDL, with XDR, with Protocol Buffers, etc. But if you stick to a simple subset of ASN.1, or to any of those other IDLs, then you end up having to write _normative_ natural language text (typically English) in specifications to cover all the things not stated in the IDL part of the spec. The problem with that is that it's easy to miss things or get them wrong, or to be ambiguous. ASN.1 in its full flower of "complexity" (all of X.680 plus all of X.681, X.682, and X.683) lets you express much more of your protocols in a _formal_ language. I maintain an ASN.1 compiler. I've implemented parts of X.681, X.682, and X.683 so that I could have the compiler generate code for the sorts of typed holes you see in PKI -all the extensions, all the SANs, and so on- so that the programmer can do much less of the work of having to invoke a codec for each of those extensions. A lot of the complexity in ASN.1 is optional, but it's very much worth at least knowing about it. Certainly it's worth not repeating mistakes of the past. Protocol Buffers is infuriating. Not only is PB a TLV encoding (why? probably because "extensibility is easy with TLV!!1!, but that's not quite true), but the IDL requires manual assignment of tag values, which makes uses of the PB IDL very ugly. ASN.1 originally also had the manual assignment of tags problem, but eventually ASN.1 was extended to not require that anymore. Cavalier attitudes like "ASN.1 is too complicated" lead to bad results.
    - RandomThoughts3 2y ago
- cryptonector 2y ago
  That "convoluted telco/ITU-T style" yields amazingly high quality specifications. I'll take X.68x / X.69x any day over most Internet RFCs (and I've written a number of Internet RFCs). The ITU-T puts a great deal of effort into its specs, or at least the ASN.1 working group did. ASN.1 is not that complicated. Pity that fools who thought ASN.1 was complicated re-invented the wheel quite poorly (Protocol Buffers I'm looking at you).
  - quotemstr 2y ago
    For our sins, our industry is doomed to suffer under unbearable weight of endless reinvented wheels. Of course it would have been better to stick with ASN.1. Of course we didn't, because inexperience and hubris. We'll never learn.
    - cryptonector 2y ago
      It sure seems that way. Sad. It's not just hubris nor inexperience -- it's cognitive load. It's often easier to wing something that later grows a lot than it is to go find a suitable technology that already exists.
    - jabl 2y ago
      One thing I liked about a Vernor Vinge sci-fi novel I read once was the concept of "computer archeologist". Spool the evolution of software forwards a few centuries, and we'll have layers upon layers of software where instead of solving problems with existing tooling, we just plaster on yet another NIH layer. Rinse and repeat, and soon enough we'll need a separate profession of people who are capable of digging down into those old layers and figure out how they work.
cryptonector 2y ago
> Structs are a part of C semantic. Uh, no, structs, records, whatever you want to call them, are in many, if not most programming languages. "Structs" is not just "C structs" -- it's just shorthand for "structured data types" (same as in C!). > Asn.1 is both quite complicated and not very efficient. Every rich encoding system is complicated. As for efficiency, ASN.1 has many encoding rules, some of which are quite bad (BER/DER/CER, which are the first family of ERs, and so many thing ASN.1 == BER/DER/CER, but that's not the case), and some of which are very efficient (PER, OER). Heck, you can use XML and JSON as ASN.1 encoding rules (XER, JER). > They could certainly have gone with protobufs or another binary serialisation format but would it really be better than the current choice? Protocol buffers is a tag-length-value (TLV) encoding, same as BER/DER/CER. Having to have a tag and length for every value encoded is very inefficient, both in terms of encoding size as well as in terms of computation. The better ASN.1 ERs -PER and OER- are much more akin to XDR and Flat buffers than to protobufs. > I don’t think the issue they are trying to solve is related to serialisation anyway. Seems to me they are unhappy about the bus part not the message format part. This definitely seems to be be the case.
- Netch 2y ago
  It seems you posit taglessness to be a universal crucial merit of any encoding scheme. This is good in an ideal world, heh. I have had a misfortune to work for 5G which is full of PER-encoded protocols. Dealing with discrepancies in them - incompatible changes in 3GPP standard versions, different vendorsʼ errors, combined with usually low level of developers and managers in a typical corporation - was an utter nightmare. IETF, in general, provides a good policy combining truly fixed binary protocols when they are unavoidable (IP/TCP/UDP levels) and flexible, often text, protocols where there is no substantial overhead from their use. Their early moves, well, suffered from over-grammaticalization (as RFC822). CBOR is nice here because it combines tagness and compactness. 3-bit basic tag combined with value (if fit) or length, it is commensurable with OER in efficiency but is decodable without scheme - and it is extremely useful in practice.
- RandomThoughts3 2y ago
  > Uh, no, structs, records, whatever you want to call them It's plenty clear from discussion context that OP is talking about C struct but yes, replace C with any languages which suit you. It will still be part of the language semantic and not an IPC specification. The point is you can't generally use memory layout as an IPC protocol because you generally have no guarantee that it will be the same for all architectures.

otabdeveloper4 2y ago

You don't quite understand how this works. One requirement is being able to strace a misbehaving service and figure out quickly what it's sending and receiving. This is a system-level protocol, not just a random user app.