4 ms·
The formats are popular because they were popular in closed-source software. And they were popular in closed-source software because there were (and still are)
by wahern 8y ago
The formats are popular because they were popular in closed-source software. And they were popular in closed-source software because there were (and still are) good commercial parser generators for ASN.1.
ASN.1 has been a failure in open source because there weren't any good parser generators. The only open source ASN.1 generator for C code I'm familiar with is asn1c[1], which was published long after OpenSSL and other projects added their ad hoc certificate parsing code. I think there may be one or two for Java, but that's about it.
Moreover, open source projects have historically disfavored using parser generators. They don't like the dependency, and there's still the sense that good protocols shouldn't need parser generators--contrast commercial protocols like X.whatever with SMTP, HTTP, etc.
ASN.1-based formats were never intended to be parsed using hand written code. Abstraction Syntax Notation refers to the grammar for specifying the wire-line formats.
ASN.1 is solid technology. The technical debt exists because open source tooling never developed around it. At first the community thought it was too complicated and unnecessary. Then when the need arose the community simply reinvented the wheel (Protocol Buffers, etc).
[1] asn1c is amazing, BTW. Not only will it generate encoders and decoders given the ASN.1 specification, but it can generate streaming encoders and decoders, something that most open source alternatives (e.g. Protocol Buffers) can't do. (And by streaming I mean streaming a single message, which is important for low-memory environments, either because of minimal hardware resources, as a performance optimization, or as a security constraint.)
- cryptonick 8y agoI agree with Your claim about ASN.1 not intended to be parsed using hand written code. The ASN.1 is indeed quite close to a grammar specification, as also shown in the paper. However, I believe that a major issue related to its parsing complexity is the binary encoding generally used, which is either BER or DER: both of them employing length fields. While the usage of length fields is usually ubiquitous in formats related to communications protocols, these fields are quite annoying to be handled from a grammar design perspective. Indeed, a length field requires to count bytes of the payload: this operation is tedious to be done with grammars, while it is extremely easy with hand written code, in turn generally making the usage of grammar based automatic parser generators for these formats a less common choice. From a grammar design standpoint, a delimiter based structure would be preferable. For instance, in the context of X.509, we proposed a new format[1] which replaces DER encoding with a new format where there are no longer length fields but the payload is terminated by a fixed delimiter. The grammar for this format was way easier than the length field base encoding, without requiring any hand written code. [1] A novel regular fomat for X.509 digital certificates, https://link.springer.com/chapter/10.1007/978-3-319-54978-1_18 https://link.springer.com/chapter/10.1007/978-3-319-54978-1_...
- wahern 8y agoI believe I read that paper recently :) Ultimately I ended up using PEGs. LPeg in particular, using LPeg's match time captures to recursively invoke PEGs for length-encoded objects. (In addition to the match time capture extension, what's especially nice about LPeg--missing from every other PEG library I've seen--is that you can build and transform the AST in one shot.) I've also tentatively rejected translation to a format like that proposed in the paper. In a secure enclave-like environment I'd rather be dealing with statically defined C-like structs with stronger invariants--i.e. no optional or sum types, no variable-length fields; basically, no need for any kind of parsing whatsoever. If I have to transform, I'd like to transform the message both syntactically and semantically into the simplest possible form. Parsing complexity is only part of the equation. The other part is semantic complexity, which is a different kind of problem that better formats and parsers can't fix. In another timeline things could have been different, but we don't live in that timeline :( We can't let perfect be the enemy of good. Even if we could move away from DER or even ASN.1 in the open source world, the entire telecommunications industry (and specifically the cell industry) is built around ASN.1 and DER/PER/XER. AFAICT the biggest users of asn1c are people working with 3GPP and similar standards. No matter how sane and secure we can make our open source ecosystems, ASN.1 and similar older tech will still lurk in the background, remaining the weakest link in the chain. If we want real security we have no choice but to develop better tooling in that regard. I appreciate your proposal is very much of that mindset, I'm just not sold on the practical utility.