3 ms·

ASN.1 is a bit unfortunate. On the one hand, I want to believe: it's a fairly ubiquitous standard (at least, as long as you're dealing with cryptography or a ha

by lvh 12y ago

ASN.1 is a bit unfortunate. On the one hand, I want to believe: it's a fairly ubiquitous standard (at least, as long as you're dealing with cryptography or a handful of other specific fields), and it's true that it can be a lot more compact than JSON or XML (see below), and it's even occasionally faster. However: 1. A lot of ASN.1 software is pretty buggy and undermaintained. How do I submit a bug to pyasn1? Because it's entrenched in libraries three or four layers below what most developers will see, it's also difficult to replace. 2. Faster and more compact depends on the encoding rules. Let's talk about the dozen or so different ways you have of encoding ASN.1. BER? DER? Maybe CER? XER? Why not both: CXER! I'm missing a bunch, I know. 3. The performance and network arguments ignore what you can do with compressed verbose formats like JSON, XML or EDN. lz4 is magic. Even totally ubiquitous gz compression works extraordinarily well. 4. While ASN.1 messages do contain descriptions of what types they contain (e.g., you can see that there's a bit string coming), they aren't self-describing in the same way that JSON or XML are, which is quite annoying for debugging. You can make self-describing messages using XML, but at that point you're literally just doing XML. Good luck finding software that lets you easily use whichever you like -- even if you subscribe to the notion that easy debuggability doens't matter for production messages, in which case I strongly disagree. 5. I'd harp about its extensibility, but it's really no better than JSON, so I won't. All-in-all, I'm compressing some EDN, and I'm pretty happy.

derefr 12y ago
> Faster and more compact depends on the encoding rules. Sure; but in reality, you just write your from-scratch systems to use DER, and then throw in a "protocol-upgrade to PER" message/bitflag if things aren't going fast enough. And then if some crazy legacy SOAP+WSDL enterprise wants to integrate their codebase with yours, you add the ability to negotiate XER just for when they're speaking to you. It's not like all the different encodings are equally valid choices when both nodes are under your control; they fall pretty clearly on a spectrum from "whenever you can get away with it" down to "only if you have to." > The performance and network arguments ignore what you can do with compressed verbose formats... Wire compression doesn't help if you're sending short intermittent messages across millions of connections, rather than batch-streaming messages across a link. Specifically in the GSM context mentioned in the article, the average control-channel ASN.1 message won't be helped at all by wire compression. Also, this isn't an argument in favor of ASN.1, but sometimes you need your wire message format to be efficient not because of throughput constraints, but because you need to directly manipulate the resulting data in memory (ala Cap'n Proto), and so efficiency on the wire directly translates to efficiency of packing in memory. The protocol can be compressed on the wire on top of this, of course, but it implies that "just use wire compression" won't solve every efficiency problem. > they aren't self-describing in the same way that JSON or XML are ASN.1 comes from an era of record-oriented storage, where a schema is transmitted once, in advance—or possibly baked into the client—and then rows defined by the schema are transferred many times. XML does this well for individual self-describing messages by allowing for a reference to a DTD+XML schema, but ASN.1, being for streams of records, expects you to create a message-container format to reference your schema, so as to remove the overhead of repeatedly mentioning it on the wire. This last one makes a more general point for me: some protocols reward good engineering done in the code that uses them, and punish bad engineering. If you're trying to take the naive-and-suboptimal approach to something and a protocol is fighting with you, it might be because the protocol "wants" to be used in an optimal fashion.
tokenrove 12y ago
I feel the same way about ASN.1. I remember implementing some ASN.1 tools as part of an SNMP engine and thinking "how could anyone possibly implement all of this correctly?" -- a few years later, all those vulnerabilities in SNMP products at the ASN.1 encoding level came to light. On the other hand, it's sad to see efficient on-the-wire encoding ignored, and the ASN.1 standards aren't really abominations when you compare them with some of the web services standards. In addition to what you mentioned, I wanted to mention EXI (and FAST, the FIX encoding) as interesting on-the-wire encodings that maybe more people should consider over just compressed JSON or XML. Generic LZ-based compression doesn't necessarily win very much with lots of short messages.
- abecedarius 12y ago
  I had a similar feeling making an SNMP MIB processor around the same time (2000?) -- the ASN.1 libraries so complicated. So I wrote my own, which turned out simple and symmetrical just by encoding from back to front instead of the messy front-to-back-and-backpatch everyone else was doing. (And I guess by leaving some things out that we didn't use, like other encoding rules? I don't remember.) When those vulnerabilities hit I never found out how that code did -- I wish they'd open-sourced it as they'd planned. So, that particular part (DER? again I forget) seemed tolerable to me. The newer stuff like Cap'n Proto is probably still better.
  - tptacek 12y ago
    Yes, this! This is the secret to simple BER/DER encoding: back-to-front. I was almost converted to ASN.1 BER after discovering this, but in the intervening 8 years the spell has (thankfully) worn off and I can see it for the clattering technological jalopy that it is. The features in the author's F# ASN.1 compiler are pretty swank. ASN.1 probably gets a bad rap because of BER/DER.
- jzwinck 12y ago
  FAST isn't exactly a FIX encoding, it's a key-value encoding that can be translated to/from FIX. Roughly speaking, FAST is to FIX as BSON is to JSON. But starting to use FAST today seems like a bad idea, because the largest publicly-known production users of FAST have already moved away from it (toward plain, uncompressed binary structs on the wire).
DominikD 12y ago
EDN ain't half bad, true. It addresses some of the JSON shortcomings (like: lack of comments) and is still readable and easy to parse. And I have to agree with ASN.1 middleware - most of it is old and of low quality. But this can be said about pretty much everything that's been around before the ubiquitousness of the Internet (i.e. when developers assumed good faith and hacking wasn't even a topic to be discussed). And yes, some of the X- encodings for ASN.1 should die in flames (alongside XML in general; flames are welcome ;>). It's still a pretty neat binary format if one uses BER though. But when it comes to binary formats, I really like everything IFF-like. It's similar to ASN.1 in terms of how data is structured, but uses far simpler rules for headers and payload. It's much harder to botch the implementation.

by lvh 12y ago

derefr 12y ago

> Faster and more compact depends on the encoding rules. Sure; but in reality, you just write your from-scratch systems to use DER, and then throw in a "protocol-upgrade to PER" message/bitflag if things aren't going fast enough. And then if some crazy legacy SOAP+WSDL enterprise wants to integrate their codebase with yours, you add the ability to negotiate XER just for when they're speaking to you. It's not like all the different encodings are equally valid choices when both nodes are under your control; they fall pretty clearly on a spectrum from "whenever you can get away with it" down to "only if you have to." > The performance and network arguments ignore what you can do with compressed verbose formats... Wire compression doesn't help if you're sending short intermittent messages across millions of connections, rather than batch-streaming messages across a link. Specifically in the GSM context mentioned in the article, the average control-channel ASN.1 message won't be helped at all by wire compression. Also, this isn't an argument in favor of ASN.1, but sometimes you need your wire message format to be efficient not because of throughput constraints, but because you need to directly manipulate the resulting data in memory (ala Cap'n Proto), and so efficiency on the wire directly translates to efficiency of packing in memory. The protocol can be compressed on the wire on top of this, of course, but it implies that "just use wire compression" won't solve every efficiency problem. > they aren't self-describing in the same way that JSON or XML are ASN.1 comes from an era of record-oriented storage, where a schema is transmitted once, in advance—or possibly baked into the client—and then rows defined by the schema are transferred many times. XML does this well for individual self-describing messages by allowing for a reference to a DTD+XML schema, but ASN.1, being for streams of records, expects you to create a message-container format to reference your schema, so as to remove the overhead of repeatedly mentioning it on the wire. This last one makes a more general point for me: some protocols reward good engineering done in the code that uses them, and punish bad engineering. If you're trying to take the naive-and-suboptimal approach to something and a protocol is fighting with you, it might be because the protocol "wants" to be used in an optimal fashion.

tokenrove 12y ago

I feel the same way about ASN.1. I remember implementing some ASN.1 tools as part of an SNMP engine and thinking "how could anyone possibly implement all of this correctly?" -- a few years later, all those vulnerabilities in SNMP products at the ASN.1 encoding level came to light. On the other hand, it's sad to see efficient on-the-wire encoding ignored, and the ASN.1 standards aren't really abominations when you compare them with some of the web services standards. In addition to what you mentioned, I wanted to mention EXI (and FAST, the FIX encoding) as interesting on-the-wire encodings that maybe more people should consider over just compressed JSON or XML. Generic LZ-based compression doesn't necessarily win very much with lots of short messages.

abecedarius 12y ago
I had a similar feeling making an SNMP MIB processor around the same time (2000?) -- the ASN.1 libraries so complicated. So I wrote my own, which turned out simple and symmetrical just by encoding from back to front instead of the messy front-to-back-and-backpatch everyone else was doing. (And I guess by leaving some things out that we didn't use, like other encoding rules? I don't remember.) When those vulnerabilities hit I never found out how that code did -- I wish they'd open-sourced it as they'd planned. So, that particular part (DER? again I forget) seemed tolerable to me. The newer stuff like Cap'n Proto is probably still better.
- tptacek 12y ago
  Yes, this! This is the secret to simple BER/DER encoding: back-to-front. I was almost converted to ASN.1 BER after discovering this, but in the intervening 8 years the spell has (thankfully) worn off and I can see it for the clattering technological jalopy that it is. The features in the author's F# ASN.1 compiler are pretty swank. ASN.1 probably gets a bad rap because of BER/DER.
jzwinck 12y ago
FAST isn't exactly a FIX encoding, it's a key-value encoding that can be translated to/from FIX. Roughly speaking, FAST is to FIX as BSON is to JSON. But starting to use FAST today seems like a bad idea, because the largest publicly-known production users of FAST have already moved away from it (toward plain, uncompressed binary structs on the wire).

DominikD 12y ago

EDN ain't half bad, true. It addresses some of the JSON shortcomings (like: lack of comments) and is still readable and easy to parse. And I have to agree with ASN.1 middleware - most of it is old and of low quality. But this can be said about pretty much everything that's been around before the ubiquitousness of the Internet (i.e. when developers assumed good faith and hacking wasn't even a topic to be discussed). And yes, some of the X- encodings for ASN.1 should die in flames (alongside XML in general; flames are welcome ;>). It's still a pretty neat binary format if one uses BER though. But when it comes to binary formats, I really like everything IFF-like. It's similar to ASN.1 in terms of how data is structured, but uses far simpler rules for headers and payload. It's much harder to botch the implementation.