10 ms·
Useful Old Technologies: ASN.1 (2013)
- ggm 5y agoSEQUENCE OF ANY --there I maximally compressed all the comments
- ttyprintk 5y agoAmong those binary formats, this one benefits from the ubiquity of OpenSSL (see asn1parse) for debugging.
- soneil 5y agoIt's the underpinnings of snmp too. ASN is one of those things that's everywhere you've never looked
- preetamjinka 5y agoI worked on a SNMP Go package many years ago [0] and it was really fun learning about ASN.1 and figuring out how to write encoders and decoders for various types. [0] https://github.com/Cistern/snmp https://github.com/Cistern/snmp
- userbinator 5y agoThe cellular network standards (GSM/xG) use it heavily too.
- p_l 5y agoAll of OSI stack used ASN.1 extensively, something that would have made building protocol implementations much easier than text-based IETF style. SNMP is, IIRC, inspired by much more capable CMI which was part of OSI stack of protocols and is still used in some high end WAN network gear for monitoring and control (whereas "write" aspects of SNMP have mostly disappeared)
- jandrewrogers 5y agoASN.1 is all over the place in low-level networking protocols. That is where I learned it, two decades ago. While I disliked it initially, it was thoughtfully designed albeit obviously from a prior era. Also, turns out that if you work with it long enough you can read the binary encodings almost like it is text.
- yuubi 5y ago> can read the binary encodings almost like it is text. I'm curious what encoding rules that applies to: BER/DER I could easily believe, aligned PER a bit less so, and unaligned PER with lots of extensions and optional parts would be impressive.
- ahepp 5y agoHow does ASN.1 compare to something like binary protobuf? I have no data to back this up, but I was under the impression ASN.1 is one of those specifications that is so large and complicated it’s essentially impossible to implement correctly; and that ASN.1 parsers are a pretty infamous source of security bugs (although I mean, most parsers seem to be).
- rixed 5y agoASN1 has a much richer type system than protobufs, and is more general in that it accept several encodings, some of which optimised for size, some for speed, some for simplicity. I believe we could prune all the unnecessary 90s fad from the specs and end up with something actually pretty nice. I wish Google had done this as ASN2 instead of protobuf.
- bawolff 5y ago> ASN1 has a much richer type system than protobufs, and is more general in that it accept several encodings, some of which optimised for size, some for speed, some for simplicity. You say that like its a good thing.
- sandermvanvliet 5y agoProtobuf doesn’t have a date/date time/time type which means you get dates encoded as strings and all the fun that entails. Not saying that either is better but just sometimes you actually do want more types instead of less.
- Matthias247 5y agoWell there is Timestamp defined as a well known type which is available to all implementations despite not being a primitive type [1]. Plus one is obviously able to define any other custom types if necessary- eg as seen in [2]. [1] https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Timestamp https://developers.google.com/protocol-buffers/docs/referenc... [2] https://github.com/googleapis/googleapis/blob/master/google/type/datetime.proto https://github.com/googleapis/googleapis/blob/master/google/...
- Ericson2314 5y agoThey should have mentioned protobuf which it is closer to. To my knowledge, there is no bad nullability in ASN.1, unlike protobuf. Good job!
- foolfoolz 5y agothis is cool. i’ve only ever used asn1 as content for metadata in x509 certs and it makes a tiny bit more sense now why this is better than a basic string encoding
- mech422 5y agoNice to see someone actually put the TL;DR at the top for a change.. Never understood the logic of people putting them at the end of an article.. If I wasn't gonna read the whole thing, what makes you think I'm gonna scroll to the bottom?
- gjvc 5y agothe phenomena of "TL;DR" is satirical idiot culture made real
- tsimionescu 5y agoTldr is a standard part of any scientific paper, with a nicer name (abstract, summary).
- jandrese 5y agoIt's a slightly humorous term for an executive summary.
- vzaliva 5y agoplease, don't!
- moreati 5y agoA warning, ASN.1 is disliked by many people in software security. It often features in security vulnerabilities involving TLS and LDAP/Active Directory. They would recommend against adopting ASN.1 for new projects. I get the impression this partially due to a) most implementations being adhoc, of varying completeness, and written in C or C++ (e.g. OpenSSL); and b) sheer complexity of ASN.1 and other standards from that family/era. Sources: following software security researchers/practioners on Twitter, a few podcasts, here, etc.
- rdpintqogeogsaa 5y agoThis root cause analysis matches up with my experience. You can absolutely have safe ASN.1 with ASN.1 compilers, but the implementations that star in these vulnerabilities tend to be generic or ad-hoc DER/BER parsers. You'd be hard-pressed to be able to blame ASN.1 with XER (XML Encoding Rules) encoding since then you'd throw an XML parser at the problem, side-stepping the bit twiddling and exchanging it for XML parsing bugs instead. Similarly, if all you have is fixed-length items, OER (Octet Encoding Rules), you essentially get a formally specified struct with a well-defined way to go from there to XML or JSON with JER (JSON Encoding Rules). I imagine we'll see this happen with CBOR (whose ASN.1 equivalent is technically CDDL) a few years down the road.
- riedel 5y agoI think the root cause of this is that ASN.1 is deemed uncool and it is difficult to implement a standard compliant FOSS compiler for all the encodings. I was surprised doing research into binary XML and JSON how far you get with ASN.1 . However, the ecosystem seems so discoupled from anything else that there is rarely an implementation path for a quick win beyond mentioned hacky parsers that are both error prone and inefficient or purchasing some obscure commercial complier as I remember. Edit: would be really happy to see some good open source compilers with extensible encoding support being listed here
- rdpintqogeogsaa 5y ago> Edit: would be really happy to see some good open source compilers with extensible encoding support being listed here I think that's the reason of ASN.1 being trapped in a very restrictive space. From Olivier Dubuisson's ASN.1 -- Communication between Heterogeneous Systems, p. 86 et seq. talking about use of ASN.1 by the IETF: > the definition of many macros and macro instances to represent semantic links instead of information object classes and information objects although no ASN.1 compiler properly takes into account the macro concept (on the other hand, no compiler of the public domain does with the information object class concept unfortunately); In essence, ecosystem failure due to a non-existent overlap of the telecom industry (full of commercial and expensive solutions) and the Internet community (working in a standards committee that tracks pre-existing permissionless innovation distributing the standards at no cost, strongly affiliated with free software ideas) caused ASN.1 to fall by the roadside. Now that ASN.1 is considered uncool, the ecosystem certainly won't be coming anymore, so we'll be forced to reinvent ASN.1, probably re-learning every lesson along the way.
- kevin_thibedeau 5y agoIts greatest weakness is 10 different string encodings. I've always wondered what sort of bugs you could expose with a cert using videotex for the CN fields.
- jhoechtl 5y agoLike Rust?
- pdimitar 5y agoAs a Rust dev it's news to me that the language has 10 different strings. Care to elaborate?
- codeflo 5y agoNot the OP, but off the top of my head, we have: * str and String, which are the ones we just call “strings” * CStr and CString, for interop with libraries that work with null-terminated strings * OsStr and OsString, for interop with system libraries that expect non-UTF-8 encodings * Path and PathBuf, which are also just strings * [u8] and Vec<u8>, which are encoding-neutral binary strings (&[u8] is the type of b"Hello world") and are useful for interop with e.g. C++’s std::string That’s 10. (For non-Rust people who are curious: In these pairs, the first one is the type of a view of a string that’s stored elsewhere, and the second is the type that actually owns the string data. All of these 10 have their uses, but outside of interfacing with non-Rust libraries, you mostly work with str/String and Path/PathBuf.)
- pdimitar 5y agoOK, technically correct but I still disagree they are actually 10 because aside from `str` (which should be used only for constants really) and `String` everything else is strictly optional and you're not mandated to work with it unless you got a specific scenario (like CStr / CString). Compare this to Haskell where you have lazy and non-lazy / Unicode and non-Unicode strings which is at least 4 (I think there was more but I forgot now) and you are practically forced into using any of them depending on which library you're interacting with -- and I don't mean with C libraries in this instance. I personally never even used anything beyond `str` and `String` for any app development, including a bit lower level stuff (though not embedded or tight integration with system libraries).
- rixrax 5y agoIf you're interested of learning more about ASN.1, two great books on the subject are: - ASN.1 Communication between Heterogeneous Systems[0] - ASN.1 Complete[1] Both of which are available online for free. Last time I checked, they may not cover the latest fads like JER etc. but will provide you with a solid understanding of ASN.1 nonetheless. [0] https://www.oss.com/asn1/resources/books-whitepapers-pubs/larmouth-asn1-book.pdf https://www.oss.com/asn1/resources/books-whitepapers-pubs/la... [1] https://www.oss.com/asn1/resources/books-whitepapers-pubs/dubuisson-asn1-book.PDF https://www.oss.com/asn1/resources/books-whitepapers-pubs/du...
- Rochus 5y agoSome ISO ASN.1 standards (unfortunately not ISO 8824 and 8825) can be downloaded for free from https://standards.iso.org/ittf/PubliclyAvailableStandards/ https://standards.iso.org/ittf/PubliclyAvailableStandards/.
- XAMPPRocky 5y agoAll standards about ASN.1 (not including standards that use ASN.1 like MMS) are available for free from the ITU under the X.680 to X.697 series of standards.[0] [0]: https://www.itu.int/rec/T-REC-X/en https://www.itu.int/rec/T-REC-X/en
- Rochus 5y agoIndeed, thanks for the hint.
- phendrenad2 5y agoOh hey! There's quite a chunk of code in the Linux kernel to support this protocol. I hope that people here find some use for this "useful" old technology, otherwise it's just taking up space.
- praptak 5y agoDoes ASN handle versioning of the messages? If I add a field to the message I expect the new binaries to read the messages serialized by the old binaries and vice versa. Obviously the old binaries are not expected to understand the added field but they should not break when handed the upgraded message.
- noselasd 5y agoYes
- p_l 5y agoYou might be interested in how Kerberos uses ASN.1, with extension messages having a field that describes whether understanding the extension is required (this is important if extension covers information that might change the result of authorization operation, for example)
- anovikov 5y agoHow is it better than Protocol Buffers?
- fmajid 5y agoUseful? Hell, no. Like most technology derived from OSI, it's garbage that has been the source of a huge number of software vulnerabilities.
- crest 5y ago~98% of the ASN.1 parsing bugs could have been prevented by generating the parser instead of handwriting yet another recursive descent parser "with a few clever optimisations".
- pjmlp 5y agoBut think about that 1us lost. /s
- fmajid 5y agoReminds me of all the Trotskyists who say Communism has never been discredited because no true Communist regime ever existed, only forms of State Capitalism. Why oh why has no actual ASN.1 parser ever hewed to the self-evident Platonic ideal of machine-generated purity?
- ttybird2 5y agoASN.1 parser generators exist though...
- Matthias247 5y agoThe question around this is whether vulnerabilities show up because the technology is bad or because it’s widely used, triggers interest by researchers, and a certain amount of any implementations will have their set of issues.
- nullc 5y agoASN.1: I can only imagine that one day someone said "Strings are bad for security" and some bold lad overheard and said "Hold my Beer!".
- jesprenj 5y agoThe bittorrent protocol uses a very simple binary data representation system: bencoding. It can be described in a single paragraph: https://www.bittorrent.org/beps/bep_0003.html#bencoding https://www.bittorrent.org/beps/bep_0003.html#bencoding Although it supports only dictionaries, lists, ints and strings, it may be enough for your use case and is easily extendable. Edit: also ASN.1 does not solve futute-extending out-of-the-box. In JSON/bencoding/XML, it's trivial to add another key/element to a dictionary/array/element and allow applications to use it if they can and ignore it otherwise. Okay, TLV looks like it could handle that, but there's nothing about it in the description language itself.
- XAMPPRocky 5y ago> also ASN.1 does not solve futute-extending out-of-the-box. In JSON/bencoding/XML, it's trivial to add another key/element to a dictionary/array/element and allow applications to use it if they can and ignore it otherwise. Okay, TLV looks like it could handle that, but there's nothing about it in the description language itself. This is wrong. ASN.1 has always had support for "extensibility" out of the box. See section 52 of X.680[0] for complete details but I've included a couple of examples below. Also ASN.1 unlike other IDLs allows encodings to optionally take advantage of whether a type is considered in extensible. For example in PER, a type is a smaller encoding when it is not extensible. A ::= INTEGER (0..10, ..., 12) -- A is 0 to 10, or 12. -- V1 of B contains `a, b`, V2 contains `a, b, c` B ::= SEQUENCE { a INTEGER, b BOOLEAN, ... c OCTET STRING, } [0]: https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-X.680-202102-I!!PDF-E&type=items https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-X.68...
- jesprenj 5y agoThanks for the information! I forgot to mention the only "research" of ASN.1 I ever did is the linked post (:
- Rochus 5y agoASN.1 was great. Another old specification language still in use today is EXPRESS, one of the foundations of the huge ISO 10303 (STEP) standard series, as defined in ISO 10303-11. It even has algebraic data types. The development of STEP started in 1984 and has been one of the largest efforts ever undertaken by ISO.
- dathinab 5y agoIf you ran into ASN.1 run away (if you can). It's so insane hopeless overenginered with so many intersecting standard, different representations and nobes that it can easily become a nightmare. Most parsers also only support an opinionated subset for a specific purpose simply rejecting everything outside of the supported sunset even if they by specification should support it. This night sound fine, until you actually run into it in production and two supposedly compatible systems more are incompatible with no easy fix around.
- retrocryptid 5y agoIt's very clear the writer has not used ASN.1 or BER/DER encoding. There's no discussion of which standard for bit fields to use, constructed encoding, the practical requirement of an ASN.1 compiler or any security concerns. Heck, why not just use CORBA?
- johnisgood 5y agoI use ASN.1 between Kotlin <> Erlang, where Kotlin is the client and Erlang is the server, of course.
- misterti 5y agoI've been using asn.1 for 20 years now, and i find it absolutely fabulous. I still learn something new from time to time, when something doesn't parse or encode correctly :)