11 ms·

I spent a year making an ASN.1 compiler in D

297 points by BradleyChatha 1y ago

BradleyChatha 1y ago
In short: I wanted to talk a bit about ASN.1, a bit about D, and a bit about the compiler itself, but couldn't think of any real cohesive format. So I threw a bunch of semi-related ramblings together and I'm daring to call it a blog post. Sorry in advance since I will admit it's not the greatest quality, but it's really not easy to talk about so much with such brevity (especially since I've already forgot a ton of stuff I wanted to talk about more deeply :( )
- whizzter 1y ago
  As someone that had the dis-pleasure to work with Asn.1 data (yes, certificates) I fully symphatise with anguish you've gone through (that 6months of Ansible HR comments cracked me up also :D ).
  - BradleyChatha 1y ago
    It makes me laugh that absolutely no one can say "I've worked with ASN.1" in a positive light :D
    - StopDisinfo910 1y ago
      There was an amusing chain of comments the last time protobuf was mentionned in which some people were arguing that it had been a terrible idea and ASN.1, as a standard, should have been used. It was hilarious because clearly none of the people who were in favor had ever used ASN.1.
      - mananaysiempre 1y ago
        Cryptonector[1] maintains an ASN.1 implementation[2] and usually has good things to say about the language and its specs. (Kind of surprised not he’s not in the comments here already :) ) [1] https://news.ycombinator.com/user?id=cryptonector https://news.ycombinator.com/user?id=cryptonector [2] https://github.com/heimdal/heimdal/tree/master/lib/asn1 https://github.com/heimdal/heimdal/tree/master/lib/asn1
        cryptonector 1y ago
        Thanks for the shout-out! Yes, I do have nice things to say about ASN.1. It's all the others that mostly suck, with a few exceptions like XDR and DCE/Microsoft RPC's IDL.
        mananaysiempre 1y ago
        Derail accepted! Is your approval of DCE based only on the serialization not being TLV or on something else too? I have to say, while I do think its IDL is tasteful, its only real distinguishing feature is AFAICT the array-passing/returning stuff, and that feels much too specialized to make sense of in anything but C (or largely-isomorphic low-level languages like vernacular varieties of Pascal).
        cryptonector 1y ago
        Well, I do disapprove of the RPC-centric nature of both, XDR and DCE RPC, and I disapprove of the emphasis on "pointers" and -in the case of DCE- support for circular data structures and such. The 1980s penchant for "look ma'! I can have local things that are remote and you can't tell because I'm pretending that latency isn't part of the API hahahaha" research really shows in these. But yeah, at least they ain't TLV encodings, and the syntax is alright. I especially like XDR, though maybe that's because I worked at Sun Microsystems :) "Pointers" in XDR are really just `OPTIONAL` in ASN.1. Seems so silly to call them pointers. The reason they called them "pointers" is that that's how they represented optionality in the generated structures and code: if the field was present on the wire then the pointer is not null, and if it was absent the then pointer is null. And that's exactly what one does in ASN.1 tooling, though maybe with a host language Optional<> type rather than with pointers and null values. Whereas in hand-coded ASN.1 codecs one does sometimes see special values used as if the member had been `DEFAULT` rather than `OPTIONAL`.
      - whizzter 1y ago
        It's not entirely horrible, parsing DER dynamically enough to handle interpreting most common certificates can be done in some 200-300 lines of C#, so I'd take that any day over XML. The main problem is that to work with the data you need to understand the semantics of the magic object identifiers and while things like the PKIX module can be found easily, the definitions for other more obscure namespaces for extensions can be harder to locate as it's scattered in documentation from various standardization organizations. So, protobuf could very well have been transported in DER, the problem issue was probably more one of Google not seeing any value of interoperability and wanting to keep it simple (or worse, clashing by oblivious users re-using the wrong less well documented namespaces).
      - cryptonector 1y ago
        You're likely to find my comments among those saying that. I've been using ASN.1 in some way for a couple of decades, and I've been an ASN.1 implementor for about half a decade.
      - thayne 1y ago
        ASN.1 seems like something that could have been good ... If it was less complicated, had more accessible documentation, and had better tooling.
    - hamburglar 1y ago
      As a former PKI enthusiast (tongue firmly in cheek with that description) I can say if you can limit your exposure to simply issuing certs so you control the data and thus avoid all edge cases, quirks, non-canonical encodings, etc, dealing with ASN.1 is “not too terrible.” But it is bad. The thing that used to regularly amaze me was the insane depths of complexity the designers went to … back in the 70’s! It is astounding to me that they managed to make a system that encapsulated so much complexity and is still in everyday use today. You are truly a masochist and I salute you.
      - cyberax 1y ago
        It's also amazing that we're basically using only a couple of free-form text fields in the WebPKI for the most crucial parts of validation. Completely ignoring the ASN.1 support for complicated structures, with more than one CVE linked to incorrect parsing of these text fields m
        cryptonector 1y ago
        No we're not. We're using dNSName subjectAlternativeName values. We used to use the CN attribute of the subject DN, and... there is still code for that, but it's obsolete. We _are_ using subject DNs for linking certs to their issuers, but though that's "free-form", we don't parse them, we only check for equality.
        cyberax 1y ago
        CN is absolutely used everywhere. And it can contain wildcards. SANs are also free-form.
        cryptonector 1y ago
        SANs are not free-form. A dNSName SAN is supposed to have an FQDN. An rfc822Name SAN is supposed to carry an email address. And, ok, sure, email addresses' mailbox part is basically free-form, but so what, you don't interpret that part unless you've accepted that certificate for that email address' domain part, and then you interpret the mailbox part the way a mail server would because you're probably the mail server. Yes, you can have directoryName SANs, but the whole point of SANs is that DNs suck because x.400/x.500 naming sucks so we want to use something that isn't that.
    - cryptonector 1y ago
      Bzzt! Wrong! I have worked with ASN.1 for many years, and I love ASN.1. :) Really, I do. In particular I like: - that ASN.1 is generic, not specific to a given encoding rules (compare to XDR, which is both a syntax and a codec specification) - that ASN.1 lets you get quite formal if you want to in your specifications For example, RFC 5280 is the base PKIX spec, and if you look at RFCs 5911 and 5912 you'll see the same types (and those of other PKIX-related RFCs) with more formalisms. I use those formalisms in the ASN.1 tooling I maintain to implement a recursive, one-shot codec for certificates in all their glory. - that ASN.1 has been through the whole evolution of "hey, TLV rules are all you need and you get extensibility for free!!1!" through "oh no, no that's not quite right is it" through "we should add extensibility functionality" and "hmm, tags should not really have to appear in modules, so let's add AUTOMATIC tagging" and "well, let's support lots of encoding rules, like non-TLV binary ones (PER, OER) and XML and JSON!". Protocol Buffers is still stuck on TLV, all done badly by comparison to BER/DER.
      - zzo38computer 1y ago
        I also like ASN.1; I think it is better than JSON, XML, Protocol Buffers, etc, in many ways. I use it in some of my programs. (However, like many other formats (including JSON, XML, etc), ASN.1 can be badly used.)
      - BradleyChatha 1y ago
        Yeah I know I'm making fun of it a lot (mostly in jest) but it genuinely is a really interesting specification, and it's definitely sad - but not surprising - it's not a very popular choice outside of its few niche areas. :) Glad to see someone else who's gone down this road as well.
      - tambre 1y ago
        How do you feel about something like CBOR? In which stage would you say it's stuck in evolution compared to ASN.1 (since you said Protobuf is still TLV)?
        cryptonector 1y ago
    - coderjames 1y ago
      I worked with ASN.1 for a few years in the embedded space because its used for communications between aircraft and air traffic control in Europe [1]. I enjoyed it. BER encoding is pretty much the tightest way to represent messages on the wire and when you're charged per-bit for messaging, it all adds up. When a messaging syntax is defined in ASN.1 in an international standard (ICAO 9880 anyone?), its going to be around for a while. Haven't been able to get my current company to adopt ASN.1 to replace our existing homegrown serialization format. [1] https://en.wikipedia.org/wiki/Aeronautical_Telecommunication_Network https://en.wikipedia.org/wiki/Aeronautical_Telecommunication...
      - lepicz 1y ago
        of all the encoding i like BER the most as well (i worked in telecommunications when ASN.1 was common thing)
      - p_l 1y ago
        Isn't PER or OER more compact? especially for the per-bit charging thing
        coderjames 1y ago
        Oh yeah, derp. I was thinking unaligned-PER, not BER.
    - HelloNurse 1y ago
      I suspect that typical interactions with ASN.1 are benign because people are interested in reading and writing a few specific preexisting data structures with whatever encoding is required for interoperability, not in designing new message structures and choosing encodings for them. For example, when I inherited a public key signature system (mainly retrieving certificates and feeding them to cryptographic primitives and downloading and checking certificate revocation lists) everything troublesome was left by dismissed consultants; there were libraries for dealing with ASN.1 and I only had to educate myself about the different messages and their structure, like with any other standard protocol.
  - i2pi 1y ago
    (void) space. :P
- throw_a_grenade 1y ago
  Don't worry, it's your blog, and your way. Keep it up, if it makes you whole.
- giancarlostoro 1y ago
  At least you might be summoning Walter Bright in talking about D. One of my favorite languages I wish more companies would use. Unfortunately for its own sake, Go and Rust are way more popular in the industry.
  - pjmlp 1y ago
    Unfortunately it lost the opportunity back when Remedy Games and Facebook were betting on it. The various WIP features, and switching focus of what might bring more people into the ecosystem, have given away to other languages. Even C#, Java and C++ have gotten many of features that were only available in D as Andrei Alexandrescu's book came out in 2011.
    - Clouudy 1y ago
      I wouldn't say that it's unable to make a comeback, there is still a valid use case from my experience with it. The syntax, mixed-memory model, UFCS, and compilation speed are nice quality of life features compared to C++, and it's still a native binary compared to C# and Java. So if you're starting out with a new project from scratch there's not much reason not to beyond documentation reasons. And you can interface pretty easily to C/C++ as well as pretty much any other language designed for that sort of thing, but without a lot of syntax changes like Carbon. I imagine that the scope of its uses has shrunk as other languages caught up, and I don't think it's necessarily a good language for general enterprise stuff (unless you're dealing with C++), but for new projects it's still valid IMO. I think that the biggest field it could be used in is probably games too, especially if you're already writing a new engine from scratch. You could start with the GC and then ease off of it as the project develops in order to speed up development, for example. And D could always add newer features again too, tbh.
      - pjmlp 1y ago
        You always have to compare ecosystems, not programming languages syntax on their own. Another thing that Java and C# got to do since 2011, is that AOT is also part of the ecosystem and free of charge (Java has had commercial compilers for a while), so not even a native binary is an advantage as you imagine. First D has to finish what is already there in features that are almost done but not quite.
- olvy0 1y ago
  Just wanted to say I enjoyed your post very much. Thank you for writing it. I love D but unfortunately I haven't touched it for several years. I also have some experience writing parsers and implementing protocols.
  - BradleyChatha 1y ago
    Thank you :)
- mananaysiempre 1y ago
  A small nitpick: I don’t think your intersection example does what you want it to do. Perhaps there’s some obscure difference in “PER-visibility” or whatnot, but at least set-theoretically, LegacyFlags2 ::= INTEGER (0 | 2 ^ 4..8) -- as in the article is exactly equivalent to LegacyFlags2 ::= INTEGER (0) -- only a single value allowed as (using standard mathematical notation and making precedence explicit) {0} ∪ ({2} ∩ {4,5,6,7,8}) = {0} ∪ ∅ = {0}.
Keyframe 1y ago
I salute your for deep dive into this. History would have it that ASN.1 was already there as both an IDL and serialization format when HTTPS certs were defined. If it were today, would it be the same or would we end up with protobuf or thrift or similar?
- woodruffw 1y ago
  > If it were today, would it be the same or would we end up with protobuf or thrift or similar? The main advantage of ASN.1 (specifically DER) in an HTTPS/PKI context is that it's a canonical encoding. To my understanding Protobuf isn't; I don't know about Thrift. (A lot of hay is made about ASN.1 being bad, but it's really BER and other non-DER encodings of ASN.1 that make things painful. If you only read and write DER and limit yourself to the set of rules that occur in e.g. the Internet PKI RFCs, it's a relatively tractable and normal looking serialization format.)
  - jcranmer 1y ago
    I'm hardly a connoisseur of DER implementations, but my understanding is that there are two main problems with DER. The first is that the format isn't really parseable without using a schema, unlike (say) XML or JSON. This means your generic DER parser needs to have an ASN.1 schema passed into it to parse the DER, and this leads to the second problem, which is that this ends up being complex enough that basically every attempt to do so is full of memory safety issues.
    - whizzter 1y ago
      I wrote an Asn.1 decoder and since it contains type/size info you can often read a subset and handle the rest as opaque data objects if you need round-tripping, this is required as there can be plenty of data that is unknown to older consumers (like the ETSI EIDAS/Pades personal information extensions in PDF signatures). However, to have a sane interface for actually working with the data you do need a schema that can be compiled to a language specific notation.
    - woodruffw 1y ago
lukeh 1y ago
I worked on a Swift ASN.1 compiler [1] a while back (not swift-asn1, mine used Codable). I saved myself some time by using the Heimdal JSON compiler, which can transform ASN.1 into a much more parseable JSON AST. [1] https://github.com/PADL/ASN1Codable https://github.com/PADL/ASN1Codable [2] https://github.com/heimdal/heimdal/tree/master/lib/asn1 https://github.com/heimdal/heimdal/tree/master/lib/asn1
- BradleyChatha 1y ago
  Not heard of either of those projects before, but I love how libasn1's README has a thinly veiled hint of disdain for ASN.1 > which can transform ASN.1 into a much more parseable JSON AST The sign of a person who's been hurt, and doesn't want others to feel the same pain :D
  - marcosdumay 1y ago
    Hey, I love how the author describes ASN.1 as a "syntax" in quotes. What I disagree is on the disdain being veiled. Seems very explicit to me. Anyway, yeah, I hadn't heard about it before either, and it's great to know that somebody out there did solve that horrible problem already, and that we can use the library.
    - cryptonector 1y ago
      Ugh, I did not mean to express disdain.
      - marcosdumay 1y ago
        I'm sorry for misinterpreting it then. But I have to say, it's funnier to imagine you did. And easy to relate.
        cryptonector 1y ago
        Fair.
lepicz 1y ago
some people simply like pain :D (i worked with asn1c (not sure which fork) and had to hack in custom allocator and 64bit support. i shiver every time something needs attention in there)
- BradleyChatha 1y ago
  :) Honestly any compiler project in pure C is pretty hardcore in my eyes, ASN.1 must amplify the sheer horror.
  - cryptonector 1y ago
    Well, C is the source of the horror, for me anyways.
- morshu9001 1y ago
  I was using asn1c with a Rust project since there was no Rust asn1 compiler at the time. It became a bottleneck, and in profiling I found that the string copying helper used everywhere was doing bit-level copying even in our byte-aligned mode, which was extra weird cause that function had a param for byte alignment. One memcpy made it like 30% faster overall.
usrbinenv 1y ago
I really love D, it's one of my favorite languages. I've started implementing a vim-like text editor in it from scratch (using only Raylib as a dependency) and was surprised how far I was able to get and how good my test coverage was for it. My personal favorite features of D: * unit tests anywhere, so I usually write my methods/functions with unit tests following them immediately * blocks like version(unittest) {} makes it easy to exclude/include things that should only be compiled for testing * enums, unions, asserts, contract programming are all great I would say I didn't have to learn D much. Whatever I wanted to do with it, I would find in its docs or asked ChatGPT and there would always be a very nice way to do things.
- gavinray 1y ago
  D is a bittersweet topic for me. From a philosophical/language-design standpoint, it ticks so many boxes. It had the potential to be wildly popular, had a few things gone differently. If the language tooling and library ecosystem were on par with the titans of today, like Rust/Go, it really would be a powerhouse language.
  - binaryturtle 1y ago
    Isn't D supported by the GNU compiler collection? I personally would prefer this type of tooling over what Rust and Go do (I can't even get their compilers to run on my old platform anymore; not to mention all this dependencies on remote resources typical Rust/Go projects seem to have: which seems to be enforced by the ecosystem?)
    - mhh__ 1y ago
      It is
    - pjmlp 1y ago
      It is, however keeping LDC and GCC up to date is a volunteer effort with not enough people, so they are always a bit behind dmd. Still much better than GCCGO, kind of useless for anything beyond Go 1.18, no one is updating it any longer, and may as well join gcj.
inflames123 1y ago
[flagged]
axus 1y ago
SNMP MIB files are written in ASN.1. That is the extent of my knowledge about ASN.1, was nice to learn a little more by reading this blog post.
YouAreWRONGtoo 1y ago
The only goal of such ridiculous standards is to act as a form of vendor lock-in for vendors implementing those standards; the vendors get to say to governments that it is a standard and the sellers of the standards also get some money. Any system designed picking such standards is basically betraying their client. I think, if you want to annoy these people maximally, you should write an annotated version of the standard in a mathematical formal language. I read the table constraints, which try to do something simple, but it's written in the most convoluted way possible. I think I considered ASN.1 for a system once, but rejected it because of more modern technically superior system. If the parser for something like ASN.1 doesn't fit in 80 lines of Haskell, perhaps you just shouldn't use it. I don't know who these assholes are that say "Sure, let's make things slow and buggy, since we all hail Satan after all".
talkingtab 1y ago
OMG ASN.1. For those of you who missed this, there was a very interesting thing that happened in the growth of the internet. At the time people were evolving the protocols through the IETF. So all the things that you rely on now - for the most part - just came into being. One day there was email. There was ftp. There was TCP. There were the Van Jacobson TCP mods. At this time corporate types paid no attention to the internet. Academic types and the IETF were from what I saw the main developers. Then one day the corporate world realized they might make money. But the development process of the protocols was incomprehensible (and incompatible) with the corporate culture. TCP was clearly a mess, all these protocols like DNS were a mess. From the corporate perspective. So began the protocol wars https://en.wikipedia.org/wiki/Protocol_Wars https://en.wikipedia.org/wiki/Protocol_Wars. Whether ASN.1 was a product of that war or just a product of the corporate mentality, it serves as a powerful instance of the what the corporate world looks like vs the academic world looks like. You can find the wreckage from the war littered around. If you see and X.something protocol it could well be one of the relics. There were a few X.things that were adopted and useful, but were others that would haunt your dreams. Although this is ancient history, and pretty much now told from the corporate perspective, it suggests to us that the corporate process for thinking is not as effective as the alternative - the IETF and Academic. One is a sort of recipe culture. You write a recipe, everyone follows it and you are happy. The other is a sort of functional culture. If you can make bread and eat it you are happy. When the bread doesn't taste good you fix it. Given the kind of bread that is commonly available in the US now, we can draw some conclusions about recipe thinking, recipe culture, corporate culture etc. One could even extend this paradigm of thinking to new things like AI. Or not.
- gorgoiler 1y ago
  My partner and I were re-watching Father of the Bride the other day (rest in peace, Diane Keaton) and during the early parents meeting the son-in-law to-be describes himself as a communications consultant, working on X.25 networking installations. I had to pause the movie and explain to my partner just how close the world came to missing out on The Internet, and having instead to suffer the ignominy of visiting sites with addresses like “CN=wikipedia, OU=org, C=US” and god knows what other dreadful protocols underlying them. I think she was surprised how angry and distressed I sounded! It would have been awful! Poor her!
nicce 1y ago
Normally, you could say when implementing some standard that you get 80% of the functionality with 20% of the planned time. But with ASN.1 the remaining 20% could take the rest of your life.
OhMeadhbh 1y ago
Ack. I wrote an ASN.1 compiler in Java in the 90s. Mostly just to make sure I understood how it and BER/DER were used in X.509. I think the BER interpretation bits are still being used somewhere I'm sorry you had to waste a year of your life. There are few things I dislike more in the computing world than ASN.1/BER. It seems to encourage over-specification and is suboptimal for loosely coupled systems. But it looks like you had a decent time...
- zzo38computer 1y ago
  Looking at the comments, it seem that many people hate ASN.1 and many people like ASN.1 (I am the latter). (I consider BER to be messy and overly complicated, but DER is much better.) (I wrote a implementation of DER (decoding and encoding), although not the schema format (which I have never needed).)
  - BradleyChatha 1y ago
    I'm in the camp of I like it, but I also love to complain about things (see: my inability to say "I love D" without also bringing up other faults).
- BradleyChatha 1y ago
  I've been enjoying my time with the project overall, but it's one of those things where it's like "it hurts in a mostly good way". I think if I were more seasoned with ASN.1 itself, as well as with compiler development it'd have been a smoother experience. Though I definitely still could've chosen more higher value things to do throughout the last year :D
  - OhMeadhbh 11mo ago
    "hurts in mostly a good way" -- sounds like you have a good attitude about it. Good on you!
lihaciudanieljr 1y ago
[dead]
horizion2025 1y ago
I have also had to work with this in many contexts... Deeply embedded systems with no parsers available and where no "proper" ones would fit. So i have hand written but basic parsing and generation a few times. Oh and there is also non compliant implementations. E.g. some passports (yes the passports with chip use tons of ASN.1) even have incorrect including of big integers (supposed to be the minimum two complement, as I recall some passports used a fixed non-complement format yanked into the 0x02 INTEGER type... Some libraries have special non-compliant parsing modes to deal with it).
cbondurant 1y ago
Every time I have ever had the displeasure of looking at an X.whatever spec, I always end up coming away with the same conclusion. Somehow, despite these specifications being 90% metadata by weight, they seem to consistently forget the part of the spec that lets you actually know what something is. and that part is just left up to context. I could well be missing something, but a majority of the time it feels to me like they set out to make a database schema, and accidentally wrote the sqlite file format spec instead. Like thanks, its nice that I can parse this into a data structure :). It would be nicer, however if doing so gave me any idea of what I can do with the data I've parsed. Though to be fair I feel the same way about the entire concept of XML schemas. The fact that you theoretically can validate an xml document against a schema is honestly completely useless. If I am parsing XML, its because my code already knows what information it needs from the XML document, and really it should also know where that information is. I don't need a separate schema definition to tell me that! its already expressed!! In the part where I am parsing out the data I need!!!
- elcritch 1y ago
  > The fact that you theoretically can validate an xml document against a schema is honestly completely useless. If I am parsing XML, its because my code already knows what information it needs from the XML document, and really it should also know where that information is. You seem to miss the entire point of XML schemas, or any schema really. Validating a document against a schema isn’t really for your code. It’s for documentation of what can be in a given document and how it needs to be structured. So others don’t need to read your code to understand that. It then allows editing tools to verify generated documents. Or developers to understand how they can structure XML output properly. Your code could also use it to verify an XML document before passing it to your code. Then you can inform the user of an invalid document and why instead of just crashing at a random point in code without rolling your own. It can also verify an entire document whereas code may only parse portions leading to later corruption.
  - cryptonector 1y ago
    > Realistically if ASN.1 weren't as badly overengineered and had shipped only with some of the more modern of it's encoding formats we probably would all be using ASN.1 for man things including maybe your web server responses and this probably would cut non image/video network bandwidth by 1/3 or more. But then the network is overloaded by image/video transmissions and similar not other stuff so I guess who cares???!??? For payment systems people really do validate messages' encoding. > You seem to miss the entire point of XML schemas, or any schema really. Validating a document against a schema isn’t really for your code. It’s for documentation of what can be in a given document and how it needs to be structured. So others don’t need to read your code to understand that. Schemas also let you parse data into ergonomic data structures in the host programming language. That's really the biggest win in having a schema. Schemas and schema-aware tooling also help you not produce invalid messages that others then have to put up with and hack their parsers to handle when you present them with a fait accompli and you have the market power to make it stick. Schemas also let you specify things formally rather than having to use English prose (or worse, prose in not-English, or even worse, having to produce prose in multiple languages and make sure they all say the same thing!). The downside to schemas is that you have to do the work of writing them, and if you're the first implementor and you're using JSON and the messages are simple you just won't care to.
galkk 1y ago
Thank you, now I’m much more disillusioned in asn.1
- cryptonector 1y ago
  Don't be! ASN.1 is rather quite awesome.
- groundzeros2015 1y ago
  [dead]
teleforce 1y ago
According to ASN.1 Wikipedia entry, most of the tools supporting ASN.1 do the following: 1) parse the ASN.1 files, 2) generates the equivalent declaration in a programming language (like C or C++), 3) generates the encoding and decoding functions based on the previous declarations All of these of exercise are apparently part of data engineering process or lifecycle [1]. Back in early 21st century Python is just another interpreted general purpose programming language alternative, not for web (PHP), not for command tool (TCL), not for system (C/C++), not for data wrangling (Perl), not for numerical (Matlab/Fortran), not for statistics (R). D will probably follow similar trajectory of Python, but it really needs a special kind of killer application that will bring it to the fore. I'm envisioning that real-time data streaming, processing and engineering can be D killer utility and defining moment that D is for data. [1] Fundamentals of Data Engineering: https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/ https://www.oreilly.com/library/view/fundamentals-of-data/97...
- password4321 1y ago
  > D will probably follow similar trajectory of Python Apologies in advance for being that guy but D's trajectory seems pretty much locked in by now, while Python has been rebirthed with machine learning.
ConanRus 1y ago
[dead]
cryptonector 1y ago
Very neat article. I too have spent countless hours (but not as many) hacking on an ASN.1 compiler, adding a subset of X.681, X.682, X.683 functionality to make it possible to -in a single codec invocation!- a whole certificate, with all its details like extensions and OtherName SANs and what not decoded recursively. So it's very nice to see a post about similar work! ASN.1 really is quite awesome. It gets a lot of hate, but it's not deserved. ASN.1 is not just a syntax or set of encoding rules -- it's a type system, and a very powerful one at that.
sedatk 1y ago
There's a Turkish saying, "a human will [use] this, a human!", to signify that the thing is so abnormal/out-of-proportion that it doesn't seem to be made for people. The verb changes based on the context. If you had made too much food for example, the verb would be "eat". I think it's a great motto for design. Remember the Game of Thrones quote, "the man who passes the sentence should swing the sword"? I think it should also be applied to specs. Anyone who comes up with a spec must be the first responsible party to develop a parser for it. The spec doesn't get ratified unless it comes with working parser code with unit tests. That kind of requirement might actually improve specs.
deepsun 1y ago
Wow, I needed to parse just one small ASN.1 with one value (one signature), but I didn't know ASN.1 can have a specification (to generate parser from it). So I ended up untangling it myself, just for that specific 256 bits. Still I think it's better to have overapecified format for security stuff, json and xml are just too vague and parsers are unpredictable.
arlyle 1y ago
see also fabrice bellard's ffasn1 (c from asn.1) compiler: https://bellard.org/ffasn1/ https://bellard.org/ffasn1/
- cryptonector 1y ago
  Oh yeah, amazing tooling no doubt, and non-free. Fabrice Bellard knew he could make good bank from this. ASN.1 for 3GPP and finance is a great little niche if you have a chance to exploit it.
lilyball 1y ago
I'm fascinated by ASN.1, I don't know why it appeals to me so much but I find working with it oddly fun. I've always wanted to find the time to write an ASN.1 compiler for Rust, because for some reason all of the Rust implementations I've seen end up just either being derive macros on Rust structs (so going the other direction), or even just providing a bunch of ASN.1 types and functions and expecting you to manually chain them together using nom.
jmspring 1y ago
As someone who extended the ASN.1 parser in the Netscape code base to handle PKCS#12 and at one point knew more about RSA standards and varios ASN.1 definitions than anyone should, I respect the blog owners diligence and masochism. It would not be the project I would pick out to learn D.
- BradleyChatha 1y ago
  > As someone who extended the ASN.1 parser in the Netscape code base to handle PKCS#12 Sounds like you have some fun potential war stories in that case.
  - jmspring 11mo ago
    War story or nightmare? The asn.1 parsing in Netscape was an interesting beast. I also added some to a VPN I worked on while at another company.
themafia 1y ago
Ctrl-F "walterbright". huh. Must be on vacation today.
- dardeaup 1y ago
  Or he may just be tired of explaining his views over and over.
zzo38computer 1y ago
> Did I ever mention that ASN.1 is complicated? The schema format is complicated, and the data format also is, but most of it is ignorable complexity, often making it simpler than it is. I do not use the ASN.1 schema notation, since I have been able to use ASN.1 without it. (I wrote my own implementation of the data format (DER) in C.) > On the one hand the sheer amount of possible encodings is daunting, but on the other hand it shows a certain flexbility that ASN.1 provides - you could even invent your own domain-specific encoding if needed. It is true. My opinion is, of the standard encodings, DER is the only one that is any good (and is the one that I usually use (although most of my uses are not having to do with cryptography); X.509 (which is a common use of ASN.1) also uses DER), but I did make up my own encodings as well: DSER (between BER and DER), SDSER (like DSER but the length is encoded differently so that streaming is possible, in a (in my opinion) better way than CER does), and TER (a text format which is intended to be converted to DER). > ANY DEFINED BY I still effectively use it (as well as ANY), since I think it is very useful. (As I mentioned, I did not actually use the ASN.1 schema format, but I sometimes use what effectively works like ANY and ANY DEFINED BY.) (I also sometimes use a nonstandard feature that I made up, which is called OBJECT IDENTIFIER RELATIVE TO, which is like OBJECT IDENTIFIER but it can be encoded more efficiently (using the RELATIVE OID type) if the value has the prefix that the schema expects it to have.)

297 points by BradleyChatha 1y ago

In short: I wanted to talk a bit about ASN.1, a bit about D, and a bit about the compiler itself, but couldn't think of any real cohesive format. So I threw a bunch of semi-related ramblings together and I'm daring to call it a blog post. Sorry in advance since I will admit it's not the greatest quality, but it's really not easy to talk about so much with such brevity (especially since I've already forgot a ton of stuff I wanted to talk about more deeply :( )

whizzter 1y ago
As someone that had the dis-pleasure to work with Asn.1 data (yes, certificates) I fully symphatise with anguish you've gone through (that 6months of Ansible HR comments cracked me up also :D ).
- BradleyChatha 1y ago
  It makes me laugh that absolutely no one can say "I've worked with ASN.1" in a positive light :D
  - StopDisinfo910 1y ago
    There was an amusing chain of comments the last time protobuf was mentionned in which some people were arguing that it had been a terrible idea and ASN.1, as a standard, should have been used. It was hilarious because clearly none of the people who were in favor had ever used ASN.1.
    - mananaysiempre 1y ago
      Cryptonector[1] maintains an ASN.1 implementation[2] and usually has good things to say about the language and its specs. (Kind of surprised not he’s not in the comments here already :) ) [1] https://news.ycombinator.com/user?id=cryptonector https://news.ycombinator.com/user?id=cryptonector [2] https://github.com/heimdal/heimdal/tree/master/lib/asn1 https://github.com/heimdal/heimdal/tree/master/lib/asn1
      - cryptonector 1y ago
        Thanks for the shout-out! Yes, I do have nice things to say about ASN.1. It's all the others that mostly suck, with a few exceptions like XDR and DCE/Microsoft RPC's IDL.
        mananaysiempre 1y ago
        Derail accepted! Is your approval of DCE based only on the serialization not being TLV or on something else too? I have to say, while I do think its IDL is tasteful, its only real distinguishing feature is AFAICT the array-passing/returning stuff, and that feels much too specialized to make sense of in anything but C (or largely-isomorphic low-level languages like vernacular varieties of Pascal).
        cryptonector 1y ago
        Well, I do disapprove of the RPC-centric nature of both, XDR and DCE RPC, and I disapprove of the emphasis on "pointers" and -in the case of DCE- support for circular data structures and such. The 1980s penchant for "look ma'! I can have local things that are remote and you can't tell because I'm pretending that latency isn't part of the API hahahaha" research really shows in these. But yeah, at least they ain't TLV encodings, and the syntax is alright. I especially like XDR, though maybe that's because I worked at Sun Microsystems :) "Pointers" in XDR are really just `OPTIONAL` in ASN.1. Seems so silly to call them pointers. The reason they called them "pointers" is that that's how they represented optionality in the generated structures and code: if the field was present on the wire then the pointer is not null, and if it was absent the then pointer is null. And that's exactly what one does in ASN.1 tooling, though maybe with a host language Optional<> type rather than with pointers and null values. Whereas in hand-coded ASN.1 codecs one does sometimes see special values used as if the member had been `DEFAULT` rather than `OPTIONAL`.
    - whizzter 1y ago
      It's not entirely horrible, parsing DER dynamically enough to handle interpreting most common certificates can be done in some 200-300 lines of C#, so I'd take that any day over XML. The main problem is that to work with the data you need to understand the semantics of the magic object identifiers and while things like the PKIX module can be found easily, the definitions for other more obscure namespaces for extensions can be harder to locate as it's scattered in documentation from various standardization organizations. So, protobuf could very well have been transported in DER, the problem issue was probably more one of Google not seeing any value of interoperability and wanting to keep it simple (or worse, clashing by oblivious users re-using the wrong less well documented namespaces).
    - cryptonector 1y ago
      You're likely to find my comments among those saying that. I've been using ASN.1 in some way for a couple of decades, and I've been an ASN.1 implementor for about half a decade.
    - thayne 1y ago
      ASN.1 seems like something that could have been good ... If it was less complicated, had more accessible documentation, and had better tooling.
  - hamburglar 1y ago
    As a former PKI enthusiast (tongue firmly in cheek with that description) I can say if you can limit your exposure to simply issuing certs so you control the data and thus avoid all edge cases, quirks, non-canonical encodings, etc, dealing with ASN.1 is “not too terrible.” But it is bad. The thing that used to regularly amaze me was the insane depths of complexity the designers went to … back in the 70’s! It is astounding to me that they managed to make a system that encapsulated so much complexity and is still in everyday use today. You are truly a masochist and I salute you.
    - cyberax 1y ago
      It's also amazing that we're basically using only a couple of free-form text fields in the WebPKI for the most crucial parts of validation. Completely ignoring the ASN.1 support for complicated structures, with more than one CVE linked to incorrect parsing of these text fields m
      - cryptonector 1y ago
        No we're not. We're using dNSName subjectAlternativeName values. We used to use the CN attribute of the subject DN, and... there is still code for that, but it's obsolete. We _are_ using subject DNs for linking certs to their issuers, but though that's "free-form", we don't parse them, we only check for equality.
        cyberax 1y ago
        CN is absolutely used everywhere. And it can contain wildcards. SANs are also free-form.
        cryptonector 1y ago
        SANs are not free-form. A dNSName SAN is supposed to have an FQDN. An rfc822Name SAN is supposed to carry an email address. And, ok, sure, email addresses' mailbox part is basically free-form, but so what, you don't interpret that part unless you've accepted that certificate for that email address' domain part, and then you interpret the mailbox part the way a mail server would because you're probably the mail server. Yes, you can have directoryName SANs, but the whole point of SANs is that DNs suck because x.400/x.500 naming sucks so we want to use something that isn't that.
  - cryptonector 1y ago
    Bzzt! Wrong! I have worked with ASN.1 for many years, and I love ASN.1. :) Really, I do. In particular I like: - that ASN.1 is generic, not specific to a given encoding rules (compare to XDR, which is both a syntax and a codec specification) - that ASN.1 lets you get quite formal if you want to in your specifications For example, RFC 5280 is the base PKIX spec, and if you look at RFCs 5911 and 5912 you'll see the same types (and those of other PKIX-related RFCs) with more formalisms. I use those formalisms in the ASN.1 tooling I maintain to implement a recursive, one-shot codec for certificates in all their glory. - that ASN.1 has been through the whole evolution of "hey, TLV rules are all you need and you get extensibility for free!!1!" through "oh no, no that's not quite right is it" through "we should add extensibility functionality" and "hmm, tags should not really have to appear in modules, so let's add AUTOMATIC tagging" and "well, let's support lots of encoding rules, like non-TLV binary ones (PER, OER) and XML and JSON!". Protocol Buffers is still stuck on TLV, all done badly by comparison to BER/DER.
    - zzo38computer 1y ago
      I also like ASN.1; I think it is better than JSON, XML, Protocol Buffers, etc, in many ways. I use it in some of my programs. (However, like many other formats (including JSON, XML, etc), ASN.1 can be badly used.)
    - BradleyChatha 1y ago
      Yeah I know I'm making fun of it a lot (mostly in jest) but it genuinely is a really interesting specification, and it's definitely sad - but not surprising - it's not a very popular choice outside of its few niche areas. :) Glad to see someone else who's gone down this road as well.
    - tambre 1y ago
      How do you feel about something like CBOR? In which stage would you say it's stuck in evolution compared to ASN.1 (since you said Protobuf is still TLV)?
      - cryptonector 1y ago
  - coderjames 1y ago
    I worked with ASN.1 for a few years in the embedded space because its used for communications between aircraft and air traffic control in Europe [1]. I enjoyed it. BER encoding is pretty much the tightest way to represent messages on the wire and when you're charged per-bit for messaging, it all adds up. When a messaging syntax is defined in ASN.1 in an international standard (ICAO 9880 anyone?), its going to be around for a while. Haven't been able to get my current company to adopt ASN.1 to replace our existing homegrown serialization format. [1] https://en.wikipedia.org/wiki/Aeronautical_Telecommunication_Network https://en.wikipedia.org/wiki/Aeronautical_Telecommunication...
    - lepicz 1y ago
      of all the encoding i like BER the most as well (i worked in telecommunications when ASN.1 was common thing)
    - p_l 1y ago
      Isn't PER or OER more compact? especially for the per-bit charging thing
      - coderjames 1y ago
        Oh yeah, derp. I was thinking unaligned-PER, not BER.
  - HelloNurse 1y ago
    I suspect that typical interactions with ASN.1 are benign because people are interested in reading and writing a few specific preexisting data structures with whatever encoding is required for interoperability, not in designing new message structures and choosing encodings for them. For example, when I inherited a public key signature system (mainly retrieving certificates and feeding them to cryptographic primitives and downloading and checking certificate revocation lists) everything troublesome was left by dismissed consultants; there were libraries for dealing with ASN.1 and I only had to educate myself about the different messages and their structure, like with any other standard protocol.
- i2pi 1y ago
  (void) space. :P
throw_a_grenade 1y ago
Don't worry, it's your blog, and your way. Keep it up, if it makes you whole.
giancarlostoro 1y ago
At least you might be summoning Walter Bright in talking about D. One of my favorite languages I wish more companies would use. Unfortunately for its own sake, Go and Rust are way more popular in the industry.
- pjmlp 1y ago
  Unfortunately it lost the opportunity back when Remedy Games and Facebook were betting on it. The various WIP features, and switching focus of what might bring more people into the ecosystem, have given away to other languages. Even C#, Java and C++ have gotten many of features that were only available in D as Andrei Alexandrescu's book came out in 2011.
  - Clouudy 1y ago
    I wouldn't say that it's unable to make a comeback, there is still a valid use case from my experience with it. The syntax, mixed-memory model, UFCS, and compilation speed are nice quality of life features compared to C++, and it's still a native binary compared to C# and Java. So if you're starting out with a new project from scratch there's not much reason not to beyond documentation reasons. And you can interface pretty easily to C/C++ as well as pretty much any other language designed for that sort of thing, but without a lot of syntax changes like Carbon. I imagine that the scope of its uses has shrunk as other languages caught up, and I don't think it's necessarily a good language for general enterprise stuff (unless you're dealing with C++), but for new projects it's still valid IMO. I think that the biggest field it could be used in is probably games too, especially if you're already writing a new engine from scratch. You could start with the GC and then ease off of it as the project develops in order to speed up development, for example. And D could always add newer features again too, tbh.
    - pjmlp 1y ago
      You always have to compare ecosystems, not programming languages syntax on their own. Another thing that Java and C# got to do since 2011, is that AOT is also part of the ecosystem and free of charge (Java has had commercial compilers for a while), so not even a native binary is an advantage as you imagine. First D has to finish what is already there in features that are almost done but not quite.
olvy0 1y ago
Just wanted to say I enjoyed your post very much. Thank you for writing it. I love D but unfortunately I haven't touched it for several years. I also have some experience writing parsers and implementing protocols.
- BradleyChatha 1y ago
  Thank you :)
mananaysiempre 1y ago
A small nitpick: I don’t think your intersection example does what you want it to do. Perhaps there’s some obscure difference in “PER-visibility” or whatnot, but at least set-theoretically, LegacyFlags2 ::= INTEGER (0 | 2 ^ 4..8) -- as in the article is exactly equivalent to LegacyFlags2 ::= INTEGER (0) -- only a single value allowed as (using standard mathematical notation and making precedence explicit) {0} ∪ ({2} ∩ {4,5,6,7,8}) = {0} ∪ ∅ = {0}.

Keyframe 1y ago

I salute your for deep dive into this. History would have it that ASN.1 was already there as both an IDL and serialization format when HTTPS certs were defined. If it were today, would it be the same or would we end up with protobuf or thrift or similar?

woodruffw 1y ago
> If it were today, would it be the same or would we end up with protobuf or thrift or similar? The main advantage of ASN.1 (specifically DER) in an HTTPS/PKI context is that it's a canonical encoding. To my understanding Protobuf isn't; I don't know about Thrift. (A lot of hay is made about ASN.1 being bad, but it's really BER and other non-DER encodings of ASN.1 that make things painful. If you only read and write DER and limit yourself to the set of rules that occur in e.g. the Internet PKI RFCs, it's a relatively tractable and normal looking serialization format.)
- jcranmer 1y ago
  I'm hardly a connoisseur of DER implementations, but my understanding is that there are two main problems with DER. The first is that the format isn't really parseable without using a schema, unlike (say) XML or JSON. This means your generic DER parser needs to have an ASN.1 schema passed into it to parse the DER, and this leads to the second problem, which is that this ends up being complex enough that basically every attempt to do so is full of memory safety issues.
  - whizzter 1y ago
    I wrote an Asn.1 decoder and since it contains type/size info you can often read a subset and handle the rest as opaque data objects if you need round-tripping, this is required as there can be plenty of data that is unknown to older consumers (like the ETSI EIDAS/Pades personal information extensions in PDF signatures). However, to have a sane interface for actually working with the data you do need a schema that can be compiled to a language specific notation.
  - woodruffw 1y ago

lukeh 1y ago

I worked on a Swift ASN.1 compiler [1] a while back (not swift-asn1, mine used Codable). I saved myself some time by using the Heimdal JSON compiler, which can transform ASN.1 into a much more parseable JSON AST. [1] https://github.com/PADL/ASN1Codable https://github.com/PADL/ASN1Codable [2] https://github.com/heimdal/heimdal/tree/master/lib/asn1 https://github.com/heimdal/heimdal/tree/master/lib/asn1

BradleyChatha 1y ago
Not heard of either of those projects before, but I love how libasn1's README has a thinly veiled hint of disdain for ASN.1 > which can transform ASN.1 into a much more parseable JSON AST The sign of a person who's been hurt, and doesn't want others to feel the same pain :D
- marcosdumay 1y ago
  Hey, I love how the author describes ASN.1 as a "syntax" in quotes. What I disagree is on the disdain being veiled. Seems very explicit to me. Anyway, yeah, I hadn't heard about it before either, and it's great to know that somebody out there did solve that horrible problem already, and that we can use the library.
  - cryptonector 1y ago
    Ugh, I did not mean to express disdain.
    - marcosdumay 1y ago
      I'm sorry for misinterpreting it then. But I have to say, it's funnier to imagine you did. And easy to relate.
      - cryptonector 1y ago
        Fair.

lepicz 1y ago

some people simply like pain :D (i worked with asn1c (not sure which fork) and had to hack in custom allocator and 64bit support. i shiver every time something needs attention in there)

BradleyChatha 1y ago
:) Honestly any compiler project in pure C is pretty hardcore in my eyes, ASN.1 must amplify the sheer horror.
- cryptonector 1y ago
  Well, C is the source of the horror, for me anyways.
morshu9001 1y ago
I was using asn1c with a Rust project since there was no Rust asn1 compiler at the time. It became a bottleneck, and in profiling I found that the string copying helper used everywhere was doing bit-level copying even in our byte-aligned mode, which was extra weird cause that function had a param for byte alignment. One memcpy made it like 30% faster overall.

usrbinenv 1y ago

I really love D, it's one of my favorite languages. I've started implementing a vim-like text editor in it from scratch (using only Raylib as a dependency) and was surprised how far I was able to get and how good my test coverage was for it. My personal favorite features of D: * unit tests anywhere, so I usually write my methods/functions with unit tests following them immediately * blocks like version(unittest) {} makes it easy to exclude/include things that should only be compiled for testing * enums, unions, asserts, contract programming are all great I would say I didn't have to learn D much. Whatever I wanted to do with it, I would find in its docs or asked ChatGPT and there would always be a very nice way to do things.

gavinray 1y ago
D is a bittersweet topic for me. From a philosophical/language-design standpoint, it ticks so many boxes. It had the potential to be wildly popular, had a few things gone differently. If the language tooling and library ecosystem were on par with the titans of today, like Rust/Go, it really would be a powerhouse language.
- binaryturtle 1y ago
  Isn't D supported by the GNU compiler collection? I personally would prefer this type of tooling over what Rust and Go do (I can't even get their compilers to run on my old platform anymore; not to mention all this dependencies on remote resources typical Rust/Go projects seem to have: which seems to be enforced by the ecosystem?)
  - mhh__ 1y ago
    It is
  - pjmlp 1y ago
    It is, however keeping LDC and GCC up to date is a volunteer effort with not enough people, so they are always a bit behind dmd. Still much better than GCCGO, kind of useless for anything beyond Go 1.18, no one is updating it any longer, and may as well join gcj.

inflames123 1y ago

[flagged]

axus 1y ago

SNMP MIB files are written in ASN.1. That is the extent of my knowledge about ASN.1, was nice to learn a little more by reading this blog post.

YouAreWRONGtoo 1y ago

The only goal of such ridiculous standards is to act as a form of vendor lock-in for vendors implementing those standards; the vendors get to say to governments that it is a standard and the sellers of the standards also get some money. Any system designed picking such standards is basically betraying their client. I think, if you want to annoy these people maximally, you should write an annotated version of the standard in a mathematical formal language. I read the table constraints, which try to do something simple, but it's written in the most convoluted way possible. I think I considered ASN.1 for a system once, but rejected it because of more modern technically superior system. If the parser for something like ASN.1 doesn't fit in 80 lines of Haskell, perhaps you just shouldn't use it. I don't know who these assholes are that say "Sure, let's make things slow and buggy, since we all hail Satan after all".

talkingtab 1y ago

OMG ASN.1. For those of you who missed this, there was a very interesting thing that happened in the growth of the internet. At the time people were evolving the protocols through the IETF. So all the things that you rely on now - for the most part - just came into being. One day there was email. There was ftp. There was TCP. There were the Van Jacobson TCP mods. At this time corporate types paid no attention to the internet. Academic types and the IETF were from what I saw the main developers. Then one day the corporate world realized they might make money. But the development process of the protocols was incomprehensible (and incompatible) with the corporate culture. TCP was clearly a mess, all these protocols like DNS were a mess. From the corporate perspective. So began the protocol wars https://en.wikipedia.org/wiki/Protocol_Wars https://en.wikipedia.org/wiki/Protocol_Wars. Whether ASN.1 was a product of that war or just a product of the corporate mentality, it serves as a powerful instance of the what the corporate world looks like vs the academic world looks like. You can find the wreckage from the war littered around. If you see and X.something protocol it could well be one of the relics. There were a few X.things that were adopted and useful, but were others that would haunt your dreams. Although this is ancient history, and pretty much now told from the corporate perspective, it suggests to us that the corporate process for thinking is not as effective as the alternative - the IETF and Academic. One is a sort of recipe culture. You write a recipe, everyone follows it and you are happy. The other is a sort of functional culture. If you can make bread and eat it you are happy. When the bread doesn't taste good you fix it. Given the kind of bread that is commonly available in the US now, we can draw some conclusions about recipe thinking, recipe culture, corporate culture etc. One could even extend this paradigm of thinking to new things like AI. Or not.

gorgoiler 1y ago
My partner and I were re-watching Father of the Bride the other day (rest in peace, Diane Keaton) and during the early parents meeting the son-in-law to-be describes himself as a communications consultant, working on X.25 networking installations. I had to pause the movie and explain to my partner just how close the world came to missing out on The Internet, and having instead to suffer the ignominy of visiting sites with addresses like “CN=wikipedia, OU=org, C=US” and god knows what other dreadful protocols underlying them. I think she was surprised how angry and distressed I sounded! It would have been awful! Poor her!

nicce 1y ago

Normally, you could say when implementing some standard that you get 80% of the functionality with 20% of the planned time. But with ASN.1 the remaining 20% could take the rest of your life.

OhMeadhbh 1y ago

Ack. I wrote an ASN.1 compiler in Java in the 90s. Mostly just to make sure I understood how it and BER/DER were used in X.509. I think the BER interpretation bits are still being used somewhere I'm sorry you had to waste a year of your life. There are few things I dislike more in the computing world than ASN.1/BER. It seems to encourage over-specification and is suboptimal for loosely coupled systems. But it looks like you had a decent time...

zzo38computer 1y ago
Looking at the comments, it seem that many people hate ASN.1 and many people like ASN.1 (I am the latter). (I consider BER to be messy and overly complicated, but DER is much better.) (I wrote a implementation of DER (decoding and encoding), although not the schema format (which I have never needed).)
- BradleyChatha 1y ago
  I'm in the camp of I like it, but I also love to complain about things (see: my inability to say "I love D" without also bringing up other faults).
BradleyChatha 1y ago
I've been enjoying my time with the project overall, but it's one of those things where it's like "it hurts in a mostly good way". I think if I were more seasoned with ASN.1 itself, as well as with compiler development it'd have been a smoother experience. Though I definitely still could've chosen more higher value things to do throughout the last year :D
- OhMeadhbh 11mo ago
  "hurts in mostly a good way" -- sounds like you have a good attitude about it. Good on you!

lihaciudanieljr 1y ago

[dead]

horizion2025 1y ago

I have also had to work with this in many contexts... Deeply embedded systems with no parsers available and where no "proper" ones would fit. So i have hand written but basic parsing and generation a few times. Oh and there is also non compliant implementations. E.g. some passports (yes the passports with chip use tons of ASN.1) even have incorrect including of big integers (supposed to be the minimum two complement, as I recall some passports used a fixed non-complement format yanked into the 0x02 INTEGER type... Some libraries have special non-compliant parsing modes to deal with it).

cbondurant 1y ago

Every time I have ever had the displeasure of looking at an X.whatever spec, I always end up coming away with the same conclusion. Somehow, despite these specifications being 90% metadata by weight, they seem to consistently forget the part of the spec that lets you actually know what something is. and that part is just left up to context. I could well be missing something, but a majority of the time it feels to me like they set out to make a database schema, and accidentally wrote the sqlite file format spec instead. Like thanks, its nice that I can parse this into a data structure :). It would be nicer, however if doing so gave me any idea of what I can do with the data I've parsed. Though to be fair I feel the same way about the entire concept of XML schemas. The fact that you theoretically can validate an xml document against a schema is honestly completely useless. If I am parsing XML, its because my code already knows what information it needs from the XML document, and really it should also know where that information is. I don't need a separate schema definition to tell me that! its already expressed!! In the part where I am parsing out the data I need!!!

elcritch 1y ago
> The fact that you theoretically can validate an xml document against a schema is honestly completely useless. If I am parsing XML, its because my code already knows what information it needs from the XML document, and really it should also know where that information is. You seem to miss the entire point of XML schemas, or any schema really. Validating a document against a schema isn’t really for your code. It’s for documentation of what can be in a given document and how it needs to be structured. So others don’t need to read your code to understand that. It then allows editing tools to verify generated documents. Or developers to understand how they can structure XML output properly. Your code could also use it to verify an XML document before passing it to your code. Then you can inform the user of an invalid document and why instead of just crashing at a random point in code without rolling your own. It can also verify an entire document whereas code may only parse portions leading to later corruption.
- cryptonector 1y ago
  > Realistically if ASN.1 weren't as badly overengineered and had shipped only with some of the more modern of it's encoding formats we probably would all be using ASN.1 for man things including maybe your web server responses and this probably would cut non image/video network bandwidth by 1/3 or more. But then the network is overloaded by image/video transmissions and similar not other stuff so I guess who cares???!??? For payment systems people really do validate messages' encoding. > You seem to miss the entire point of XML schemas, or any schema really. Validating a document against a schema isn’t really for your code. It’s for documentation of what can be in a given document and how it needs to be structured. So others don’t need to read your code to understand that. Schemas also let you parse data into ergonomic data structures in the host programming language. That's really the biggest win in having a schema. Schemas and schema-aware tooling also help you not produce invalid messages that others then have to put up with and hack their parsers to handle when you present them with a fait accompli and you have the market power to make it stick. Schemas also let you specify things formally rather than having to use English prose (or worse, prose in not-English, or even worse, having to produce prose in multiple languages and make sure they all say the same thing!). The downside to schemas is that you have to do the work of writing them, and if you're the first implementor and you're using JSON and the messages are simple you just won't care to.

galkk 1y ago

Thank you, now I’m much more disillusioned in asn.1

cryptonector 1y ago
Don't be! ASN.1 is rather quite awesome.
groundzeros2015 1y ago
[dead]

teleforce 1y ago

According to ASN.1 Wikipedia entry, most of the tools supporting ASN.1 do the following: 1) parse the ASN.1 files, 2) generates the equivalent declaration in a programming language (like C or C++), 3) generates the encoding and decoding functions based on the previous declarations All of these of exercise are apparently part of data engineering process or lifecycle [1]. Back in early 21st century Python is just another interpreted general purpose programming language alternative, not for web (PHP), not for command tool (TCL), not for system (C/C++), not for data wrangling (Perl), not for numerical (Matlab/Fortran), not for statistics (R). D will probably follow similar trajectory of Python, but it really needs a special kind of killer application that will bring it to the fore. I'm envisioning that real-time data streaming, processing and engineering can be D killer utility and defining moment that D is for data. [1] Fundamentals of Data Engineering: https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/ https://www.oreilly.com/library/view/fundamentals-of-data/97...

password4321 1y ago
> D will probably follow similar trajectory of Python Apologies in advance for being that guy but D's trajectory seems pretty much locked in by now, while Python has been rebirthed with machine learning.

ConanRus 1y ago

[dead]

cryptonector 1y ago

Very neat article. I too have spent countless hours (but not as many) hacking on an ASN.1 compiler, adding a subset of X.681, X.682, X.683 functionality to make it possible to -in a single codec invocation!- a whole certificate, with all its details like extensions and OtherName SANs and what not decoded recursively. So it's very nice to see a post about similar work! ASN.1 really is quite awesome. It gets a lot of hate, but it's not deserved. ASN.1 is not just a syntax or set of encoding rules -- it's a type system, and a very powerful one at that.

sedatk 1y ago

There's a Turkish saying, "a human will [use] this, a human!", to signify that the thing is so abnormal/out-of-proportion that it doesn't seem to be made for people. The verb changes based on the context. If you had made too much food for example, the verb would be "eat". I think it's a great motto for design. Remember the Game of Thrones quote, "the man who passes the sentence should swing the sword"? I think it should also be applied to specs. Anyone who comes up with a spec must be the first responsible party to develop a parser for it. The spec doesn't get ratified unless it comes with working parser code with unit tests. That kind of requirement might actually improve specs.

deepsun 1y ago

Wow, I needed to parse just one small ASN.1 with one value (one signature), but I didn't know ASN.1 can have a specification (to generate parser from it). So I ended up untangling it myself, just for that specific 256 bits. Still I think it's better to have overapecified format for security stuff, json and xml are just too vague and parsers are unpredictable.

arlyle 1y ago

see also fabrice bellard's ffasn1 (c from asn.1) compiler: https://bellard.org/ffasn1/ https://bellard.org/ffasn1/

cryptonector 1y ago
Oh yeah, amazing tooling no doubt, and non-free. Fabrice Bellard knew he could make good bank from this. ASN.1 for 3GPP and finance is a great little niche if you have a chance to exploit it.

lilyball 1y ago

I'm fascinated by ASN.1, I don't know why it appeals to me so much but I find working with it oddly fun. I've always wanted to find the time to write an ASN.1 compiler for Rust, because for some reason all of the Rust implementations I've seen end up just either being derive macros on Rust structs (so going the other direction), or even just providing a bunch of ASN.1 types and functions and expecting you to manually chain them together using nom.

jmspring 1y ago

As someone who extended the ASN.1 parser in the Netscape code base to handle PKCS#12 and at one point knew more about RSA standards and varios ASN.1 definitions than anyone should, I respect the blog owners diligence and masochism. It would not be the project I would pick out to learn D.

BradleyChatha 1y ago
> As someone who extended the ASN.1 parser in the Netscape code base to handle PKCS#12 Sounds like you have some fun potential war stories in that case.
- jmspring 11mo ago
  War story or nightmare? The asn.1 parsing in Netscape was an interesting beast. I also added some to a VPN I worked on while at another company.

themafia 1y ago

Ctrl-F "walterbright". huh. Must be on vacation today.

dardeaup 1y ago
Or he may just be tired of explaining his views over and over.

zzo38computer 1y ago

> Did I ever mention that ASN.1 is complicated? The schema format is complicated, and the data format also is, but most of it is ignorable complexity, often making it simpler than it is. I do not use the ASN.1 schema notation, since I have been able to use ASN.1 without it. (I wrote my own implementation of the data format (DER) in C.) > On the one hand the sheer amount of possible encodings is daunting, but on the other hand it shows a certain flexbility that ASN.1 provides - you could even invent your own domain-specific encoding if needed. It is true. My opinion is, of the standard encodings, DER is the only one that is any good (and is the one that I usually use (although most of my uses are not having to do with cryptography); X.509 (which is a common use of ASN.1) also uses DER), but I did make up my own encodings as well: DSER (between BER and DER), SDSER (like DSER but the length is encoded differently so that streaming is possible, in a (in my opinion) better way than CER does), and TER (a text format which is intended to be converted to DER). > ANY DEFINED BY I still effectively use it (as well as ANY), since I think it is very useful. (As I mentioned, I did not actually use the ASN.1 schema format, but I sometimes use what effectively works like ANY and ANY DEFINED BY.) (I also sometimes use a nonstandard feature that I made up, which is called OBJECT IDENTIFIER RELATIVE TO, which is like OBJECT IDENTIFIER but it can be encoded more efficiently (using the RELATIVE OID type) if the value has the prefix that the schema expects it to have.)