8 ms·

In short: I wanted to talk a bit about ASN.1, a bit about D, and a bit about the compiler itself, but couldn't think of any real cohesive format. So I threw a

In short: I wanted to talk a bit about ASN.1, a bit about D, and a bit about the compiler itself, but couldn't think of any real cohesive format. So I threw a bunch of semi-related ramblings together and I'm daring to call it a blog post. Sorry in advance since I will admit it's not the greatest quality, but it's really not easy to talk about so much with such brevity (especially since I've already forgot a ton of stuff I wanted to talk about more deeply :( )

whizzter 1y ago
As someone that had the dis-pleasure to work with Asn.1 data (yes, certificates) I fully symphatise with anguish you've gone through (that 6months of Ansible HR comments cracked me up also :D ).
- BradleyChatha 1y ago
  It makes me laugh that absolutely no one can say "I've worked with ASN.1" in a positive light :D
  - StopDisinfo910 1y ago
    There was an amusing chain of comments the last time protobuf was mentionned in which some people were arguing that it had been a terrible idea and ASN.1, as a standard, should have been used. It was hilarious because clearly none of the people who were in favor had ever used ASN.1.
    - mananaysiempre 1y ago
      Cryptonector[1] maintains an ASN.1 implementation[2] and usually has good things to say about the language and its specs. (Kind of surprised not he’s not in the comments here already :) ) [1] https://news.ycombinator.com/user?id=cryptonector https://news.ycombinator.com/user?id=cryptonector [2] https://github.com/heimdal/heimdal/tree/master/lib/asn1 https://github.com/heimdal/heimdal/tree/master/lib/asn1
      - cryptonector 1y ago
        Thanks for the shout-out! Yes, I do have nice things to say about ASN.1. It's all the others that mostly suck, with a few exceptions like XDR and DCE/Microsoft RPC's IDL.
        mananaysiempre 1y ago
        Derail accepted! Is your approval of DCE based only on the serialization not being TLV or on something else too? I have to say, while I do think its IDL is tasteful, its only real distinguishing feature is AFAICT the array-passing/returning stuff, and that feels much too specialized to make sense of in anything but C (or largely-isomorphic low-level languages like vernacular varieties of Pascal).
        cryptonector 1y ago
        Well, I do disapprove of the RPC-centric nature of both, XDR and DCE RPC, and I disapprove of the emphasis on "pointers" and -in the case of DCE- support for circular data structures and such. The 1980s penchant for "look ma'! I can have local things that are remote and you can't tell because I'm pretending that latency isn't part of the API hahahaha" research really shows in these. But yeah, at least they ain't TLV encodings, and the syntax is alright. I especially like XDR, though maybe that's because I worked at Sun Microsystems :) "Pointers" in XDR are really just `OPTIONAL` in ASN.1. Seems so silly to call them pointers. The reason they called them "pointers" is that that's how they represented optionality in the generated structures and code: if the field was present on the wire then the pointer is not null, and if it was absent the then pointer is null. And that's exactly what one does in ASN.1 tooling, though maybe with a host language Optional<> type rather than with pointers and null values. Whereas in hand-coded ASN.1 codecs one does sometimes see special values used as if the member had been `DEFAULT` rather than `OPTIONAL`.
    - whizzter 1y ago
      It's not entirely horrible, parsing DER dynamically enough to handle interpreting most common certificates can be done in some 200-300 lines of C#, so I'd take that any day over XML. The main problem is that to work with the data you need to understand the semantics of the magic object identifiers and while things like the PKIX module can be found easily, the definitions for other more obscure namespaces for extensions can be harder to locate as it's scattered in documentation from various standardization organizations. So, protobuf could very well have been transported in DER, the problem issue was probably more one of Google not seeing any value of interoperability and wanting to keep it simple (or worse, clashing by oblivious users re-using the wrong less well documented namespaces).
    - cryptonector 1y ago
      You're likely to find my comments among those saying that. I've been using ASN.1 in some way for a couple of decades, and I've been an ASN.1 implementor for about half a decade.
    - thayne 1y ago
      ASN.1 seems like something that could have been good ... If it was less complicated, had more accessible documentation, and had better tooling.
  - hamburglar 1y ago
    As a former PKI enthusiast (tongue firmly in cheek with that description) I can say if you can limit your exposure to simply issuing certs so you control the data and thus avoid all edge cases, quirks, non-canonical encodings, etc, dealing with ASN.1 is “not too terrible.” But it is bad. The thing that used to regularly amaze me was the insane depths of complexity the designers went to … back in the 70’s! It is astounding to me that they managed to make a system that encapsulated so much complexity and is still in everyday use today. You are truly a masochist and I salute you.
    - cyberax 1y ago
      It's also amazing that we're basically using only a couple of free-form text fields in the WebPKI for the most crucial parts of validation. Completely ignoring the ASN.1 support for complicated structures, with more than one CVE linked to incorrect parsing of these text fields m
      - cryptonector 1y ago
        No we're not. We're using dNSName subjectAlternativeName values. We used to use the CN attribute of the subject DN, and... there is still code for that, but it's obsolete. We _are_ using subject DNs for linking certs to their issuers, but though that's "free-form", we don't parse them, we only check for equality.
        cyberax 1y ago
        CN is absolutely used everywhere. And it can contain wildcards. SANs are also free-form.
        cryptonector 1y ago
        SANs are not free-form. A dNSName SAN is supposed to have an FQDN. An rfc822Name SAN is supposed to carry an email address. And, ok, sure, email addresses' mailbox part is basically free-form, but so what, you don't interpret that part unless you've accepted that certificate for that email address' domain part, and then you interpret the mailbox part the way a mail server would because you're probably the mail server. Yes, you can have directoryName SANs, but the whole point of SANs is that DNs suck because x.400/x.500 naming sucks so we want to use something that isn't that.
  - cryptonector 1y ago
    Bzzt! Wrong! I have worked with ASN.1 for many years, and I love ASN.1. :) Really, I do. In particular I like: - that ASN.1 is generic, not specific to a given encoding rules (compare to XDR, which is both a syntax and a codec specification) - that ASN.1 lets you get quite formal if you want to in your specifications For example, RFC 5280 is the base PKIX spec, and if you look at RFCs 5911 and 5912 you'll see the same types (and those of other PKIX-related RFCs) with more formalisms. I use those formalisms in the ASN.1 tooling I maintain to implement a recursive, one-shot codec for certificates in all their glory. - that ASN.1 has been through the whole evolution of "hey, TLV rules are all you need and you get extensibility for free!!1!" through "oh no, no that's not quite right is it" through "we should add extensibility functionality" and "hmm, tags should not really have to appear in modules, so let's add AUTOMATIC tagging" and "well, let's support lots of encoding rules, like non-TLV binary ones (PER, OER) and XML and JSON!". Protocol Buffers is still stuck on TLV, all done badly by comparison to BER/DER.
    - zzo38computer 1y ago
      I also like ASN.1; I think it is better than JSON, XML, Protocol Buffers, etc, in many ways. I use it in some of my programs. (However, like many other formats (including JSON, XML, etc), ASN.1 can be badly used.)
    - BradleyChatha 1y ago
      Yeah I know I'm making fun of it a lot (mostly in jest) but it genuinely is a really interesting specification, and it's definitely sad - but not surprising - it's not a very popular choice outside of its few niche areas. :) Glad to see someone else who's gone down this road as well.
    - tambre 1y ago
      How do you feel about something like CBOR? In which stage would you say it's stuck in evolution compared to ASN.1 (since you said Protobuf is still TLV)?
      - cryptonector 1y ago
  - coderjames 1y ago
    I worked with ASN.1 for a few years in the embedded space because its used for communications between aircraft and air traffic control in Europe [1]. I enjoyed it. BER encoding is pretty much the tightest way to represent messages on the wire and when you're charged per-bit for messaging, it all adds up. When a messaging syntax is defined in ASN.1 in an international standard (ICAO 9880 anyone?), its going to be around for a while. Haven't been able to get my current company to adopt ASN.1 to replace our existing homegrown serialization format. [1] https://en.wikipedia.org/wiki/Aeronautical_Telecommunication_Network https://en.wikipedia.org/wiki/Aeronautical_Telecommunication...
    - lepicz 1y ago
      of all the encoding i like BER the most as well (i worked in telecommunications when ASN.1 was common thing)
    - p_l 1y ago
      Isn't PER or OER more compact? especially for the per-bit charging thing
      - coderjames 1y ago
        Oh yeah, derp. I was thinking unaligned-PER, not BER.
  - HelloNurse 1y ago
    I suspect that typical interactions with ASN.1 are benign because people are interested in reading and writing a few specific preexisting data structures with whatever encoding is required for interoperability, not in designing new message structures and choosing encodings for them. For example, when I inherited a public key signature system (mainly retrieving certificates and feeding them to cryptographic primitives and downloading and checking certificate revocation lists) everything troublesome was left by dismissed consultants; there were libraries for dealing with ASN.1 and I only had to educate myself about the different messages and their structure, like with any other standard protocol.
- i2pi 1y ago
  (void) space. :P
throw_a_grenade 1y ago
Don't worry, it's your blog, and your way. Keep it up, if it makes you whole.
giancarlostoro 1y ago
At least you might be summoning Walter Bright in talking about D. One of my favorite languages I wish more companies would use. Unfortunately for its own sake, Go and Rust are way more popular in the industry.
- pjmlp 1y ago
  Unfortunately it lost the opportunity back when Remedy Games and Facebook were betting on it. The various WIP features, and switching focus of what might bring more people into the ecosystem, have given away to other languages. Even C#, Java and C++ have gotten many of features that were only available in D as Andrei Alexandrescu's book came out in 2011.
  - Clouudy 1y ago
    I wouldn't say that it's unable to make a comeback, there is still a valid use case from my experience with it. The syntax, mixed-memory model, UFCS, and compilation speed are nice quality of life features compared to C++, and it's still a native binary compared to C# and Java. So if you're starting out with a new project from scratch there's not much reason not to beyond documentation reasons. And you can interface pretty easily to C/C++ as well as pretty much any other language designed for that sort of thing, but without a lot of syntax changes like Carbon. I imagine that the scope of its uses has shrunk as other languages caught up, and I don't think it's necessarily a good language for general enterprise stuff (unless you're dealing with C++), but for new projects it's still valid IMO. I think that the biggest field it could be used in is probably games too, especially if you're already writing a new engine from scratch. You could start with the GC and then ease off of it as the project develops in order to speed up development, for example. And D could always add newer features again too, tbh.
    - pjmlp 1y ago
      You always have to compare ecosystems, not programming languages syntax on their own. Another thing that Java and C# got to do since 2011, is that AOT is also part of the ecosystem and free of charge (Java has had commercial compilers for a while), so not even a native binary is an advantage as you imagine. First D has to finish what is already there in features that are almost done but not quite.
olvy0 1y ago
Just wanted to say I enjoyed your post very much. Thank you for writing it. I love D but unfortunately I haven't touched it for several years. I also have some experience writing parsers and implementing protocols.
- BradleyChatha 1y ago
  Thank you :)
mananaysiempre 1y ago
A small nitpick: I don’t think your intersection example does what you want it to do. Perhaps there’s some obscure difference in “PER-visibility” or whatnot, but at least set-theoretically, LegacyFlags2 ::= INTEGER (0 | 2 ^ 4..8) -- as in the article is exactly equivalent to LegacyFlags2 ::= INTEGER (0) -- only a single value allowed as (using standard mathematical notation and making precedence explicit) {0} ∪ ({2} ∩ {4,5,6,7,8}) = {0} ∪ ∅ = {0}.

whizzter 1y ago

As someone that had the dis-pleasure to work with Asn.1 data (yes, certificates) I fully symphatise with anguish you've gone through (that 6months of Ansible HR comments cracked me up also :D ).

BradleyChatha 1y ago
It makes me laugh that absolutely no one can say "I've worked with ASN.1" in a positive light :D
- StopDisinfo910 1y ago
  There was an amusing chain of comments the last time protobuf was mentionned in which some people were arguing that it had been a terrible idea and ASN.1, as a standard, should have been used. It was hilarious because clearly none of the people who were in favor had ever used ASN.1.
  - mananaysiempre 1y ago
    Cryptonector[1] maintains an ASN.1 implementation[2] and usually has good things to say about the language and its specs. (Kind of surprised not he’s not in the comments here already :) ) [1] https://news.ycombinator.com/user?id=cryptonector https://news.ycombinator.com/user?id=cryptonector [2] https://github.com/heimdal/heimdal/tree/master/lib/asn1 https://github.com/heimdal/heimdal/tree/master/lib/asn1
    - cryptonector 1y ago
      Thanks for the shout-out! Yes, I do have nice things to say about ASN.1. It's all the others that mostly suck, with a few exceptions like XDR and DCE/Microsoft RPC's IDL.
      - mananaysiempre 1y ago
        Derail accepted! Is your approval of DCE based only on the serialization not being TLV or on something else too? I have to say, while I do think its IDL is tasteful, its only real distinguishing feature is AFAICT the array-passing/returning stuff, and that feels much too specialized to make sense of in anything but C (or largely-isomorphic low-level languages like vernacular varieties of Pascal).
        cryptonector 1y ago
        Well, I do disapprove of the RPC-centric nature of both, XDR and DCE RPC, and I disapprove of the emphasis on "pointers" and -in the case of DCE- support for circular data structures and such. The 1980s penchant for "look ma'! I can have local things that are remote and you can't tell because I'm pretending that latency isn't part of the API hahahaha" research really shows in these. But yeah, at least they ain't TLV encodings, and the syntax is alright. I especially like XDR, though maybe that's because I worked at Sun Microsystems :) "Pointers" in XDR are really just `OPTIONAL` in ASN.1. Seems so silly to call them pointers. The reason they called them "pointers" is that that's how they represented optionality in the generated structures and code: if the field was present on the wire then the pointer is not null, and if it was absent the then pointer is null. And that's exactly what one does in ASN.1 tooling, though maybe with a host language Optional<> type rather than with pointers and null values. Whereas in hand-coded ASN.1 codecs one does sometimes see special values used as if the member had been `DEFAULT` rather than `OPTIONAL`.
  - whizzter 1y ago
    It's not entirely horrible, parsing DER dynamically enough to handle interpreting most common certificates can be done in some 200-300 lines of C#, so I'd take that any day over XML. The main problem is that to work with the data you need to understand the semantics of the magic object identifiers and while things like the PKIX module can be found easily, the definitions for other more obscure namespaces for extensions can be harder to locate as it's scattered in documentation from various standardization organizations. So, protobuf could very well have been transported in DER, the problem issue was probably more one of Google not seeing any value of interoperability and wanting to keep it simple (or worse, clashing by oblivious users re-using the wrong less well documented namespaces).
  - cryptonector 1y ago
    You're likely to find my comments among those saying that. I've been using ASN.1 in some way for a couple of decades, and I've been an ASN.1 implementor for about half a decade.
  - thayne 1y ago
    ASN.1 seems like something that could have been good ... If it was less complicated, had more accessible documentation, and had better tooling.
- hamburglar 1y ago
  As a former PKI enthusiast (tongue firmly in cheek with that description) I can say if you can limit your exposure to simply issuing certs so you control the data and thus avoid all edge cases, quirks, non-canonical encodings, etc, dealing with ASN.1 is “not too terrible.” But it is bad. The thing that used to regularly amaze me was the insane depths of complexity the designers went to … back in the 70’s! It is astounding to me that they managed to make a system that encapsulated so much complexity and is still in everyday use today. You are truly a masochist and I salute you.
  - cyberax 1y ago
    It's also amazing that we're basically using only a couple of free-form text fields in the WebPKI for the most crucial parts of validation. Completely ignoring the ASN.1 support for complicated structures, with more than one CVE linked to incorrect parsing of these text fields m
    - cryptonector 1y ago
      No we're not. We're using dNSName subjectAlternativeName values. We used to use the CN attribute of the subject DN, and... there is still code for that, but it's obsolete. We _are_ using subject DNs for linking certs to their issuers, but though that's "free-form", we don't parse them, we only check for equality.
      - cyberax 1y ago
        CN is absolutely used everywhere. And it can contain wildcards. SANs are also free-form.
        cryptonector 1y ago
        SANs are not free-form. A dNSName SAN is supposed to have an FQDN. An rfc822Name SAN is supposed to carry an email address. And, ok, sure, email addresses' mailbox part is basically free-form, but so what, you don't interpret that part unless you've accepted that certificate for that email address' domain part, and then you interpret the mailbox part the way a mail server would because you're probably the mail server. Yes, you can have directoryName SANs, but the whole point of SANs is that DNs suck because x.400/x.500 naming sucks so we want to use something that isn't that.
- cryptonector 1y ago
  Bzzt! Wrong! I have worked with ASN.1 for many years, and I love ASN.1. :) Really, I do. In particular I like: - that ASN.1 is generic, not specific to a given encoding rules (compare to XDR, which is both a syntax and a codec specification) - that ASN.1 lets you get quite formal if you want to in your specifications For example, RFC 5280 is the base PKIX spec, and if you look at RFCs 5911 and 5912 you'll see the same types (and those of other PKIX-related RFCs) with more formalisms. I use those formalisms in the ASN.1 tooling I maintain to implement a recursive, one-shot codec for certificates in all their glory. - that ASN.1 has been through the whole evolution of "hey, TLV rules are all you need and you get extensibility for free!!1!" through "oh no, no that's not quite right is it" through "we should add extensibility functionality" and "hmm, tags should not really have to appear in modules, so let's add AUTOMATIC tagging" and "well, let's support lots of encoding rules, like non-TLV binary ones (PER, OER) and XML and JSON!". Protocol Buffers is still stuck on TLV, all done badly by comparison to BER/DER.
  - zzo38computer 1y ago
    I also like ASN.1; I think it is better than JSON, XML, Protocol Buffers, etc, in many ways. I use it in some of my programs. (However, like many other formats (including JSON, XML, etc), ASN.1 can be badly used.)
  - BradleyChatha 1y ago
    Yeah I know I'm making fun of it a lot (mostly in jest) but it genuinely is a really interesting specification, and it's definitely sad - but not surprising - it's not a very popular choice outside of its few niche areas. :) Glad to see someone else who's gone down this road as well.
  - tambre 1y ago
    How do you feel about something like CBOR? In which stage would you say it's stuck in evolution compared to ASN.1 (since you said Protobuf is still TLV)?
    - cryptonector 1y ago
- coderjames 1y ago
  I worked with ASN.1 for a few years in the embedded space because its used for communications between aircraft and air traffic control in Europe [1]. I enjoyed it. BER encoding is pretty much the tightest way to represent messages on the wire and when you're charged per-bit for messaging, it all adds up. When a messaging syntax is defined in ASN.1 in an international standard (ICAO 9880 anyone?), its going to be around for a while. Haven't been able to get my current company to adopt ASN.1 to replace our existing homegrown serialization format. [1] https://en.wikipedia.org/wiki/Aeronautical_Telecommunication_Network https://en.wikipedia.org/wiki/Aeronautical_Telecommunication...
  - lepicz 1y ago
    of all the encoding i like BER the most as well (i worked in telecommunications when ASN.1 was common thing)
  - p_l 1y ago
    Isn't PER or OER more compact? especially for the per-bit charging thing
    - coderjames 1y ago
      Oh yeah, derp. I was thinking unaligned-PER, not BER.
- HelloNurse 1y ago
  I suspect that typical interactions with ASN.1 are benign because people are interested in reading and writing a few specific preexisting data structures with whatever encoding is required for interoperability, not in designing new message structures and choosing encodings for them. For example, when I inherited a public key signature system (mainly retrieving certificates and feeding them to cryptographic primitives and downloading and checking certificate revocation lists) everything troublesome was left by dismissed consultants; there were libraries for dealing with ASN.1 and I only had to educate myself about the different messages and their structure, like with any other standard protocol.
i2pi 1y ago
(void) space. :P

throw_a_grenade 1y ago

Don't worry, it's your blog, and your way. Keep it up, if it makes you whole.

giancarlostoro 1y ago

At least you might be summoning Walter Bright in talking about D. One of my favorite languages I wish more companies would use. Unfortunately for its own sake, Go and Rust are way more popular in the industry.

pjmlp 1y ago
Unfortunately it lost the opportunity back when Remedy Games and Facebook were betting on it. The various WIP features, and switching focus of what might bring more people into the ecosystem, have given away to other languages. Even C#, Java and C++ have gotten many of features that were only available in D as Andrei Alexandrescu's book came out in 2011.
- Clouudy 1y ago
  I wouldn't say that it's unable to make a comeback, there is still a valid use case from my experience with it. The syntax, mixed-memory model, UFCS, and compilation speed are nice quality of life features compared to C++, and it's still a native binary compared to C# and Java. So if you're starting out with a new project from scratch there's not much reason not to beyond documentation reasons. And you can interface pretty easily to C/C++ as well as pretty much any other language designed for that sort of thing, but without a lot of syntax changes like Carbon. I imagine that the scope of its uses has shrunk as other languages caught up, and I don't think it's necessarily a good language for general enterprise stuff (unless you're dealing with C++), but for new projects it's still valid IMO. I think that the biggest field it could be used in is probably games too, especially if you're already writing a new engine from scratch. You could start with the GC and then ease off of it as the project develops in order to speed up development, for example. And D could always add newer features again too, tbh.
  - pjmlp 1y ago
    You always have to compare ecosystems, not programming languages syntax on their own. Another thing that Java and C# got to do since 2011, is that AOT is also part of the ecosystem and free of charge (Java has had commercial compilers for a while), so not even a native binary is an advantage as you imagine. First D has to finish what is already there in features that are almost done but not quite.

olvy0 1y ago

Just wanted to say I enjoyed your post very much. Thank you for writing it. I love D but unfortunately I haven't touched it for several years. I also have some experience writing parsers and implementing protocols.

BradleyChatha 1y ago
Thank you :)

mananaysiempre 1y ago

A small nitpick: I don’t think your intersection example does what you want it to do. Perhaps there’s some obscure difference in “PER-visibility” or whatnot, but at least set-theoretically, LegacyFlags2 ::= INTEGER (0 | 2 ^ 4..8) -- as in the article is exactly equivalent to LegacyFlags2 ::= INTEGER (0) -- only a single value allowed as (using standard mathematical notation and making precedence explicit) {0} ∪ ({2} ∩ {4,5,6,7,8}) = {0} ∪ ∅ = {0}.