9 ms·

A New ASN.1 API for Python

171 points by woodruffw 2y ago

woodrowbarlow 2y ago
neat! related: you can also create wireshark dissectors from ASN.1 files https://www.wireshark.org/docs/wsdg_html_chunked/ASN1StepByStepInstructions.html https://www.wireshark.org/docs/wsdg_html_chunked/ASN1StepByS...
venamresm__ 2y ago
In the ASN.1 space everyone hopes that someone can dethrone OSS Nokalva's proprietary solutions
- woodruffw 2y ago
  I think it's context-dependent: I don't have insight into OSS Nokalva's use inside big companies, but in the Open Source world it certainly isn't dominant. In Open Source, I think the biggest ASN.1 implementations I come across are OpenSSL's, libtasn1, asn1c, and then per-language implementations like pyasn1.
  - nicce 2y ago
    Basically any commercial ASN.1 compiler prevents usage of the output in any open-source project. There is that.
    - sobkas 2y ago
      Licence also prevent you from modifying generated code.
  - cryptonector 2y ago
    https://github.com/heimdal/heimdal/tree/master/lib/asn1 https://github.com/heimdal/heimdal/tree/master/lib/asn1
  - venamresm__ 2y ago
    Most of the open source tools need patching to properly support certain scenarios (been there done that). They also lack support for parsing ASN.1 Value Notation format (textual), which is used everywhere in specifications, OSS Nokalva offers the full set of tools to handle this even with a playground and ASN.1 editor, this is non-existent in open source right now. For now the open source tools only focus on the crypto aspect, and doesn't really dive into telco, banking, biometric, and others.
- memling 2y ago
  > In the ASN.1 space everyone hopes that someone can dethrone OSS Nokalva's proprietary solutions You're buying more than a compiler and runtime, though: you're also getting an SLA and a stricter guarantee about interoperability and bugs and so forth. I have no idea how good their support is (maybe it's atrocious?), but these are important. I had a client who relied on the open-sourced asn1c once who complained about some of the bugs they found in it; they got pushed into buying commercial when the cost-benefit outweighed the software licensing issues.
  - cryptonector 2y ago
    Meh. After all, if you're not using ASN.1 you're using something like ProtocolBuffers or FlatBuffers or whatever and all open source tooling.
    - memling 2y ago
      > Meh. After all, if you're not using ASN.1 you're using something like ProtocolBuffers or FlatBuffers or whatever and all open source tooling. Oh sure--there are plenty of alternatives to ASN.1. My guess is that most people who have the choice don't use ASN.1 precisely because open-source alternatives exist and can feasibly work for most use cases. But if you happen to have one of the use cases that require ASN.1, open sourced tooling can be problematic precisely because of the need for a robust SLA.
      - cryptonector 2y ago
        > But if you happen to have one of the use cases that require ASN.1, open sourced tooling can be problematic precisely because of the need for a robust SLA. Why would you need a support SLA for ASN.1 and not for PB/FB? That makes no sense. And there's plenty of open source ASN.1 tooling now -- just look around this thread!
        memling 2y ago
        > Why would you need a support SLA for ASN.1 and not for PB/FB? That makes no sense. And there's plenty of open source ASN.1 tooling now -- just look around this thread! If your business depends on five nines plus of reliability in your 5g communications stack, you might be willing to fork over the price for it. Or if you need a bug fix made in a timely fashion to the compiker or runtime, likewise. As I've noted above, a client of mine moved to a commercial suite of tools for this reason. Protobuf and flatbuffers have different use cases in my experience, although that's somewhat limited. Protobuf at least also introduced breaking changes between versions 2 and 3. ASN.1 isn't perfect in this regard, but these days incompatibikities have to go through ISO or ITU, etc. Your experience may be different of course. I'm just pointing out that there are reasons people will opt for a commercial product.
- cryptonector 2y ago
  I don't have the time, though I do have the inclination, to finish Heimdal's ASN.1 compiler, which is already quite awesome. u/lukeh used Heimdal's ASN.1 compiler's JSON transformation of modules output to build an ASN.1 compiler and runtime for Swift.
johnisgood 2y ago
Erlang also has great ASN.1 support. For the rest, I hope OSS Nokalva'a proprietary solutions will go away, eventually. For Java I used yafred's asn1-tool, which is apparently not available anymore. Other than that, it worked well. Originally it was available here: https://github.com/yafred/asn1-tool https://github.com/yafred/asn1-tool (archived: https://web.archive.org/web/20240416031004/https://github.com/yafred/asn1-tool https://web.archive.org/web/20240416031004/https://github.co...) Any recommendations?
- elFarto 2y ago
  I was using the asn1bean Java library yesterday funnily enough. I'm sure it's fine for X.509 stuff, however lucky me got to use it with the more obscure parts of X.400. It's lacking support for COMPONENTS OF, and a bunch of other things that were likely deprecated from the ASN.1 spec a few decades ago.
  - johnisgood 2y ago
    Any luck with it? Check the README of: https://web.archive.org/web/20240416031004/https://github.com/yafred/asn1-tool https://web.archive.org/web/20240416031004/https://github.co... I need something like this.
- nightpool 2y ago
  https://mvnrepository.com/artifact/de.rub.nds/asn1-tool https://mvnrepository.com/artifact/de.rub.nds/asn1-tool links to https://github.com/tls-attacker/ASN.1-Tool https://github.com/tls-attacker/ASN.1-Tool which redirects to https://github.com/tls-attacker/ASN.1-Attacker https://github.com/tls-attacker/ASN.1-Attacker, so that seems like it's the new name for the tool you're looking for.
  - johnisgood 2y ago
    No, it is not the same. Sadly the author probably made it private, might reach out to him, just don't know how. :( He did not put his e-mail anywhere from what I have found so far. https://github.com/zhonghuihuo/asn1-tool https://github.com/zhonghuihuo/asn1-tool is still available, but it is very old, it is probably a VERY old fork. I need something like this :(. I need it for Java / Kotlin. I do not have the repository cloned, so I am kind of in the dark.[1] Found the archived page of the previously mentioned project that is probably private now: https://web.archive.org/web/20240416031004/https://github.com/yafred/asn1-tool https://web.archive.org/web/20240416031004/https://github.co... [1] Never mind, I found the newest (probably) asn1-compiler.jar! I still need an actively maintained alternative, however, for Java / Kotlin. For ASN.1 between Erlang / Elixir <> Java / Kotlin.
nicce 2y ago
There is also rasn library for Rust that now supports most of the codecs (BER/CER/DER/PER/APER/OER/COER/JER/XER). Disclaimer: I have contributed a lot recently. OER codec (modern flair of ASN.1) is very optimized (almost as much as it can be with safe Rust and without CPU specific stuff). I am still working with benchmarking results, which I plan to share in close future. But it starts to be the fastest there is in open-source world. It is also faster than Google's Protobuf libraries or any Protobuf library in Rust. (naive comparison, no reflection support). Hopefully other codecs could be optimized too. [1] https://github.com/librasn/rasn https://github.com/librasn/rasn
- timewizard 2y ago
  [flagged]
  - mananaysiempre 2y ago
    The Python library proposed in TFA is to be based on a different, DER-only Rust ASN.1 library[1]. So ASN.1 in Rust is more than tangentially relevant here. [1] https://github.com/alex/rust-asn1 https://github.com/alex/rust-asn1
    - timewizard 2y ago
      The discussion of rust and ASN.1 libraries and interoperability with other languages might be. The relative differences between the two libraries might be. The efforts of the author to get their library into Python might be. Simply advertising a different project, with all the standard "rust tropes," is not what most people would consider relevant. It's hamfisted and weird.
- deleted 2y ago
  [deleted]
- lilyball 2y ago
  This one looks interesting. A few years ago I looked at all of the Rust ASN.1 libraries I could find and they all had various issues. I'm a little surprised I didn't find this one.
dikei 2y ago
DER is still easy, UPER (unaligned packed encoding rules) is so much harder, yet it's prevalent in Telecom industry. Last I checked, there was no freely available tool than can handle UPER l00%
- bryancoxwell 2y ago
  Not only is UPER hard to parse, but (I believe) 3GPP ASN1 definitions are provided only in .docx files which aren’t exactly the easiest to work with. It’s just really not a fun domain.
  - jeroenhd 2y ago
    The ASN.1 format itself isn't too bad. It shows its age and has some very weird decisions behind it as places, but it's not that difficult to encode and is quite efficient. Unfortunately, the protocols themselves can be confusing, badly (or worse: inconsistently) documented, and the binary formats often lack sensible backwards compatibility (or, even worse: optional backwards compatibility). Definitions are spread across different protocols (and versions thereof) and vendors within the space like to make their own custom protocols that are like the official standardised protocols, but slightly different in infuriating ways. If you parser works (something open source rarely cares about so good luck finding one for your platform), the definitions extracted from those DOCX files are probably the least of your challenges.
  - sobkas 2y ago
    First you can download specifications in either PDF or doc(x). Second doc(x) are simple enough that simple doc(x) to ASCII/text is good enough to produce working ASN.1 definition. Copy&paste is also an option.
- masklinn 2y ago
  FWIW rasn linked above claims to support UPER, but I couldn't tell you how completely.
  - nicce 2y ago
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
- deleted 2y ago
  [deleted]
ChuckMcM 2y ago
A small bit of historical context. When I was participating in the PKP meetings at RSADSI, I believe it was Ron who insisted that DER was the only reasonable choice if we were going to encode things with ASN.1 (which we were because both DEC and RSA had already insisted that it had to be OSI compatible or they wouldn't support it, my suggestion that we use Sun's XDR was soundly rebuked, but hey I had to offer) Generally it was presumed that because these were 'handshake' type steps (which is to say the prelude to establishing a cryptographic context for what would happen next) performance wasn't as important as determinism.
- OhMeadhbh 2y ago
  oh. did i meet you there? i was contracting at RSADSI at the time and argued w/ Burt K. about how easy it was to mess up a general DER parser, much less an ASN.1 compiler. I remember we found about two bugs per week in ICL's compiler. Burt and Ron were BIG ASN.1 fans at the time and I could never figure out why. Ron kept pushing Burt and Bob Baldwin to include more generic ASN.1 features in BSAFE. Part of my misery during SET development can be directly traced to ICL's crappy ASN.1 compiler, yet it was probably the best one on the market at the time. Anywho... XDR isn't my favourite, but I would have definitely preferred it to DER/BER/ASN.1. Stop me before I make a CORBA reference.
  - ChuckMcM 2y ago
    > oh. did i meet you there? Probably :-). Ron was a huge fan of Roger Needham's (and, ngl, I was too) and Roger along with Andy Birrell and others were on a kick to make RPCs "seamless" so that you could reason about them like you did computer programs that were all local. Roger and I debated whether or not it was achievable (vs. desirable) at Cambridge when we had the PKI meeting there. We both agreed that computers would get fast and cheap enough that the value of having a canonical form on the wire vastly outweighed any disadvantage that "some" clients would have to conversion to put things in a native format they understood. (Andy wasn't convinced of that, at least at that time). But I believe that was the principle behind the insistence on ASN.1, determinism and canonical formats. Once you built the marshalling/unmarshalling libraries you could treat them as a constant tax on latency. That made analyzing state machines easier and debugging race conditions. Plus when they improved you could just replace the constants you used for the time it would take.
flowerthoughts 2y ago
Related: if you ever want to create your own serialization format, please at least have a cursory look at the basics of ASN.1. It's very complete both in terms of textual descriptions (how it started) and breadth of encoding rules (because it's practical.) (You can skip the classes and macros, though they are indeed cool...)
- tptacek 2y ago
  This sounds dangerously like a suggestion that more people use ASN.1.
  - RainyDayTmrw 2y ago
    What should people use today, given the choice, that isn't ASN.1? Edited to add: If they need something with a canonical byte representation, for example for hashing or MAC purposes?
    - wglb 2y ago
      Here are some issues: https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=asn.1 https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=asn.1
      - zzo38computer 2y ago
        Some of the bug reports here are not actually about ASN.1 even if they are in programs that also use ASN.1. However, some are actually about ASN.1. But, there are bugs in many computer programs, whether or not they use ASN.1, anyways. CVE-2022-0778 does not seem to be about ASN.1 (although ASN.1 is mentioned in the description); it seems to be a bug with computing a modular square root for non-prime moduli, and these numbers can come from any source and does not necessarily have anything to do with ASN.1. CVE-2021-3712 does have to do with ASN.1 implementation, but this is a bad assumption in some other parts of the program that use the ASN.1 structure. (My own implementation also does not require the string stored in the ASN1_Value structure to be null-terminated, but none of the functions implicitly null-terminate it or expect it to be. One reason for this is to avoid memory allocations when they are not needed.) Many programs dealing with OIDs have problems with it, since the program is badly designed; a properly designed program (which is not that difficult to do) will not have these problems with OIDs. It is rarely necessary to decode OIDs, except for display (my own implementation limits it to 160 digits per part when displaying a OID, which is probably much more than is needed, but should avoid the problem described in CVE-2023-2650 anyways). When comparing OIDs, you can compare them in binary format directly (if one is in text format, you should convert that one to binary to compare them, instead of the other way around). If you only want to validate OIDs, that can be done without decoding the numbers: Check that it is at least one byte long, the first byte is not 0x80, the last byte does not have the high bit set, and any byte that does not have the high bit set is not immediately followed by a byte 0x80. (The same validation can apply to relative OIDs, although some applications may allow relative OIDs to be empty, which is OK; but absolute OIDs are never allowed to be empty.) (Some other reports listed there also relate to OIDs. If the program is well-designed then it will not have these problems, as I described.) My own implementation never decodes ASN.1 values until you explicitly tell it to do so, with a function according to the type being decoded, and returns an error condition if the type is incorrect. All values are stored in a ASN1_Value structure which works the same way. Some of the CVE reports are about things which can occur just as easily in other programs not related to ASN.1. Things such as buffer overflows, improperly determining the length, integer overflows, etc, can potentially occur in any other program too. None of the things listed by CVE reports seems to be inherent security issues with ASN.1 itself.
dec0dedab0de 2y ago
Does anyone miss when "pure python" was a selling point of your library? I understand the need for speed, but I wish it were more common to ship a compiled binary that does the thing fast, as well as the same algorithm in python to fall back on if it's not available.
- pphysch 2y ago
  I certainly don't miss needing to install additional system libraries in addition to my pip install.
  - pjmlp 2y ago
    In what way does having to compile a Rust library prevent this?
    - pphysch 2y ago
      IME, and I may be off base, the new generation of Rust/Go binaries have a more "batteries-included" philosophy, i.e. developers don't assume that they can piggyback off existing user system libraries, which generally makes it a nicer install UX.
      - pjmlp 2y ago
        It will fail to compile as well if they depend on any system library or OS specific feature not present on your system.
- pjmlp 2y ago
  I rather find tragic that contrary to other dynamic languages, Python seems to fall under the curse of rewriting bindings into C and C++, or nowadays more fashionable Rust. And yes, Smalltalk, Self and various Lisp variants are just as dynamic.
  - woodruffw 2y ago
    Why is it tragic? It's more or less idiomatic in Python to put the hot or performance-sensitive paths of a package in native code; Rust has arguably made that into a much safer practice.
orthecreedence 2y ago
I was writing a cryptographically-inclined system with serialization in msgpack. At one point, I upgraded the libraries I was using and all my signatures started breaking because the msgpack library started using a different representation under the hood for some of my data structures. That's when I did some research and found ASN.1 DER and haven't really looked back since switching over to it. If you plan on signing your data structures and don't want to implement your own serialization format, give ASN.1 DER a look.
- amluto 2y ago
  If you are planning to sign your data structures, IMO your first choice should be to sign byte strings: be explicit that the thing that is signed is a specific string of bytes (which cryptographic protocol people love to call octets). Anything interpreting the signed data needs to start with those bytes and interpret them — do NOT assume that, just because you have some data structure that you think serializes to those bytes, then that data structure is authentic. Many, many cryptographic disasters would have been avoided by following the advice above.
  - RainyDayTmrw 2y ago
    That matches the advice from Latacora[1]. That advice makes a lot of sense from a security correctness and surface area perspective. There's a potential developer experience and efficiency concern, though. This likely forces two deserialization operations, and therefore two big memory copies, once for deserializing the envelope and once for deserializing the inner message. If we assume that most of the outer message is the inner message, and relatively little of it is the signature or MAC, then our extra memory copy is for almost the full length of the full message. [1]: https://www.latacora.com/blog/2019/07/24/how-not-to/ https://www.latacora.com/blog/2019/07/24/how-not-to/
    - amluto 2y ago
      There are a few serialization/deserialization systems that are close enough to zero-copy that this has no overhead. Cap’n Proto and FlatBuffers were designed around roughly this idea. Even some protobuf implementations allow in-place reads of bytes.
retrocryptid 2y ago
[flagged]
lilyball 2y ago
Oh right, the asn1 crate, which supports CHOICE but only up to 3 alternatives, which means it can't even be used to implement X.509 certificate decoding. Makes me wonder what they're going to do when they get that far.
- woodruffw 2y ago
  I don't know why you think this; the asn1-rust crate is in fact used to implement not only X.509 certificate decoding[1] but also path validation[2]. Edit: Here's an example of a CHOICE implemented with rust-asn1 that has more than three variants[3]. [1]: https://cryptography.io/en/latest/x509/reference/ https://cryptography.io/en/latest/x509/reference/ [2]: https://cryptography.io/en/latest/x509/verification/ https://cryptography.io/en/latest/x509/verification/ [3]: https://github.com/pyca/cryptography/blob/be6c53dd03172dde6af2de775a9d8e65e6b60fb0/src/rust/cryptography-x509/src/name.rs#L52-L83 https://github.com/pyca/cryptography/blob/be6c53dd03172dde6a...
  - Arnavion 2y ago
    The asn1 crate provides three builtin Choice enums itself - asn1::{Choice1, Choice2, Choice3} - which support 1, 2 and 3 choices respectively. I assume that is what lilyball is referring to. But as you correctly point out the custom derive supports mapping enums with more variants to CHOICE just fine, so the builtin enums are not a relevant limitation.
mootptr 2y ago
Parser differential exploits are a understated problem, especially with ASN.1, which I didn't expect to see anyone thinking about. Kudos on this initiative!
- kccqzy 2y ago
  I understand that it is a problem but I'm more used to seeing arguments that monocultures and single implementations are bad: WebSQL for example didn't become a standard because there was only a single implementation.
  - cryptonector 2y ago
    Where is the monoculture here?
    - kccqzy 2y ago
      If there were only one implementation for ASN.1 people would decry that whatever that implementation does effectively becomes the standard, and people would be clamoring to write a second implementation.
      - cryptonector 2y ago
        Ok, but there are many implementations. And the ASN.1 specs are really clear and readable (once you have a mental model of them).
        kccqzy 2y ago
        Yes, which is why the contrast with monoculture is interesting. Hence my original comment.
        cryptonector 2y ago
        Ah, I see. Thanks.
benatkin 2y ago
From the post > with the help of funding from Alpha-Omega From the site: > funded by Microsoft, Google, and Amazon Also it's a Linux Foundation project. Interesting. Python's a big community, and there's some disagreement here over whether this would be better done in pure python. I think it's good that there's a rust/cloud contingent in python land but hope pure python remains popular.
- woodruffw 2y ago
  For the record: I shopped this project to them; it didn’t originate as an idea from LF or any major company. The idea itself came from PyCA’s maintainers, who have wanted this for a while. (The native code desirability question is also a red herring here, since PyCA has and will always need to have native code for cryptographic operations. So having a first class ASN.1 API in PyCA was always going to involve native code, with the only variable being how.)
time4tea 2y ago
20+ years ago used ASN.1 for talking between micro services. (HTTP Services, as they were called then) Very performant. Had to buy OSS tools licence but other than that quite nice.
zzo38computer 2y ago
I do not use Python, but I wrote my own library in C for reading/writing DER. (I have made a variant, which adds a few new types such as: key/value list, BCD, TRON character code, etc. The program works even if you do not use these new types.) DER does have the advantages they mention in that article, and other advantages. Some people mention that DER is not compact or not efficient; but often what is used instead is formats that are even less compact or less efficient than DER, and/or that are significantly more complicated to handle.

171 points by woodruffw 2y ago

neat! related: you can also create wireshark dissectors from ASN.1 files https://www.wireshark.org/docs/wsdg_html_chunked/ASN1StepByStepInstructions.html https://www.wireshark.org/docs/wsdg_html_chunked/ASN1StepByS...

venamresm__ 2y ago

In the ASN.1 space everyone hopes that someone can dethrone OSS Nokalva's proprietary solutions

woodruffw 2y ago
I think it's context-dependent: I don't have insight into OSS Nokalva's use inside big companies, but in the Open Source world it certainly isn't dominant. In Open Source, I think the biggest ASN.1 implementations I come across are OpenSSL's, libtasn1, asn1c, and then per-language implementations like pyasn1.
- nicce 2y ago
  Basically any commercial ASN.1 compiler prevents usage of the output in any open-source project. There is that.
  - sobkas 2y ago
    Licence also prevent you from modifying generated code.
- cryptonector 2y ago
  https://github.com/heimdal/heimdal/tree/master/lib/asn1 https://github.com/heimdal/heimdal/tree/master/lib/asn1
- venamresm__ 2y ago
  Most of the open source tools need patching to properly support certain scenarios (been there done that). They also lack support for parsing ASN.1 Value Notation format (textual), which is used everywhere in specifications, OSS Nokalva offers the full set of tools to handle this even with a playground and ASN.1 editor, this is non-existent in open source right now. For now the open source tools only focus on the crypto aspect, and doesn't really dive into telco, banking, biometric, and others.
memling 2y ago
> In the ASN.1 space everyone hopes that someone can dethrone OSS Nokalva's proprietary solutions You're buying more than a compiler and runtime, though: you're also getting an SLA and a stricter guarantee about interoperability and bugs and so forth. I have no idea how good their support is (maybe it's atrocious?), but these are important. I had a client who relied on the open-sourced asn1c once who complained about some of the bugs they found in it; they got pushed into buying commercial when the cost-benefit outweighed the software licensing issues.
- cryptonector 2y ago
  Meh. After all, if you're not using ASN.1 you're using something like ProtocolBuffers or FlatBuffers or whatever and all open source tooling.
  - memling 2y ago
    > Meh. After all, if you're not using ASN.1 you're using something like ProtocolBuffers or FlatBuffers or whatever and all open source tooling. Oh sure--there are plenty of alternatives to ASN.1. My guess is that most people who have the choice don't use ASN.1 precisely because open-source alternatives exist and can feasibly work for most use cases. But if you happen to have one of the use cases that require ASN.1, open sourced tooling can be problematic precisely because of the need for a robust SLA.
    - cryptonector 2y ago
      > But if you happen to have one of the use cases that require ASN.1, open sourced tooling can be problematic precisely because of the need for a robust SLA. Why would you need a support SLA for ASN.1 and not for PB/FB? That makes no sense. And there's plenty of open source ASN.1 tooling now -- just look around this thread!
      - memling 2y ago
        > Why would you need a support SLA for ASN.1 and not for PB/FB? That makes no sense. And there's plenty of open source ASN.1 tooling now -- just look around this thread! If your business depends on five nines plus of reliability in your 5g communications stack, you might be willing to fork over the price for it. Or if you need a bug fix made in a timely fashion to the compiker or runtime, likewise. As I've noted above, a client of mine moved to a commercial suite of tools for this reason. Protobuf and flatbuffers have different use cases in my experience, although that's somewhat limited. Protobuf at least also introduced breaking changes between versions 2 and 3. ASN.1 isn't perfect in this regard, but these days incompatibikities have to go through ISO or ITU, etc. Your experience may be different of course. I'm just pointing out that there are reasons people will opt for a commercial product.
cryptonector 2y ago
I don't have the time, though I do have the inclination, to finish Heimdal's ASN.1 compiler, which is already quite awesome. u/lukeh used Heimdal's ASN.1 compiler's JSON transformation of modules output to build an ASN.1 compiler and runtime for Swift.

johnisgood 2y ago

Erlang also has great ASN.1 support. For the rest, I hope OSS Nokalva'a proprietary solutions will go away, eventually. For Java I used yafred's asn1-tool, which is apparently not available anymore. Other than that, it worked well. Originally it was available here: https://github.com/yafred/asn1-tool https://github.com/yafred/asn1-tool (archived: https://web.archive.org/web/20240416031004/https://github.com/yafred/asn1-tool https://web.archive.org/web/20240416031004/https://github.co...) Any recommendations?

elFarto 2y ago
I was using the asn1bean Java library yesterday funnily enough. I'm sure it's fine for X.509 stuff, however lucky me got to use it with the more obscure parts of X.400. It's lacking support for COMPONENTS OF, and a bunch of other things that were likely deprecated from the ASN.1 spec a few decades ago.
- johnisgood 2y ago
  Any luck with it? Check the README of: https://web.archive.org/web/20240416031004/https://github.com/yafred/asn1-tool https://web.archive.org/web/20240416031004/https://github.co... I need something like this.
nightpool 2y ago
https://mvnrepository.com/artifact/de.rub.nds/asn1-tool https://mvnrepository.com/artifact/de.rub.nds/asn1-tool links to https://github.com/tls-attacker/ASN.1-Tool https://github.com/tls-attacker/ASN.1-Tool which redirects to https://github.com/tls-attacker/ASN.1-Attacker https://github.com/tls-attacker/ASN.1-Attacker, so that seems like it's the new name for the tool you're looking for.
- johnisgood 2y ago
  No, it is not the same. Sadly the author probably made it private, might reach out to him, just don't know how. :( He did not put his e-mail anywhere from what I have found so far. https://github.com/zhonghuihuo/asn1-tool https://github.com/zhonghuihuo/asn1-tool is still available, but it is very old, it is probably a VERY old fork. I need something like this :(. I need it for Java / Kotlin. I do not have the repository cloned, so I am kind of in the dark.[1] Found the archived page of the previously mentioned project that is probably private now: https://web.archive.org/web/20240416031004/https://github.com/yafred/asn1-tool https://web.archive.org/web/20240416031004/https://github.co... [1] Never mind, I found the newest (probably) asn1-compiler.jar! I still need an actively maintained alternative, however, for Java / Kotlin. For ASN.1 between Erlang / Elixir <> Java / Kotlin.

nicce 2y ago

There is also rasn library for Rust that now supports most of the codecs (BER/CER/DER/PER/APER/OER/COER/JER/XER). Disclaimer: I have contributed a lot recently. OER codec (modern flair of ASN.1) is very optimized (almost as much as it can be with safe Rust and without CPU specific stuff). I am still working with benchmarking results, which I plan to share in close future. But it starts to be the fastest there is in open-source world. It is also faster than Google's Protobuf libraries or any Protobuf library in Rust. (naive comparison, no reflection support). Hopefully other codecs could be optimized too. [1] https://github.com/librasn/rasn https://github.com/librasn/rasn

timewizard 2y ago
[flagged]
- mananaysiempre 2y ago
  The Python library proposed in TFA is to be based on a different, DER-only Rust ASN.1 library[1]. So ASN.1 in Rust is more than tangentially relevant here. [1] https://github.com/alex/rust-asn1 https://github.com/alex/rust-asn1
  - timewizard 2y ago
    The discussion of rust and ASN.1 libraries and interoperability with other languages might be. The relative differences between the two libraries might be. The efforts of the author to get their library into Python might be. Simply advertising a different project, with all the standard "rust tropes," is not what most people would consider relevant. It's hamfisted and weird.
deleted 2y ago
[deleted]
lilyball 2y ago
This one looks interesting. A few years ago I looked at all of the Rust ASN.1 libraries I could find and they all had various issues. I'm a little surprised I didn't find this one.

dikei 2y ago

DER is still easy, UPER (unaligned packed encoding rules) is so much harder, yet it's prevalent in Telecom industry. Last I checked, there was no freely available tool than can handle UPER l00%

bryancoxwell 2y ago
Not only is UPER hard to parse, but (I believe) 3GPP ASN1 definitions are provided only in .docx files which aren’t exactly the easiest to work with. It’s just really not a fun domain.
- jeroenhd 2y ago
  The ASN.1 format itself isn't too bad. It shows its age and has some very weird decisions behind it as places, but it's not that difficult to encode and is quite efficient. Unfortunately, the protocols themselves can be confusing, badly (or worse: inconsistently) documented, and the binary formats often lack sensible backwards compatibility (or, even worse: optional backwards compatibility). Definitions are spread across different protocols (and versions thereof) and vendors within the space like to make their own custom protocols that are like the official standardised protocols, but slightly different in infuriating ways. If you parser works (something open source rarely cares about so good luck finding one for your platform), the definitions extracted from those DOCX files are probably the least of your challenges.
- sobkas 2y ago
  First you can download specifications in either PDF or doc(x). Second doc(x) are simple enough that simple doc(x) to ASCII/text is good enough to produce working ASN.1 definition. Copy&paste is also an option.
masklinn 2y ago
FWIW rasn linked above claims to support UPER, but I couldn't tell you how completely.
- nicce 2y ago

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

deleted 2y ago
[deleted]

ChuckMcM 2y ago

A small bit of historical context. When I was participating in the PKP meetings at RSADSI, I believe it was Ron who insisted that DER was the only reasonable choice if we were going to encode things with ASN.1 (which we were because both DEC and RSA had already insisted that it had to be OSI compatible or they wouldn't support it, my suggestion that we use Sun's XDR was soundly rebuked, but hey I had to offer) Generally it was presumed that because these were 'handshake' type steps (which is to say the prelude to establishing a cryptographic context for what would happen next) performance wasn't as important as determinism.

OhMeadhbh 2y ago
oh. did i meet you there? i was contracting at RSADSI at the time and argued w/ Burt K. about how easy it was to mess up a general DER parser, much less an ASN.1 compiler. I remember we found about two bugs per week in ICL's compiler. Burt and Ron were BIG ASN.1 fans at the time and I could never figure out why. Ron kept pushing Burt and Bob Baldwin to include more generic ASN.1 features in BSAFE. Part of my misery during SET development can be directly traced to ICL's crappy ASN.1 compiler, yet it was probably the best one on the market at the time. Anywho... XDR isn't my favourite, but I would have definitely preferred it to DER/BER/ASN.1. Stop me before I make a CORBA reference.
- ChuckMcM 2y ago
  > oh. did i meet you there? Probably :-). Ron was a huge fan of Roger Needham's (and, ngl, I was too) and Roger along with Andy Birrell and others were on a kick to make RPCs "seamless" so that you could reason about them like you did computer programs that were all local. Roger and I debated whether or not it was achievable (vs. desirable) at Cambridge when we had the PKI meeting there. We both agreed that computers would get fast and cheap enough that the value of having a canonical form on the wire vastly outweighed any disadvantage that "some" clients would have to conversion to put things in a native format they understood. (Andy wasn't convinced of that, at least at that time). But I believe that was the principle behind the insistence on ASN.1, determinism and canonical formats. Once you built the marshalling/unmarshalling libraries you could treat them as a constant tax on latency. That made analyzing state machines easier and debugging race conditions. Plus when they improved you could just replace the constants you used for the time it would take.

flowerthoughts 2y ago

Related: if you ever want to create your own serialization format, please at least have a cursory look at the basics of ASN.1. It's very complete both in terms of textual descriptions (how it started) and breadth of encoding rules (because it's practical.) (You can skip the classes and macros, though they are indeed cool...)

tptacek 2y ago
This sounds dangerously like a suggestion that more people use ASN.1.
- RainyDayTmrw 2y ago
  What should people use today, given the choice, that isn't ASN.1? Edited to add: If they need something with a canonical byte representation, for example for hashing or MAC purposes?
  - wglb 2y ago
    Here are some issues: https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=asn.1 https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=asn.1
    - zzo38computer 2y ago
      Some of the bug reports here are not actually about ASN.1 even if they are in programs that also use ASN.1. However, some are actually about ASN.1. But, there are bugs in many computer programs, whether or not they use ASN.1, anyways. CVE-2022-0778 does not seem to be about ASN.1 (although ASN.1 is mentioned in the description); it seems to be a bug with computing a modular square root for non-prime moduli, and these numbers can come from any source and does not necessarily have anything to do with ASN.1. CVE-2021-3712 does have to do with ASN.1 implementation, but this is a bad assumption in some other parts of the program that use the ASN.1 structure. (My own implementation also does not require the string stored in the ASN1_Value structure to be null-terminated, but none of the functions implicitly null-terminate it or expect it to be. One reason for this is to avoid memory allocations when they are not needed.) Many programs dealing with OIDs have problems with it, since the program is badly designed; a properly designed program (which is not that difficult to do) will not have these problems with OIDs. It is rarely necessary to decode OIDs, except for display (my own implementation limits it to 160 digits per part when displaying a OID, which is probably much more than is needed, but should avoid the problem described in CVE-2023-2650 anyways). When comparing OIDs, you can compare them in binary format directly (if one is in text format, you should convert that one to binary to compare them, instead of the other way around). If you only want to validate OIDs, that can be done without decoding the numbers: Check that it is at least one byte long, the first byte is not 0x80, the last byte does not have the high bit set, and any byte that does not have the high bit set is not immediately followed by a byte 0x80. (The same validation can apply to relative OIDs, although some applications may allow relative OIDs to be empty, which is OK; but absolute OIDs are never allowed to be empty.) (Some other reports listed there also relate to OIDs. If the program is well-designed then it will not have these problems, as I described.) My own implementation never decodes ASN.1 values until you explicitly tell it to do so, with a function according to the type being decoded, and returns an error condition if the type is incorrect. All values are stored in a ASN1_Value structure which works the same way. Some of the CVE reports are about things which can occur just as easily in other programs not related to ASN.1. Things such as buffer overflows, improperly determining the length, integer overflows, etc, can potentially occur in any other program too. None of the things listed by CVE reports seems to be inherent security issues with ASN.1 itself.

dec0dedab0de 2y ago

Does anyone miss when "pure python" was a selling point of your library? I understand the need for speed, but I wish it were more common to ship a compiled binary that does the thing fast, as well as the same algorithm in python to fall back on if it's not available.

pphysch 2y ago
I certainly don't miss needing to install additional system libraries in addition to my pip install.
- pjmlp 2y ago
  In what way does having to compile a Rust library prevent this?
  - pphysch 2y ago
    IME, and I may be off base, the new generation of Rust/Go binaries have a more "batteries-included" philosophy, i.e. developers don't assume that they can piggyback off existing user system libraries, which generally makes it a nicer install UX.
    - pjmlp 2y ago
      It will fail to compile as well if they depend on any system library or OS specific feature not present on your system.
pjmlp 2y ago
I rather find tragic that contrary to other dynamic languages, Python seems to fall under the curse of rewriting bindings into C and C++, or nowadays more fashionable Rust. And yes, Smalltalk, Self and various Lisp variants are just as dynamic.
- woodruffw 2y ago
  Why is it tragic? It's more or less idiomatic in Python to put the hot or performance-sensitive paths of a package in native code; Rust has arguably made that into a much safer practice.

orthecreedence 2y ago

I was writing a cryptographically-inclined system with serialization in msgpack. At one point, I upgraded the libraries I was using and all my signatures started breaking because the msgpack library started using a different representation under the hood for some of my data structures. That's when I did some research and found ASN.1 DER and haven't really looked back since switching over to it. If you plan on signing your data structures and don't want to implement your own serialization format, give ASN.1 DER a look.

amluto 2y ago
If you are planning to sign your data structures, IMO your first choice should be to sign byte strings: be explicit that the thing that is signed is a specific string of bytes (which cryptographic protocol people love to call octets). Anything interpreting the signed data needs to start with those bytes and interpret them — do NOT assume that, just because you have some data structure that you think serializes to those bytes, then that data structure is authentic. Many, many cryptographic disasters would have been avoided by following the advice above.
- RainyDayTmrw 2y ago
  That matches the advice from Latacora[1]. That advice makes a lot of sense from a security correctness and surface area perspective. There's a potential developer experience and efficiency concern, though. This likely forces two deserialization operations, and therefore two big memory copies, once for deserializing the envelope and once for deserializing the inner message. If we assume that most of the outer message is the inner message, and relatively little of it is the signature or MAC, then our extra memory copy is for almost the full length of the full message. [1]: https://www.latacora.com/blog/2019/07/24/how-not-to/ https://www.latacora.com/blog/2019/07/24/how-not-to/
  - amluto 2y ago
    There are a few serialization/deserialization systems that are close enough to zero-copy that this has no overhead. Cap’n Proto and FlatBuffers were designed around roughly this idea. Even some protobuf implementations allow in-place reads of bytes.

retrocryptid 2y ago

[flagged]

lilyball 2y ago

Oh right, the asn1 crate, which supports CHOICE but only up to 3 alternatives, which means it can't even be used to implement X.509 certificate decoding. Makes me wonder what they're going to do when they get that far.

woodruffw 2y ago
I don't know why you think this; the asn1-rust crate is in fact used to implement not only X.509 certificate decoding[1] but also path validation[2]. Edit: Here's an example of a CHOICE implemented with rust-asn1 that has more than three variants[3]. [1]: https://cryptography.io/en/latest/x509/reference/ https://cryptography.io/en/latest/x509/reference/ [2]: https://cryptography.io/en/latest/x509/verification/ https://cryptography.io/en/latest/x509/verification/ [3]: https://github.com/pyca/cryptography/blob/be6c53dd03172dde6af2de775a9d8e65e6b60fb0/src/rust/cryptography-x509/src/name.rs#L52-L83 https://github.com/pyca/cryptography/blob/be6c53dd03172dde6a...
- Arnavion 2y ago
  The asn1 crate provides three builtin Choice enums itself - asn1::{Choice1, Choice2, Choice3} - which support 1, 2 and 3 choices respectively. I assume that is what lilyball is referring to. But as you correctly point out the custom derive supports mapping enums with more variants to CHOICE just fine, so the builtin enums are not a relevant limitation.

mootptr 2y ago

Parser differential exploits are a understated problem, especially with ASN.1, which I didn't expect to see anyone thinking about. Kudos on this initiative!

kccqzy 2y ago
I understand that it is a problem but I'm more used to seeing arguments that monocultures and single implementations are bad: WebSQL for example didn't become a standard because there was only a single implementation.
- cryptonector 2y ago
  Where is the monoculture here?
  - kccqzy 2y ago
    If there were only one implementation for ASN.1 people would decry that whatever that implementation does effectively becomes the standard, and people would be clamoring to write a second implementation.
    - cryptonector 2y ago
      Ok, but there are many implementations. And the ASN.1 specs are really clear and readable (once you have a mental model of them).
      - kccqzy 2y ago
        Yes, which is why the contrast with monoculture is interesting. Hence my original comment.
        cryptonector 2y ago
        Ah, I see. Thanks.

benatkin 2y ago

From the post > with the help of funding from Alpha-Omega From the site: > funded by Microsoft, Google, and Amazon Also it's a Linux Foundation project. Interesting. Python's a big community, and there's some disagreement here over whether this would be better done in pure python. I think it's good that there's a rust/cloud contingent in python land but hope pure python remains popular.

woodruffw 2y ago
For the record: I shopped this project to them; it didn’t originate as an idea from LF or any major company. The idea itself came from PyCA’s maintainers, who have wanted this for a while. (The native code desirability question is also a red herring here, since PyCA has and will always need to have native code for cryptographic operations. So having a first class ASN.1 API in PyCA was always going to involve native code, with the only variable being how.)

time4tea 2y ago

20+ years ago used ASN.1 for talking between micro services. (HTTP Services, as they were called then) Very performant. Had to buy OSS tools licence but other than that quite nice.

zzo38computer 2y ago

I do not use Python, but I wrote my own library in C for reading/writing DER. (I have made a variant, which adds a few new types such as: key/value list, BCD, TRON character code, etc. The program works even if you do not use these new types.) DER does have the advantages they mention in that article, and other advantages. Some people mention that DER is not compact or not efficient; but often what is used instead is formats that are even less compact or less efficient than DER, and/or that are significantly more complicated to handle.