3 ms·

To be fair, if you don't need to support anything other than Unicode, then this is not an advantage, and over time we're all going to need non-Unicode less and

To be fair, if you don't need to support anything other than Unicode, then this is not an advantage, and over time we're all going to need non-Unicode less and less. That said I'm a big fan of ASN.1 (see my comment history).

morshu9001 10mo ago
I'm still confused how these ISO 2022 strings even work, and the ASN1 docs discourage using the UniversalString and GraphicString types. All these different string types are intimidating if I just want unicode/ascii, and even if I were using an obscure encoding, I'd use generic bytes instead of wanting asn1 to care about it.
- cryptonector 10mo ago
  GeneralString relies on control characters to "load" character sets into the C0 and C1 registers. This is madness -- specifically it's pre-Unicode madness, but before Unicode it made sense.
  - morshu9001 10mo ago
    Oh gosh. Fair enough that this exists and something uses it, but I'd absolutely want to handle that on the ends only, not get asn1 involved in parsing it.
    - cryptonector 10mo ago
      Oh GeneralString is madness. It's pre-Unicode madness. It exists because Unicode didn't exist in 1984, but people still wanted to be able to exchange text in multiple scripts, which necessitated being able to "switch codesets" in the middle. It's... yeah, it's.. it's nuts. I've _not_ implemented GeneralString, and practically no one needs to even when specs say to. E.g., in Kerberos the strings are GeneralString, but all the implementations just-send-8 and do not attempt to interpret any codeset switching escapes.
    - zzo38computer 10mo ago
      ASN.1 does not necessarily need to get involved in parsing the values; for some applications doing so is unnecessary anyways (this is true for many fields of many types and not only this one, though). ASN.1 will need to be involved in parsing the framing; whether or not it is involved in parsing the values depends on whether the application requires it for that specific value (for example, it is commonly not necessary to parse OIDs (you can usually just treat them as opaque data which can be compared for equality (or looked up in a table), although sometimes it is useful to display them), although some implementations insist on doing so anyways).
      - cryptonector 10mo ago
        Correct, ASN.1 does not tell you how to implement all its semantics. You can totally have tooling that exposes a GeneralString's blob payload to the application and lets the app handle the codeset switching aspects of GeneralString. I want to object that anyone who really has to handle multi-codeset GeneralString values would want a better library but... ...what would that be if not a converter to/from Unicode? But what if one really wants an array/list of {codeset, string} pairs? At that point open-coding support for those escapes is probably just as well since that one might have the only application in the world that wants that!! :laugh:
        zzo38computer 10mo ago
        > You can totally have tooling that exposes a GeneralString's blob payload to the application and lets the app handle the codeset switching aspects of GeneralString. You can do that for all types, and my own implementation does just expose the payloads for all types, although it also has functions to encode and decode many of them, they are functions that must be called separately (e.g. asn1_decode_number and asn1_decode_date). For some applications, there is no need to decode the payload anyways, and you can just treat it as opaque data if you do not need to display them (the same is true for many other types). > I want to object that anyone who really has to handle multi-codeset GeneralString values would want a better library I did start to try to write such a thing (it is not published yet), and the library for ISO 2022 is separate than the library for ASN.1, although they can be used together. It is intended to be usable for multiple uses. (I might also add support for character codes other than ISO 2022 (such as the encodings of Unicode and TRON code), although it is mainly intended to support ISO 2022.) > ...what would that be if not a converter to/from Unicode? For one thing, not all of the codes can be correctly converted to/from Unicode (especially control characters), and even if they can be, this is does not preserve some of the details, because the way the character set works is different from Unicode (e.g. some things might be considered to match in Unicode but not other character sets and vice-versa; this would also be true of case-folding, missing details, character properties, etc). For example, some information may be lost when converting to Unicode. (This does not necessarily mean that converting to/from Unicode is never useful, although you should consider if you can do it in a better way; for example, if your program accepts input that is meant to be added to a General string (or Graphic string) in a DER file then it would be better to accept ISO 2022 input directly if possible. Converting to TRON code (especially for CJK text) might be better, but even that depends on what you are using it for; some uses do not require conversion at all.) If you really want to store Unicode text anyways, you should consider if UTF-8 is a valid type according to the schema you are using (and use that type instead if so; note that some schemas might not care so much about the type in some cases); if not, consider prepending the three bytes <1B 25 47> when encoding it as a ISO 2022 string. (As far as I can tell, this is not really supposed to be allowed in ASN.1, but I suppose it is possible if you really need to. You might also check if it is only ASCII and avoid adding this prefix if so.) > But what if one really wants an array/list of {codeset, string} pairs? At that point open-coding support for those escapes is probably just as well since that one might have the only application in the world that wants that!! I think that what you will want will depend on the specific application. Some will want that, some will want something else, and also different ways that you might want handling control characters, etc.
- zzo38computer 10mo ago
  > I'm still confused how these ISO 2022 strings even work There is C0, G0, C1, and G1 sets (C0 and C1 are control characters and G0 and G1 are graphic characters), and escape sequences are used to select the C or G set for bytes with or without the high bit set. Graphic string does not allow control characters and General string does allow control characters. You probably do not need all control characters; your schema should probably restrict which control characters are allowed in each context (although the ASN.1 schema format does not seem to have any way to do this). This way, you will only handle the control characters which are appropriate for your use. This is messy, although canonical form simplifies it by adding some restrictions (this is one of the reasons why DER is better than BER, in my opinion). TRON code is better and is much simpler than the working of ISO 2022. (Unicode has a different kind of mess; although decoding is simpler, actually handling the decoded characters in text is its own big mess for many reasons. Unicode is a stateful character set, even though the encoding is stateless; TRON code is the other way around (and with a significantly simpler stateful encoding than ISO 2022).) > the ASN1 docs discourage using the UniversalString and GraphicString types UniversalString is UTF-32BE and GraphicString is ISO 2022 without control characters. By knowing what they are, you should know in which circumstances they should be considered useful or not useful; I think that they should not be discouraged in general (although usually if you want Unicode, you would use UTF-8 rather than UTF-32, there are some circumstances where you might want to use UTF-32, such as if the data or program is already UTF-32 for other reasons). (The data type which probably should be avoided is the UTC time type, which is not Y2K compliant.) > All these different string types are intimidating if I just want unicode/ascii If you only want ASCII, use the IA5 type (or Visible if you do not want control characters); if you only want Unicode, use the UTF-8 string type (or Universal if you want UTF-32 instead for some reason). ("IA5" is another name for ASCII that as far as I can tell hardly anyone other than ITU uses.) However, Unicode is not a very good character set, and they should not force or expect you to use it. As I had mentioned before, you do not need to use or implement all of the ASN.1 data types; only use the ones appropriate for your application (so, if you do not like most of the types, then don't use those types). I also made up some additional nonstandard ASN.1 types (called ASN.1X), which also might be useful for some applications; you are not required to use or implement these either.
  - 10mo ago

morshu9001 10mo ago

I'm still confused how these ISO 2022 strings even work, and the ASN1 docs discourage using the UniversalString and GraphicString types. All these different string types are intimidating if I just want unicode/ascii, and even if I were using an obscure encoding, I'd use generic bytes instead of wanting asn1 to care about it.

cryptonector 10mo ago
GeneralString relies on control characters to "load" character sets into the C0 and C1 registers. This is madness -- specifically it's pre-Unicode madness, but before Unicode it made sense.
- morshu9001 10mo ago
  Oh gosh. Fair enough that this exists and something uses it, but I'd absolutely want to handle that on the ends only, not get asn1 involved in parsing it.
  - cryptonector 10mo ago
    Oh GeneralString is madness. It's pre-Unicode madness. It exists because Unicode didn't exist in 1984, but people still wanted to be able to exchange text in multiple scripts, which necessitated being able to "switch codesets" in the middle. It's... yeah, it's.. it's nuts. I've _not_ implemented GeneralString, and practically no one needs to even when specs say to. E.g., in Kerberos the strings are GeneralString, but all the implementations just-send-8 and do not attempt to interpret any codeset switching escapes.
  - zzo38computer 10mo ago
    ASN.1 does not necessarily need to get involved in parsing the values; for some applications doing so is unnecessary anyways (this is true for many fields of many types and not only this one, though). ASN.1 will need to be involved in parsing the framing; whether or not it is involved in parsing the values depends on whether the application requires it for that specific value (for example, it is commonly not necessary to parse OIDs (you can usually just treat them as opaque data which can be compared for equality (or looked up in a table), although sometimes it is useful to display them), although some implementations insist on doing so anyways).
    - cryptonector 10mo ago
      Correct, ASN.1 does not tell you how to implement all its semantics. You can totally have tooling that exposes a GeneralString's blob payload to the application and lets the app handle the codeset switching aspects of GeneralString. I want to object that anyone who really has to handle multi-codeset GeneralString values would want a better library but... ...what would that be if not a converter to/from Unicode? But what if one really wants an array/list of {codeset, string} pairs? At that point open-coding support for those escapes is probably just as well since that one might have the only application in the world that wants that!! :laugh:
      - zzo38computer 10mo ago
        > You can totally have tooling that exposes a GeneralString's blob payload to the application and lets the app handle the codeset switching aspects of GeneralString. You can do that for all types, and my own implementation does just expose the payloads for all types, although it also has functions to encode and decode many of them, they are functions that must be called separately (e.g. asn1_decode_number and asn1_decode_date). For some applications, there is no need to decode the payload anyways, and you can just treat it as opaque data if you do not need to display them (the same is true for many other types). > I want to object that anyone who really has to handle multi-codeset GeneralString values would want a better library I did start to try to write such a thing (it is not published yet), and the library for ISO 2022 is separate than the library for ASN.1, although they can be used together. It is intended to be usable for multiple uses. (I might also add support for character codes other than ISO 2022 (such as the encodings of Unicode and TRON code), although it is mainly intended to support ISO 2022.) > ...what would that be if not a converter to/from Unicode? For one thing, not all of the codes can be correctly converted to/from Unicode (especially control characters), and even if they can be, this is does not preserve some of the details, because the way the character set works is different from Unicode (e.g. some things might be considered to match in Unicode but not other character sets and vice-versa; this would also be true of case-folding, missing details, character properties, etc). For example, some information may be lost when converting to Unicode. (This does not necessarily mean that converting to/from Unicode is never useful, although you should consider if you can do it in a better way; for example, if your program accepts input that is meant to be added to a General string (or Graphic string) in a DER file then it would be better to accept ISO 2022 input directly if possible. Converting to TRON code (especially for CJK text) might be better, but even that depends on what you are using it for; some uses do not require conversion at all.) If you really want to store Unicode text anyways, you should consider if UTF-8 is a valid type according to the schema you are using (and use that type instead if so; note that some schemas might not care so much about the type in some cases); if not, consider prepending the three bytes <1B 25 47> when encoding it as a ISO 2022 string. (As far as I can tell, this is not really supposed to be allowed in ASN.1, but I suppose it is possible if you really need to. You might also check if it is only ASCII and avoid adding this prefix if so.) > But what if one really wants an array/list of {codeset, string} pairs? At that point open-coding support for those escapes is probably just as well since that one might have the only application in the world that wants that!! I think that what you will want will depend on the specific application. Some will want that, some will want something else, and also different ways that you might want handling control characters, etc.
zzo38computer 10mo ago
> I'm still confused how these ISO 2022 strings even work There is C0, G0, C1, and G1 sets (C0 and C1 are control characters and G0 and G1 are graphic characters), and escape sequences are used to select the C or G set for bytes with or without the high bit set. Graphic string does not allow control characters and General string does allow control characters. You probably do not need all control characters; your schema should probably restrict which control characters are allowed in each context (although the ASN.1 schema format does not seem to have any way to do this). This way, you will only handle the control characters which are appropriate for your use. This is messy, although canonical form simplifies it by adding some restrictions (this is one of the reasons why DER is better than BER, in my opinion). TRON code is better and is much simpler than the working of ISO 2022. (Unicode has a different kind of mess; although decoding is simpler, actually handling the decoded characters in text is its own big mess for many reasons. Unicode is a stateful character set, even though the encoding is stateless; TRON code is the other way around (and with a significantly simpler stateful encoding than ISO 2022).) > the ASN1 docs discourage using the UniversalString and GraphicString types UniversalString is UTF-32BE and GraphicString is ISO 2022 without control characters. By knowing what they are, you should know in which circumstances they should be considered useful or not useful; I think that they should not be discouraged in general (although usually if you want Unicode, you would use UTF-8 rather than UTF-32, there are some circumstances where you might want to use UTF-32, such as if the data or program is already UTF-32 for other reasons). (The data type which probably should be avoided is the UTC time type, which is not Y2K compliant.) > All these different string types are intimidating if I just want unicode/ascii If you only want ASCII, use the IA5 type (or Visible if you do not want control characters); if you only want Unicode, use the UTF-8 string type (or Universal if you want UTF-32 instead for some reason). ("IA5" is another name for ASCII that as far as I can tell hardly anyone other than ITU uses.) However, Unicode is not a very good character set, and they should not force or expect you to use it. As I had mentioned before, you do not need to use or implement all of the ASN.1 data types; only use the ones appropriate for your application (so, if you do not like most of the types, then don't use those types). I also made up some additional nonstandard ASN.1 types (called ASN.1X), which also might be useful for some applications; you are not required to use or implement these either.
- 10mo ago