4 ms·

that means the compiler needs to have the Unicode database. I'd rather have a seperate 'recode' to NFC preprocessing in the build process, if the developers ed

by froh 3y ago

that means the compiler needs to have the Unicode database. I'd rather have a seperate 'recode' to NFC preprocessing in the build process, if the developers editor doesn't do NFC normalization. if I'm not mistaken checking for NFC is simpler than transforming to NFC, right?

cryptonector 3y ago
The compiler is going to need a Unicode library anyways. It's true that checking that some string is in some canonical form (say, NFC) is easier than normalizing it. In fact, it's even easier to check that some string is in NFD than it is to check that it's in NFC because NFC is defined in terms of NFD. That's not enough to justify not doing form-insensitive string matching in the compiler and forcing a pre-compilation normalization step. Also, you'd only want symbols normalized, not string literals (since one might need a string literal that is not in a canonical form, or not in NFC). Thus the pre-compilation normalization step would have to be language-aware, so it might as well be part of the compiler and not a separate step.
- froh 3y ago
  > you'd only want symbols normalized, not string literals _this_ is the argument that convinces me. thank you:-)
  - cryptonector 3y ago
    You're welcome :) But I'd like to convince you and everyone that in general we want to accept inputs in any form and be form-insensitive because that's much more user-friendly. We don't in fact have universal NFC-only input modes. We have accidentally NFC-mostly input modes -- accidental because typically we have transcoding from legacy codesets, which yields NFC, but no normalization is actually done so that copy-paste operations can cause non-canonical input to be provided to applications. We don't have universal agreement on NFC. This makes for a mess with occasional user-visible problems. Form-insensitivity is essentially the same as normalizing character-by-character when hashing or comparing strings, but this can be very fast for mostly-ASCII text, so form-insensitivity can be very fast. For string comparison form-insensitivity is faster than first normalizing because either the strings are equal in content and probably equal in form, or they differ early and possibly at some character where normalization is not necessary to determine the result of the comparison, and thus less work need be done to compare strings form-insensitively than first normalizing all inputs (unless so many string comparisons will be done that maybe normalizing first might be a win). For string hashing form-insensitivity is not faster because one has to normalize every input, whereas if all inputs are normalized once then one need never normalize to hash, but still form-insensitivity yields a better user experience.
rurban 3y ago
Only a tiny part of the Unicode database, the normalization tables. Problem is these tables have to be updated every year, and they dont with similar tables, such as the glibc. And Unicode identifiers are entirely insecure, because they are not identifiable. My proposal was postponed to C26.
- cryptonector 3y ago
  > Problem is these tables have to be updated every year, and they dont with similar tables, such as the glibc. The language standard can avoid this by committing to a single Unicode version for each language version. > And Unicode identifiers are entirely insecure, ... Eh, it depends on what we're talking about using them for. If it's symbols in object files, it's not really a problem. More importantly confusables are unavoidable -- even ASCII all by itself has confusable characters (1 and l for example). The thing to do is to forbid arbitrary mixing of scripts, allowing only those scripts that are relevant in the relevant contexts. For DNS this means that DNS TLD registries should come up with per-registry rules, for example.
  - rurban 3y ago
    See my https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2932.htm https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2932.htm
    - cryptonector 3y ago
      Can you expand on what the security concerns are as to confusables in symbol names in C source code? Clearly there's a security concern with cut-n-paste, but that's true regardless of what the rules might be for C identifiers. It's not like it's obvious that UTR#39 applies literally everywhere that there are "identifiers". Also, can you speak to what is the security concern with form-insensitivity (rather than confusables) as to symbols in input source files? I just don't see a concern at all there, but maybe I'm missing something. Lastly, I think `#include` is the most important place to get this right since that does interface with the world outside the compiler (specifically: the filesystem), but as you note the filesystems mostly are just-use-8bit -- very few filesystems normalize on create (HFS+) or are form-insensitive (ZFS). The other place to get this right is on the object file output side, where symbols definitely must be normalized. Oh, one more thing: the platform might impose some rules regarding symbols in ELF and any other object file formats. Are they known to? I suppose C can't necessarily cater to all platform-imposed limitations on symbol naming, but it'd be useful to know about them.

by froh 3y ago

cryptonector 3y ago

The compiler is going to need a Unicode library anyways. It's true that checking that some string is in some canonical form (say, NFC) is easier than normalizing it. In fact, it's even easier to check that some string is in NFD than it is to check that it's in NFC because NFC is defined in terms of NFD. That's not enough to justify not doing form-insensitive string matching in the compiler and forcing a pre-compilation normalization step. Also, you'd only want symbols normalized, not string literals (since one might need a string literal that is not in a canonical form, or not in NFC). Thus the pre-compilation normalization step would have to be language-aware, so it might as well be part of the compiler and not a separate step.

froh 3y ago
> you'd only want symbols normalized, not string literals _this_ is the argument that convinces me. thank you:-)
- cryptonector 3y ago
  You're welcome :) But I'd like to convince you and everyone that in general we want to accept inputs in any form and be form-insensitive because that's much more user-friendly. We don't in fact have universal NFC-only input modes. We have accidentally NFC-mostly input modes -- accidental because typically we have transcoding from legacy codesets, which yields NFC, but no normalization is actually done so that copy-paste operations can cause non-canonical input to be provided to applications. We don't have universal agreement on NFC. This makes for a mess with occasional user-visible problems. Form-insensitivity is essentially the same as normalizing character-by-character when hashing or comparing strings, but this can be very fast for mostly-ASCII text, so form-insensitivity can be very fast. For string comparison form-insensitivity is faster than first normalizing because either the strings are equal in content and probably equal in form, or they differ early and possibly at some character where normalization is not necessary to determine the result of the comparison, and thus less work need be done to compare strings form-insensitively than first normalizing all inputs (unless so many string comparisons will be done that maybe normalizing first might be a win). For string hashing form-insensitivity is not faster because one has to normalize every input, whereas if all inputs are normalized once then one need never normalize to hash, but still form-insensitivity yields a better user experience.

rurban 3y ago

Only a tiny part of the Unicode database, the normalization tables. Problem is these tables have to be updated every year, and they dont with similar tables, such as the glibc. And Unicode identifiers are entirely insecure, because they are not identifiable. My proposal was postponed to C26.

cryptonector 3y ago
> Problem is these tables have to be updated every year, and they dont with similar tables, such as the glibc. The language standard can avoid this by committing to a single Unicode version for each language version. > And Unicode identifiers are entirely insecure, ... Eh, it depends on what we're talking about using them for. If it's symbols in object files, it's not really a problem. More importantly confusables are unavoidable -- even ASCII all by itself has confusable characters (1 and l for example). The thing to do is to forbid arbitrary mixing of scripts, allowing only those scripts that are relevant in the relevant contexts. For DNS this means that DNS TLD registries should come up with per-registry rules, for example.
- rurban 3y ago
  See my https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2932.htm https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2932.htm
  - cryptonector 3y ago
    Can you expand on what the security concerns are as to confusables in symbol names in C source code? Clearly there's a security concern with cut-n-paste, but that's true regardless of what the rules might be for C identifiers. It's not like it's obvious that UTR#39 applies literally everywhere that there are "identifiers". Also, can you speak to what is the security concern with form-insensitivity (rather than confusables) as to symbols in input source files? I just don't see a concern at all there, but maybe I'm missing something. Lastly, I think `#include` is the most important place to get this right since that does interface with the world outside the compiler (specifically: the filesystem), but as you note the filesystems mostly are just-use-8bit -- very few filesystems normalize on create (HFS+) or are form-insensitive (ZFS). The other place to get this right is on the object file output side, where symbols definitely must be normalized. Oh, one more thing: the platform might impose some rules regarding symbols in ELF and any other object file formats. Are they known to? I suppose C can't necessarily cater to all platform-imposed limitations on symbol naming, but it'd be useful to know about them.