3 ms·

> This proposal also recommends adoption of Unicode normalization form C (NFC) for identifiers to ensure that when compared, identifiers intended to be the same

> This proposal also recommends adoption of Unicode normalization form C (NFC) for identifiers to ensure that when compared, identifiers intended to be the same will compare as equal. Legacy encodings are generally naturally in NFC when converted to Unicode. Most tools will, by default, produce NFC tex. Er, a much better approach is to allow unnormalized Unicode in source code and use form-insensitive matching of symbol names so that all forms of a symbol are equivalent. This can be done by normalizing during the parse, or by implementing form-insensitive string comparison and hashing functions that normalize glyph by glyph as needed -- the latter can be very fast for all-ASCII and mostly-ASCII symbols! The reason this is a better way is that there's too many places that don't produce NFC. For example, HFS+ uses NFD, so if you cut-n-paste a file name from HFS+ into other contexts, you'll be pasting NFD unless the cut-n-paste system normalizes to NFC. Also, while it's true that input modes typically produce NFC, it's more that they produce NFC for a small subset of Unicode, not that they will normalize other forms seen on input. Using form-insensitive string comparison/hashing/matching yields a better user experience at not that much implementation cost: you're gonna need a Unicode library, and that library will need to have normalization support, so you can implement form-insensitivity.

wahern 3y ago
> Er, a much better approach is to allow unnormalized Unicode in source code and use form-insensitive matching of symbol names so that all forms of a symbol are equivalent. Linkers will often be blissfully unaware of Unicode or any form of localization. This was the impetus for UTF-8, so that the bulk of software which is 8-bit clean or which operates on opaque, NUL-terminated strings can continue working as-is. This can't be changed without breaking backwards ABI compatibility; therefore, it's very unlikely to change. There are countless half-measures that could be taken, but few if any are suitable for standardization. If the history of software localization is any guide, in the face of strict, forward-looking specifications various vendors and ecosystems will likely go there own way, with the one sure thing being a failure to fully adopt or properly implement the specification.
- cryptonector 3y ago
  Yes, the compiler should normalize symbols before writing object files, no doubt. I'm talking about the inputs though -the source files- which should not have to be normalized.
  - froh 3y ago
    that means the compiler needs to have the Unicode database. I'd rather have a seperate 'recode' to NFC preprocessing in the build process, if the developers editor doesn't do NFC normalization. if I'm not mistaken checking for NFC is simpler than transforming to NFC, right?
    - cryptonector 3y ago
      The compiler is going to need a Unicode library anyways. It's true that checking that some string is in some canonical form (say, NFC) is easier than normalizing it. In fact, it's even easier to check that some string is in NFD than it is to check that it's in NFC because NFC is defined in terms of NFD. That's not enough to justify not doing form-insensitive string matching in the compiler and forcing a pre-compilation normalization step. Also, you'd only want symbols normalized, not string literals (since one might need a string literal that is not in a canonical form, or not in NFC). Thus the pre-compilation normalization step would have to be language-aware, so it might as well be part of the compiler and not a separate step.
      - froh 3y ago
        > you'd only want symbols normalized, not string literals _this_ is the argument that convinces me. thank you:-)
        cryptonector 3y ago
        You're welcome :) But I'd like to convince you and everyone that in general we want to accept inputs in any form and be form-insensitive because that's much more user-friendly. We don't in fact have universal NFC-only input modes. We have accidentally NFC-mostly input modes -- accidental because typically we have transcoding from legacy codesets, which yields NFC, but no normalization is actually done so that copy-paste operations can cause non-canonical input to be provided to applications. We don't have universal agreement on NFC. This makes for a mess with occasional user-visible problems. Form-insensitivity is essentially the same as normalizing character-by-character when hashing or comparing strings, but this can be very fast for mostly-ASCII text, so form-insensitivity can be very fast. For string comparison form-insensitivity is faster than first normalizing because either the strings are equal in content and probably equal in form, or they differ early and possibly at some character where normalization is not necessary to determine the result of the comparison, and thus less work need be done to compare strings form-insensitively than first normalizing all inputs (unless so many string comparisons will be done that maybe normalizing first might be a win). For string hashing form-insensitivity is not faster because one has to normalize every input, whereas if all inputs are normalized once then one need never normalize to hash, but still form-insensitivity yields a better user experience.

wahern 3y ago

> Er, a much better approach is to allow unnormalized Unicode in source code and use form-insensitive matching of symbol names so that all forms of a symbol are equivalent. Linkers will often be blissfully unaware of Unicode or any form of localization. This was the impetus for UTF-8, so that the bulk of software which is 8-bit clean or which operates on opaque, NUL-terminated strings can continue working as-is. This can't be changed without breaking backwards ABI compatibility; therefore, it's very unlikely to change. There are countless half-measures that could be taken, but few if any are suitable for standardization. If the history of software localization is any guide, in the face of strict, forward-looking specifications various vendors and ecosystems will likely go there own way, with the one sure thing being a failure to fully adopt or properly implement the specification.

cryptonector 3y ago
Yes, the compiler should normalize symbols before writing object files, no doubt. I'm talking about the inputs though -the source files- which should not have to be normalized.
- froh 3y ago
  that means the compiler needs to have the Unicode database. I'd rather have a seperate 'recode' to NFC preprocessing in the build process, if the developers editor doesn't do NFC normalization. if I'm not mistaken checking for NFC is simpler than transforming to NFC, right?
  - cryptonector 3y ago
    The compiler is going to need a Unicode library anyways. It's true that checking that some string is in some canonical form (say, NFC) is easier than normalizing it. In fact, it's even easier to check that some string is in NFD than it is to check that it's in NFC because NFC is defined in terms of NFD. That's not enough to justify not doing form-insensitive string matching in the compiler and forcing a pre-compilation normalization step. Also, you'd only want symbols normalized, not string literals (since one might need a string literal that is not in a canonical form, or not in NFC). Thus the pre-compilation normalization step would have to be language-aware, so it might as well be part of the compiler and not a separate step.
    - froh 3y ago
      > you'd only want symbols normalized, not string literals _this_ is the argument that convinces me. thank you:-)
      - cryptonector 3y ago
        You're welcome :) But I'd like to convince you and everyone that in general we want to accept inputs in any form and be form-insensitive because that's much more user-friendly. We don't in fact have universal NFC-only input modes. We have accidentally NFC-mostly input modes -- accidental because typically we have transcoding from legacy codesets, which yields NFC, but no normalization is actually done so that copy-paste operations can cause non-canonical input to be provided to applications. We don't have universal agreement on NFC. This makes for a mess with occasional user-visible problems. Form-insensitivity is essentially the same as normalizing character-by-character when hashing or comparing strings, but this can be very fast for mostly-ASCII text, so form-insensitivity can be very fast. For string comparison form-insensitivity is faster than first normalizing because either the strings are equal in content and probably equal in form, or they differ early and possibly at some character where normalization is not necessary to determine the result of the comparison, and thus less work need be done to compare strings form-insensitively than first normalizing all inputs (unless so many string comparisons will be done that maybe normalizing first might be a win). For string hashing form-insensitivity is not faster because one has to normalize every input, whereas if all inputs are normalized once then one need never normalize to hash, but still form-insensitivity yields a better user experience.