5 ms·

Even then, it's not just filepaths. Internal representations can range from "bytes that are sometimes string-ish" to "bytes which are valid strings", while in

Even then, it's not just filepaths. Internal representations can range from "bytes that are sometimes string-ish" to "bytes which are valid strings", while in no way covering the domain where the string-like properties that normalization brings are desireable. I doubt there are hard-and-fast rules where normalization is appropriate, but if you start applying it to all input, you're going to break things. Email addresses immediately spring to mind, for example.

jrochkind1 8y ago
Can you even have unicode/non-ascii in email addresses? If you can, and your email address is josé@gmail.com, but whether it arrives to you depends on whether the address was entered as unicode codepoint 'LATIN SMALL LETTER E WITH ACUTE' or 'LATIN SMALL LETTER E'+'COMBINING ACUTE ACCENT'... you're going to have a lot of trouble getting email. Becauase people entering your address have no idea which they are entering. I know I can type an "é" on MacOS US keyboard layout by typing `option-e,e`, but I have _no idea_ what actual bytes (corresponding to which of these codepoints) are sent to HN when I enter that in an HN comment text box. And most people sending email, of course, don't even know any of this is a thing. If it works at all, it's actually gonna be because the software on either or both ends (sending and receiving) is already doing NFC normalization! So, yes, this illustrates that normalization _is a thing_ you have to pay attention to, and can get weird. But in general, for "text" input (rather than addresses or filepaths), normalizing to NFC is _usually_ a good call. If it's addresses or filepaths or what have you... it's gonna get confusing no matter what. If you know what you're doing, maybe NFC isn't right. If you don't know what you're doing, NFC is a lot better start-out-with call than not normalizing at all. Not normalizing at all will bite you more often than NFC will.
- ucarion 8y ago
  Is running normalization algorithms against UCS-2 -- i.e., where it's possible to have invalid surrogate pairs -- well-defined?
  - jrochkind1 8y ago
    I'm not certain, I'm not familiar with that situation. (I think "UCS-2" is basically "legacy" and not actually a unicode standard at this point? I'm not familiar with it). If the issue is invalid bytes for a given unicode encoding (I am familiar with those!), I'm not certain what the official unicode normalization algorithms would do to them. They might just leave invalid bytes alone. The discussion of normalization seems to be in terms of codepoints though; a bytestream with invalid bytes (that can't be turned into unicode codepoints)... I would guess most actually existing normalization implementations are gonna complain about that. One could certainly write an algorithm that would just leave those bytes alone, I'm not sure if it would be considered compliant with the standard or not, just can't say. Why does this come up?
    - happytoexplain 8y ago
      UCS-2 refers to the UTF-16 encoding of the first plane, defined when there was only one plane's worth of assignable space. UTF-16 extends UCS-2, and defines the reserved surrogates, thereby introducing the possibility of invalid surrogate usage. The parent post is referring to that (so it would be more accurate for the parent post to have said "UTF-16").
      - jrochkind1 8y ago
        OK, thanks. You can certainly have invalid bytes for UTF-8 too. I think in any unicode encoding, not every possible byte sequence is a valid representation of unicode codepoints. It doesn't require UCS-2/UTF-16. Here's an invalid byte sequence in UTF-8: "\xc3\x28"
        happytoexplain 8y ago
        True, though the GP's example was somewhat unique in that UTF-16 created the possibility of invalid combinations at the code point level, rather than only the byte level.
        deleted 8y ago
- throw0101a 8y ago
  > Can you even have unicode/non-ascii in email addresses? Yes, per RFC 6531: * https://en.wikipedia.org/wiki/Unicode_and_email https://en.wikipedia.org/wiki/Unicode_and_email * https://en.wikipedia.org/wiki/International_email https://en.wikipedia.org/wiki/International_email
  - jrochkind1 8y ago
    Strangely, RFC 6531 gives the impression of being written by people who didn't realize unicode normalization was a concern. I'm reading it for the first time, but it basically just says "an SMTP server can advertise it supports UTF8, and then it should accept UTF8 in all the places that were ascii before." I mean, a little bit more complicated than that, but with regard to _mailboxes_ (the part before the `@`), not much. It doens't seem to say anything about normalization, or the fact that a mailbox `josé@` could be using 'LATIN SMALL LETTER E WITH ACUTE' or 'LATIN SMALL LETTER E'+'COMBINING ACUTE ACCENT'. I suspect any SMTP servers supporting UTF8 are actualy doing NFC normalization to match mailboxes to requests.
    - marcosdumay 8y ago
      Servers are free to do any transformation on the account names they accept, and shouldn't apply any on the ones they relay. It's expected that they would do many kinds of transformations, like case normalization. Unicode normalization should be one of those but none of the email RFCs intend to enumerate them all.

jrochkind1 8y ago

Can you even have unicode/non-ascii in email addresses? If you can, and your email address is josé@gmail.com, but whether it arrives to you depends on whether the address was entered as unicode codepoint 'LATIN SMALL LETTER E WITH ACUTE' or 'LATIN SMALL LETTER E'+'COMBINING ACUTE ACCENT'... you're going to have a lot of trouble getting email. Becauase people entering your address have no idea which they are entering. I know I can type an "é" on MacOS US keyboard layout by typing `option-e,e`, but I have _no idea_ what actual bytes (corresponding to which of these codepoints) are sent to HN when I enter that in an HN comment text box. And most people sending email, of course, don't even know any of this is a thing. If it works at all, it's actually gonna be because the software on either or both ends (sending and receiving) is already doing NFC normalization! So, yes, this illustrates that normalization _is a thing_ you have to pay attention to, and can get weird. But in general, for "text" input (rather than addresses or filepaths), normalizing to NFC is _usually_ a good call. If it's addresses or filepaths or what have you... it's gonna get confusing no matter what. If you know what you're doing, maybe NFC isn't right. If you don't know what you're doing, NFC is a lot better start-out-with call than not normalizing at all. Not normalizing at all will bite you more often than NFC will.

ucarion 8y ago
Is running normalization algorithms against UCS-2 -- i.e., where it's possible to have invalid surrogate pairs -- well-defined?
- jrochkind1 8y ago
  I'm not certain, I'm not familiar with that situation. (I think "UCS-2" is basically "legacy" and not actually a unicode standard at this point? I'm not familiar with it). If the issue is invalid bytes for a given unicode encoding (I am familiar with those!), I'm not certain what the official unicode normalization algorithms would do to them. They might just leave invalid bytes alone. The discussion of normalization seems to be in terms of codepoints though; a bytestream with invalid bytes (that can't be turned into unicode codepoints)... I would guess most actually existing normalization implementations are gonna complain about that. One could certainly write an algorithm that would just leave those bytes alone, I'm not sure if it would be considered compliant with the standard or not, just can't say. Why does this come up?
  - happytoexplain 8y ago
    UCS-2 refers to the UTF-16 encoding of the first plane, defined when there was only one plane's worth of assignable space. UTF-16 extends UCS-2, and defines the reserved surrogates, thereby introducing the possibility of invalid surrogate usage. The parent post is referring to that (so it would be more accurate for the parent post to have said "UTF-16").
    - jrochkind1 8y ago
      OK, thanks. You can certainly have invalid bytes for UTF-8 too. I think in any unicode encoding, not every possible byte sequence is a valid representation of unicode codepoints. It doesn't require UCS-2/UTF-16. Here's an invalid byte sequence in UTF-8: "\xc3\x28"
      - happytoexplain 8y ago
        True, though the GP's example was somewhat unique in that UTF-16 created the possibility of invalid combinations at the code point level, rather than only the byte level.
        deleted 8y ago
throw0101a 8y ago
> Can you even have unicode/non-ascii in email addresses? Yes, per RFC 6531: * https://en.wikipedia.org/wiki/Unicode_and_email https://en.wikipedia.org/wiki/Unicode_and_email * https://en.wikipedia.org/wiki/International_email https://en.wikipedia.org/wiki/International_email
- jrochkind1 8y ago
  Strangely, RFC 6531 gives the impression of being written by people who didn't realize unicode normalization was a concern. I'm reading it for the first time, but it basically just says "an SMTP server can advertise it supports UTF8, and then it should accept UTF8 in all the places that were ascii before." I mean, a little bit more complicated than that, but with regard to _mailboxes_ (the part before the `@`), not much. It doens't seem to say anything about normalization, or the fact that a mailbox `josé@` could be using 'LATIN SMALL LETTER E WITH ACUTE' or 'LATIN SMALL LETTER E'+'COMBINING ACUTE ACCENT'. I suspect any SMTP servers supporting UTF8 are actualy doing NFC normalization to match mailboxes to requests.
  - marcosdumay 8y ago
    Servers are free to do any transformation on the account names they accept, and shouldn't apply any on the ones they relay. It's expected that they would do many kinds of transformations, like case normalization. Unicode normalization should be one of those but none of the email RFCs intend to enumerate them all.