3 ms·

Is running normalization algorithms against UCS-2 -- i.e., where it's possible to have invalid surrogate pairs -- well-defined?

jrochkind1 8y ago
I'm not certain, I'm not familiar with that situation. (I think "UCS-2" is basically "legacy" and not actually a unicode standard at this point? I'm not familiar with it). If the issue is invalid bytes for a given unicode encoding (I am familiar with those!), I'm not certain what the official unicode normalization algorithms would do to them. They might just leave invalid bytes alone. The discussion of normalization seems to be in terms of codepoints though; a bytestream with invalid bytes (that can't be turned into unicode codepoints)... I would guess most actually existing normalization implementations are gonna complain about that. One could certainly write an algorithm that would just leave those bytes alone, I'm not sure if it would be considered compliant with the standard or not, just can't say. Why does this come up?
- happytoexplain 8y ago
  UCS-2 refers to the UTF-16 encoding of the first plane, defined when there was only one plane's worth of assignable space. UTF-16 extends UCS-2, and defines the reserved surrogates, thereby introducing the possibility of invalid surrogate usage. The parent post is referring to that (so it would be more accurate for the parent post to have said "UTF-16").
  - jrochkind1 8y ago
    OK, thanks. You can certainly have invalid bytes for UTF-8 too. I think in any unicode encoding, not every possible byte sequence is a valid representation of unicode codepoints. It doesn't require UCS-2/UTF-16. Here's an invalid byte sequence in UTF-8: "\xc3\x28"
    - happytoexplain 8y ago
      True, though the GP's example was somewhat unique in that UTF-16 created the possibility of invalid combinations at the code point level, rather than only the byte level.
      - deleted 8y ago
        [deleted]
- throw0101a 8y ago
  > I'm not certain, I'm not familiar with that situation. See RFC 6531.
- ucarion 8y ago
  This comes up because many implementations of "UTF-16" are in fact naive, and just operate within the mentality of UCS-2, where everything is just a 16-bit number. This is relevant within this discussion because we were talking about filesystems, and Windows stores file names as an arbitrary array of 16-bit numbers, but UTF-16 has additional requirements that Windows file names do not satisfy. When I say UCS-2, I just mean pretending that UTF-16 is short for `[]uint16`, and not worrying about surrogate pairs.
  - jrochkind1 8y ago
    I think we can all agree that you should never be applying any normalization to bytes representing a file system path, unless you are doing it according to specifications for that particular file system+OS combo. file system paths are tricky with non-ascii, that is for sure! Especially if you are trying to be agnostic to OS/filesystem. They are an edge case.
happytoexplain 8y ago
Normalization has nothing to say about what happens if you have a string containing invalid surrogates - you have an invalid string at that point and normalization won't get you into or out of that situation - it only affects how valid characters are composed of code points.

jrochkind1 8y ago

I'm not certain, I'm not familiar with that situation. (I think "UCS-2" is basically "legacy" and not actually a unicode standard at this point? I'm not familiar with it). If the issue is invalid bytes for a given unicode encoding (I am familiar with those!), I'm not certain what the official unicode normalization algorithms would do to them. They might just leave invalid bytes alone. The discussion of normalization seems to be in terms of codepoints though; a bytestream with invalid bytes (that can't be turned into unicode codepoints)... I would guess most actually existing normalization implementations are gonna complain about that. One could certainly write an algorithm that would just leave those bytes alone, I'm not sure if it would be considered compliant with the standard or not, just can't say. Why does this come up?

happytoexplain 8y ago
UCS-2 refers to the UTF-16 encoding of the first plane, defined when there was only one plane's worth of assignable space. UTF-16 extends UCS-2, and defines the reserved surrogates, thereby introducing the possibility of invalid surrogate usage. The parent post is referring to that (so it would be more accurate for the parent post to have said "UTF-16").
- jrochkind1 8y ago
  OK, thanks. You can certainly have invalid bytes for UTF-8 too. I think in any unicode encoding, not every possible byte sequence is a valid representation of unicode codepoints. It doesn't require UCS-2/UTF-16. Here's an invalid byte sequence in UTF-8: "\xc3\x28"
  - happytoexplain 8y ago
    True, though the GP's example was somewhat unique in that UTF-16 created the possibility of invalid combinations at the code point level, rather than only the byte level.
    - deleted 8y ago
      [deleted]
throw0101a 8y ago
> I'm not certain, I'm not familiar with that situation. See RFC 6531.
ucarion 8y ago
This comes up because many implementations of "UTF-16" are in fact naive, and just operate within the mentality of UCS-2, where everything is just a 16-bit number. This is relevant within this discussion because we were talking about filesystems, and Windows stores file names as an arbitrary array of 16-bit numbers, but UTF-16 has additional requirements that Windows file names do not satisfy. When I say UCS-2, I just mean pretending that UTF-16 is short for `[]uint16`, and not worrying about surrogate pairs.
- jrochkind1 8y ago
  I think we can all agree that you should never be applying any normalization to bytes representing a file system path, unless you are doing it according to specifications for that particular file system+OS combo. file system paths are tricky with non-ascii, that is for sure! Especially if you are trying to be agnostic to OS/filesystem. They are an edge case.

happytoexplain 8y ago

Normalization has nothing to say about what happens if you have a string containing invalid surrogates - you have an invalid string at that point and normalization won't get you into or out of that situation - it only affects how valid characters are composed of code points.