8 ms·

No need to go to the kernel API to create filenames that are invalid UTF-16. The Win32 API will happily let you do it.

magicalhippo 2y ago
I was AFK so couldn't check, but yeah you're right. Just made two files named ä.txt which happily sat next to each other, one being NFC and other NFD. So yeah, don't mess with the normalization of filenames.
- panzi 2y ago
  They're both valid UTF-16, though. Can you create a filename with only half of a surrogate pair in it? I don't use Windows, so I can't check. Linux literally allows any arbitrary byte except for 0x00 and 0x2F ('/' in ASCII/UTF-8). It's a problem for programming languages that want to only use valid Unicode strings, like Python. Rust has a separate type "OsString" to handle that, with either lossy conversion to "String" or a conversion method that can fail. Python uses the custom use Unicode range to represent invalid byte sequences in filenames. It's all a mess. JavaScript doesn't give a damn about the validity of their UTF-16 strings. (Note that Rust's OsString is different from it's CString type. Well, I guess under Unix they're the same, but under Windows OsString is UTF-16 (or "WTF-16", because it isn't actually valid UTF-16 in all cases).)
  - magicalhippo 2y ago
    Yeah seems to be same with Win32. I tried using U+13161 EGYPTIAN HIEROGLYPH G029[1], which resulted in a string of length 2 as expected. Using both chars (code units) and just the first char (code unit) worked equally fine. In Windows Explorer the first one shows the stork as expected, while the second shows that "invalid character" rectangle. So yeah, treating filenames as nearly-opaque byte sequences is probably the best approach. [1]: https://en.wiktionary.org/wiki/%F0%93%85%A1 https://en.wiktionary.org/wiki/%F0%93%85%A1
- CRConrad 2y ago
  But ä.txt is a perfectly normal filename! WTF is there to "normalize" about that?!? Äh, nu går vi och gör något annat.
  - magicalhippo 2y ago
    Either ä as a single code unit or a + ¨ as two code units. Win32/NTFS treats them as two different filenames, as it doesn't normalize them before storing/comparing.

I was AFK so couldn't check, but yeah you're right. Just made two files named ä.txt which happily sat next to each other, one being NFC and other NFD. So yeah, don't mess with the normalization of filenames.

panzi 2y ago
They're both valid UTF-16, though. Can you create a filename with only half of a surrogate pair in it? I don't use Windows, so I can't check. Linux literally allows any arbitrary byte except for 0x00 and 0x2F ('/' in ASCII/UTF-8). It's a problem for programming languages that want to only use valid Unicode strings, like Python. Rust has a separate type "OsString" to handle that, with either lossy conversion to "String" or a conversion method that can fail. Python uses the custom use Unicode range to represent invalid byte sequences in filenames. It's all a mess. JavaScript doesn't give a damn about the validity of their UTF-16 strings. (Note that Rust's OsString is different from it's CString type. Well, I guess under Unix they're the same, but under Windows OsString is UTF-16 (or "WTF-16", because it isn't actually valid UTF-16 in all cases).)
- magicalhippo 2y ago
  Yeah seems to be same with Win32. I tried using U+13161 EGYPTIAN HIEROGLYPH G029[1], which resulted in a string of length 2 as expected. Using both chars (code units) and just the first char (code unit) worked equally fine. In Windows Explorer the first one shows the stork as expected, while the second shows that "invalid character" rectangle. So yeah, treating filenames as nearly-opaque byte sequences is probably the best approach. [1]: https://en.wiktionary.org/wiki/%F0%93%85%A1 https://en.wiktionary.org/wiki/%F0%93%85%A1
CRConrad 2y ago
But ä.txt is a perfectly normal filename! WTF is there to "normalize" about that?!? Äh, nu går vi och gör något annat.
- magicalhippo 2y ago
  Either ä as a single code unit or a + ¨ as two code units. Win32/NTFS treats them as two different filenames, as it doesn't normalize them before storing/comparing.