4 ms·
> Here's the problem: it is possible to have unmatched surrogates in a file name (or in other places that Windows accepts UTF-16). NTFS (and Windows as a whole
by chungy 9y ago
> Here's the problem: it is possible to have unmatched surrogates in a file name (or in other places that Windows accepts UTF-16).
NTFS (and Windows as a whole) does not use UTF-16, it uses UCS-2. It is a subtle difference, but surrogate pairs didn't exist in UCS-2.
- quietbritishjim 9y agoIndeed. I should have written something more like "in places Windows appears to accept UTF-16". But there is certainly a little truth to it: if you pass a surrogate pair to e.g. the text of a Windows label using an appropriate Unicode font, I believe it will show its UTF-16 interpretation.
- burfog 9y agoOld versions use UCS-2. New versions use UTF-16. (correctness not enforced) This is also how Java and OS X were updated. The kernel generally uses the 16-bit equivalent of the old Pascal string, that being a 16-bit count of 16-bit pieces of UTF-16 data. This allows a 16-bit NUL to get into various places that make the Win32 API choke.
- masklinn 9y ago> Old versions use UCS-2. New versions use UTF-16. (correctness not enforced) This is also how Java and OS X were updated. I usually call that ucs2-plus-surrogates, to make it clear that you may encounter unpaired surrogates, and thus invalid paths if you assume proper UTF-16.
- leeter 9y agoThis is actually kinda painful with NTFS as NTFS doesn't really care what's in a path other than the directory separators it's all binary. This means that different applications using different Unicode normalization will result in odd things happening. To me the right answer is NTFS should normalize all paths the same way internally but they have yet to implement it because it would break legacy systems that have un-normalized paths (I assume).
- freeone3000 9y agoPrecisely. Programs that internally use ShiftJIS, for instance, would stop functioning on UTF-16 enforced compatability or normalization. They're currently "broken" (as in, operating incorrectly) but in a way that works.
- leeter 9y agoCan you explain why this would be the case? In theory this shouldn't be an issue because any ShiftJIS conversion to Unicode should be reversible.
- TazeTSchnitzel 9y agoI think it's not a perfect round-trip due to differences in how the two standards encode certain characters. You will get a correct conversion either way, but the result of a round-trip might not be bit-identical.
- rspeer 9y agoIt should be reversible, sure. Just let me know when you find a standardized, correct, one-to-one mapping between JIS codepoints and Unicode codepoints. Also make sure it hasn't changed in the lifetime of the oldest software that anyone is using.
- freeone3000 9y agoThe apps I'm thinking of write bytes, not strings. If you enforced UTF-16 compatibility, they'd have to say what encoding they're using (they don't) or convert it themselves (they don't) - and changing either of these would require at least an application recompile. The reason they currently work is because bytes out == bytes in, so they can read the files they create, despite what mojibake the user sees.