4 ms·
It gets worse. Windows doesn't use UTF-16 for file paths it uses UCS-2. Example: Windows paths can contain unpaired UTF-16 surrogates which is illegal in UTF-16
by hgs3 3y ago
It gets worse. Windows doesn't use UTF-16 for file paths it uses UCS-2. Example: Windows paths can contain unpaired UTF-16 surrogates which is illegal in UTF-16. Unix paths are no better; they are "bags of bytes" and not necessarily UTF-8. File paths are not guaranteed to be Unicode normalized. Even if they are canonically equivalent (equal graphemes) it doesn't mean they are binary equivalent (equal code points). You need to treat file paths as their own special thing and not strings.
- qsdf38100 3y agoWrong. Windows uses UTF-16. It used to use UCS-2 a long time ago.
- pornel 3y agoUnpaired surrogates are not allowed in UTF-16, so whatever Windows uses is not quite it.
- jeroenhd 3y agoWindows doesn't bother checking every string after every modification, but neither does Linux with UTF-8. You can pass invalid UTF-8 to tons of APIs and almost all of them will just work, just like you can with UTF-16 in Windows. Doing a string validation check in every single API call would waste cycles for no good reason.
- Karellen 3y agoLinux (the kernel) doesn't claim (AFAIK) to use UTF-8. It takes NUL-terminated strings in any charset you choose (or none) and either compares them with other NUL-terminated strings, or spits them back out again later. Interpreting a string as encoding text in particular character set is, as far as the kernel is concerned, a problem for userspace. Windows does make claims of "supporting" UTF-16.
- jeroenhd 3y agoLinux supports UTF-8 the same way Windows does: by trusting you that the encoding is correct, unless you access APIs that are encoding dependent somehow. The Windows API is a lot more complete than the Linux API, so comparing them is rather pointless. Windows supports UTF-16, it doesn't guarantee UTF-16 correctness. The native methods annotated with the W suffix all take UTF-16, so unless you want to render your own fonts, you're going to need to provide it with either that or ASCII.
- chrismorgan 3y agoWindows uses potentially-ill-formed UTF-16. No language or environment that uses UTF-16 validates it, ever. Not a single one that I know of. But that doesn’t make it UCS-2, it just makes it potentially-ill-formed UTF-16.
- GoblinSlayer 3y agoWindows validates UTF-16 when ansi functions are called, they convert string arguments to UTF-16 and call unicode function, then convert results from UTF-16 to ansi.
- chrismorgan 3y agoThat’s nothing to do with validating UTF-16. What you’re describing is converting from code pages to UTF-16, which incidentally should produce well-formed UTF-16.
- hnfong 3y agoI just have the impression that they s/UCS-2/UTF-16/g without changing anything. Was that not the case? Genuinely curious whether there were UTF-16-specific changes made before they declared that Windows uses/supports UTF-16.
- Guvante 3y agoAnd Linux paths can contain anything that isn't a nul character or a /. Your points about normalization only matter in so far as Windows is generally setup to be case insensitive (which can be disabled). All UTF strings face normalization issues for a whole host of reasons.
- jeroenhd 3y ago> (which can be disabled) Wouldn't that break spectacularly? The whole "the fs is case insensitive" assumption has kind of been a given on Windows for about 40 years now. I remember accidentally creating an NTFS flash drive with two names with difference cases in Linux. Windows programs sure didn't like that.
- Guvante 3y agoIt was added in 2018 as a folder attribute. There isn't a global option but you can annotate something as case sensitive.