3 ms·

Ideally… yes. Consider that Windows APIs use UTF-16, not UTF-8, for wide characters. Microsoft's extensions to ISO 9660 used (big endian!) UCS-2 (a.k.a. UTF-1

Ideally… yes. Consider that Windows APIs use UTF-16, not UTF-8, for wide characters. Microsoft's extensions to ISO 9660 used (big endian!) UCS-2 (a.k.a. UTF-16) for long filenames, NTFS uses proper UTF-16 I believe.

arthur2e5 3y ago
You are right: NTFS exclusively runs on UCS-2, now "UTF-16 without validation". Back in the 1980s "Unicode" referred to a way to represent the ISO standard "Universal Character Set", i.e. UCS-2, instead of being synonymous to the entire character set like today. The wisdom of i18n back then was to use "Unicode" text encoding because it covered the known universe of writing. Qt made a similar choice.
wolf550e 3y ago
I think NTFS is UCS-2, that is why WTF-8 was invented https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/
- layer8 3y ago
  Surrogate pairs are interpreted by client software, so it’s UTF-16 in that sense. The file system just doesn’t ensure that there won’t be unpaired surrogates, or other noncharacters. This is similar to strings in .NET, Java, and JavaScript.
  - heinrich5991 3y ago
    Which unfortunately means you can't rely on it being UTF-16.
    - naniwaduni 3y ago
      Nor should you. Even a well-formed sequence of utf-16 codepoints can be utter nonsense; there's approximately no level of abstraction between "sequence of fixed-width code units" and "run it through a full-blown a font rendering stack" where it makes sense to assume your input is "well-formed".
delfinom 3y ago
Actually, you can manifest your windows app as UTF8 aware and all the win32 "-A" (formerly ascii only) APIs switch to handling UTF8 :3 This is a feature of Windows 10+ https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page#-a-vs--w-apis https://learn.microsoft.com/en-us/windows/apps/design/global...
- Fulgen 3y ago
  Or notepad++ could just use the WinAPI (WideCharToMultiByte) to convert UTF-16 to UTF-8 instead of hand writing their own conversion routine.

arthur2e5 3y ago

You are right: NTFS exclusively runs on UCS-2, now "UTF-16 without validation". Back in the 1980s "Unicode" referred to a way to represent the ISO standard "Universal Character Set", i.e. UCS-2, instead of being synonymous to the entire character set like today. The wisdom of i18n back then was to use "Unicode" text encoding because it covered the known universe of writing. Qt made a similar choice.

wolf550e 3y ago

I think NTFS is UCS-2, that is why WTF-8 was invented https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/

layer8 3y ago
Surrogate pairs are interpreted by client software, so it’s UTF-16 in that sense. The file system just doesn’t ensure that there won’t be unpaired surrogates, or other noncharacters. This is similar to strings in .NET, Java, and JavaScript.
- heinrich5991 3y ago
  Which unfortunately means you can't rely on it being UTF-16.
  - naniwaduni 3y ago
    Nor should you. Even a well-formed sequence of utf-16 codepoints can be utter nonsense; there's approximately no level of abstraction between "sequence of fixed-width code units" and "run it through a full-blown a font rendering stack" where it makes sense to assume your input is "well-formed".

delfinom 3y ago

Actually, you can manifest your windows app as UTF8 aware and all the win32 "-A" (formerly ascii only) APIs switch to handling UTF8 :3 This is a feature of Windows 10+ https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page#-a-vs--w-apis https://learn.microsoft.com/en-us/windows/apps/design/global...

Fulgen 3y ago
Or notepad++ could just use the WinAPI (WideCharToMultiByte) to convert UTF-16 to UTF-8 instead of hand writing their own conversion routine.