3 ms·

Certainly not true for Windows. Windows uses UTF-16; e.g. it uses proper surrogate pairs. https://learn.microsoft.com/en-us/windows/win32/intl/surrogates-and-s

• skitter 3y ago
  That would be great, but that article is about recommendations for applications running on Windows, not about what valid file names applications may encounter. Here's a counter-example: https://github.com/golang/go/issues/32334 https://github.com/golang/go/issues/32334
  • sedatk 3y ago
    No, I mean Windows API honors UTF-16 surrogate pairs, and processes them correctly. It doesn't produce invalid UTF-16 strings either. Apps may not support UTF-16 properly, and that's not on Windows, is it? NTFS, on the other hand, has no dictated format for filename encoding. It just stores raw bytes as filenames, so anything can be a filename on NTFS, including invalid strings if the caller decides to do so. That's not on Windows either, otherwise, we should add Linux to the list too as ext4 and most other file systems also don't care about filename encoding.
• chungy 3y ago
  Windows was designed around UCS-2, while it can be upwards compatible with storing UTF-16 data, the Windows APIs don't stop you from inserting almost any sequence of 16-bit words as file names, including unpaired surrogates (because Windows doesn't actually care about the contents of the 16-bit words). It just happens that most Windows programs will treat it as UTF-16 and it'll mostly be fine. Until you run into the edge case where you do have file names that aren't UTF-16. This is basically a result of Windows's file API being fixed in 1993, a good few years before Unicode decided that 16 bits wasn't enough for everyone, and hence a good few years before UTF-16 existed.
  • dahfizz 3y ago
    Why is that a problem? That's how unix does it as well. A file path is just a sequence of bytes. Don't try to validate it in any sort of encoding, it's just bytes that the OS uses to identify a file.
    • chungy 3y ago
      The problem is that if the operating system allows sequences that are not valid UTF-16, then you cannot just state "it uses UTF-16": anything that is expecting UTF-16 and only UTF-16 is going to break on valid file names. "That's how unix does it as well" is the exact same thing. Most people these days expect file names in UTF-8, but not all file systems restrict the valid character sequences to be UTF-8 compliant. If you treat everything as if it must be UTF-8, your application may just break when it encounters a name that isn't UTF-8. (This detail gets messy because there are configurations of, eg, ext4 and ZFS that limit valid file names to be valid UTF-8.)
      • OkayPhysicist 3y ago
        Does this actually matter? Who is out there, in the wild, creating file paths that use incredibly cursed random sequences of bytes? Maybe it's just me, but I don't really see the need to accommodate people who do stupid things to see what breaks.
        • chrismorgan 3y ago
          On Windows, it doesn’t matter much because, although the file system doesn’t validate Unicode, the usual system calls that work with file names do, so it’s very rare to encounter non-Unicode file names. On Linux, it’s not as rare as you might expect to end up with a few non-UTF–8 file names, though they’ll normally be on things you won’t often touch directly, so things like desktop software can reasonably only support UTF-8 paths.
          • OkayPhysicist 3y ago