4 ms·

Annoyingly, Java, JavaScript, Windows file paths and more don't quite use UTF-16 (well, even if they did, that would be annoying) — they allow unpaired surrogat

Annoyingly, Java, JavaScript, Windows file paths and more don't quite use UTF-16 (well, even if they did, that would be annoying) — they allow unpaired surrogates, which don't represent any Unicode character. So if you want to represent e.g. an arbitrary Windows file path in UTF-8, you can't; you have to use WTF-8 (wobbly transformation format) instead.

deadbeeves 3y ago
But UTF-8 is just a way to encode a number as a variable-length string of octets. Why would you be unable to encode, say, a terminating U+D800 as a string of three bytes at the end of a UTF-8 stream?
- skitter 3y ago
  Because that's how UTF-8 is defined[1]. WTF-8 lifts that restriction. [1] https://simonsapin.github.io/wtf-8/#utf-8 https://simonsapin.github.io/wtf-8/#utf-8
  - deadbeeves 3y ago
    It doesn't sound very annoying, then. You use the exact same encoding scheme, but skip a verification step. Actually it sounds more convenient.
    - jraph 3y ago
      Still potentially annoying if you deal with some other code that expects UTF-8 proper and you pass it a wtf-8 string that fails the lifted verification.
- deleted 3y ago
  [deleted]
sedatk 3y ago
Certainly not true for Windows. Windows uses UTF-16; e.g. it uses proper surrogate pairs. https://learn.microsoft.com/en-us/windows/win32/intl/surrogates-and-supplementary-characters https://learn.microsoft.com/en-us/windows/win32/intl/surroga...
- skitter 3y ago
  That would be great, but that article is about recommendations for applications running on Windows, not about what valid file names applications may encounter. Here's a counter-example: https://github.com/golang/go/issues/32334 https://github.com/golang/go/issues/32334
  - sedatk 3y ago
    No, I mean Windows API honors UTF-16 surrogate pairs, and processes them correctly. It doesn't produce invalid UTF-16 strings either. Apps may not support UTF-16 properly, and that's not on Windows, is it? NTFS, on the other hand, has no dictated format for filename encoding. It just stores raw bytes as filenames, so anything can be a filename on NTFS, including invalid strings if the caller decides to do so. That's not on Windows either, otherwise, we should add Linux to the list too as ext4 and most other file systems also don't care about filename encoding.
- chungy 3y ago
  Windows was designed around UCS-2, while it can be upwards compatible with storing UTF-16 data, the Windows APIs don't stop you from inserting almost any sequence of 16-bit words as file names, including unpaired surrogates (because Windows doesn't actually care about the contents of the 16-bit words). It just happens that most Windows programs will treat it as UTF-16 and it'll mostly be fine. Until you run into the edge case where you do have file names that aren't UTF-16. This is basically a result of Windows's file API being fixed in 1993, a good few years before Unicode decided that 16 bits wasn't enough for everyone, and hence a good few years before UTF-16 existed.
  - dahfizz 3y ago
Knee_Pain 3y ago
>WTF-8 truly an appropriate name
chrismorgan 3y ago
You can call this potentially ill-formed UTF-16, or more practically unvalidated UTF-16. In practice, UTF-16 validation is rare.

deadbeeves 3y ago

But UTF-8 is just a way to encode a number as a variable-length string of octets. Why would you be unable to encode, say, a terminating U+D800 as a string of three bytes at the end of a UTF-8 stream?

skitter 3y ago
Because that's how UTF-8 is defined[1]. WTF-8 lifts that restriction. [1] https://simonsapin.github.io/wtf-8/#utf-8 https://simonsapin.github.io/wtf-8/#utf-8
- deadbeeves 3y ago
  It doesn't sound very annoying, then. You use the exact same encoding scheme, but skip a verification step. Actually it sounds more convenient.
  - jraph 3y ago
    Still potentially annoying if you deal with some other code that expects UTF-8 proper and you pass it a wtf-8 string that fails the lifted verification.
deleted 3y ago
[deleted]

sedatk 3y ago

Certainly not true for Windows. Windows uses UTF-16; e.g. it uses proper surrogate pairs. https://learn.microsoft.com/en-us/windows/win32/intl/surrogates-and-supplementary-characters https://learn.microsoft.com/en-us/windows/win32/intl/surroga...

skitter 3y ago
That would be great, but that article is about recommendations for applications running on Windows, not about what valid file names applications may encounter. Here's a counter-example: https://github.com/golang/go/issues/32334 https://github.com/golang/go/issues/32334
- sedatk 3y ago
  No, I mean Windows API honors UTF-16 surrogate pairs, and processes them correctly. It doesn't produce invalid UTF-16 strings either. Apps may not support UTF-16 properly, and that's not on Windows, is it? NTFS, on the other hand, has no dictated format for filename encoding. It just stores raw bytes as filenames, so anything can be a filename on NTFS, including invalid strings if the caller decides to do so. That's not on Windows either, otherwise, we should add Linux to the list too as ext4 and most other file systems also don't care about filename encoding.
chungy 3y ago
Windows was designed around UCS-2, while it can be upwards compatible with storing UTF-16 data, the Windows APIs don't stop you from inserting almost any sequence of 16-bit words as file names, including unpaired surrogates (because Windows doesn't actually care about the contents of the 16-bit words). It just happens that most Windows programs will treat it as UTF-16 and it'll mostly be fine. Until you run into the edge case where you do have file names that aren't UTF-16. This is basically a result of Windows's file API being fixed in 1993, a good few years before Unicode decided that 16 bits wasn't enough for everyone, and hence a good few years before UTF-16 existed.
- dahfizz 3y ago

Knee_Pain 3y ago

>WTF-8 truly an appropriate name

chrismorgan 3y ago

You can call this potentially ill-formed UTF-16, or more practically unvalidated UTF-16. In practice, UTF-16 validation is rare.