3 ms·

WTF-8 is actually a real encoding, used for encoding invalid UTF-16 unpaired surrogates for UTF-8 systems: https://simonsapin.github.io/wtf-8/ https://simonsapi

WTF-8 is actually a real encoding, used for encoding invalid UTF-16 unpaired surrogates for UTF-8 systems: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/

bjackman 2y ago
I believe this is what Rust OsStrings are under the hood on Windows.
- extraduder_ire 2y ago
  Which I assume stands for "Windows-Transformation-Format-8(bits)".
  - mmoskal 2y ago
    Abstract WTF-8 (Wobbly Transformation Format − 8-bit) is a superset of UTF-8 that encodes surrogate code points if they are not in a pair.
    - hedora 2y ago
      Can you still assume the bytes 0x00 and 0xFF are not present in the string (like in UTF-8?)
      - int_19h 2y ago
        Yes. The only difference between UTF-8 and WTF-8 is that the latter does not reject otherwise valid UTF-8 byte sequences that correspond to codepoints in range U+D800 to U+DFFF (which means that, in practice, a lot of things that say they are UTF-8 are actually WTF-8).
        account42 2y ago
        Not really since you are unlikely to end up with unpaired surrogates outside of UTF-16 unless you explicitly implement a WTF-16 decoder - most other things are going to error out or remove/replace the garbage data when converting to another encoding. And if you convert valid UTF-16 by interpreting them as UCS-2 and then not check for invalid code points you are going to end up with either valid UTF-8 or something that isn't even valid WTF-8 since that encoding disallows paired surrogates to be encoded individually. WTF-16 is something that occurs naturally. WTF-8 isn't.
ptx 2y ago
Yeah, that had me confused for a bit. And you would never use "charset=wtf-8" (as in the title for this page) because the spec says: "Any WTF-8 data must be converted to a Unicode encoding at the system’s boundary before being emitted. UTF-8 is recommended. WTF-8 must not be used to represent text in a file format or for transmission over the Internet."
- account42 2y ago
  Some specs like to claim things that are out of their jurisdiction. A format spec has no say on where that format is used. It's best to ignore such hubris. And in this particular case it doesn't even make sense because the entire point is to round trip WTF-16. If that requires one "system" to communicate WTF-8 with another "system" (which is really an arbitrary boundary) then so be it. And anything that expects UTF-8 will need to deal with "invalid" data in a usecase-dependent way anyway.

WTF-8 is actually a real encoding, used for encoding invalid UTF-16 unpaired surrogates for UTF-8 systems: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/

bjackman 2y ago

I believe this is what Rust OsStrings are under the hood on Windows.

extraduder_ire 2y ago
Which I assume stands for "Windows-Transformation-Format-8(bits)".
- mmoskal 2y ago
  Abstract WTF-8 (Wobbly Transformation Format − 8-bit) is a superset of UTF-8 that encodes surrogate code points if they are not in a pair.
  - hedora 2y ago
    Can you still assume the bytes 0x00 and 0xFF are not present in the string (like in UTF-8?)
    - int_19h 2y ago
      Yes. The only difference between UTF-8 and WTF-8 is that the latter does not reject otherwise valid UTF-8 byte sequences that correspond to codepoints in range U+D800 to U+DFFF (which means that, in practice, a lot of things that say they are UTF-8 are actually WTF-8).
      - account42 2y ago
        Not really since you are unlikely to end up with unpaired surrogates outside of UTF-16 unless you explicitly implement a WTF-16 decoder - most other things are going to error out or remove/replace the garbage data when converting to another encoding. And if you convert valid UTF-16 by interpreting them as UCS-2 and then not check for invalid code points you are going to end up with either valid UTF-8 or something that isn't even valid WTF-8 since that encoding disallows paired surrogates to be encoded individually. WTF-16 is something that occurs naturally. WTF-8 isn't.

ptx 2y ago

Yeah, that had me confused for a bit. And you would never use "charset=wtf-8" (as in the title for this page) because the spec says: "Any WTF-8 data must be converted to a Unicode encoding at the system’s boundary before being emitted. UTF-8 is recommended. WTF-8 must not be used to represent text in a file format or for transmission over the Internet."

account42 2y ago
Some specs like to claim things that are out of their jurisdiction. A format spec has no say on where that format is used. It's best to ignore such hubris. And in this particular case it doesn't even make sense because the entire point is to round trip WTF-16. If that requires one "system" to communicate WTF-8 with another "system" (which is really an arbitrary boundary) then so be it. And anything that expects UTF-8 will need to deal with "invalid" data in a usecase-dependent way anyway.