3 ms·

Serious question -- is this a serious project or a joke?

TazeTSchnitzel 11y ago
The name might throw you off, but it's very much serious. It's like CESU-8 and Modified UTF-8, which both deal with various encoding issues in legacy systems by modifying UTF-8: https://en.wikipedia.org/wiki/UTF-8#Derivatives https://en.wikipedia.org/wiki/UTF-8#Derivatives Note the WTF-8 entry has only been there fore a few minutes, I just added it. It might be removed for non-notability.
- TazeTSchnitzel 11y ago
  s/Note/Note that/ s/fore/for/
masklinn 11y ago
The name is unserious but the project is very serious, its writer has responded to a few comments and linked to a presentation of his on the subject[0]. It's an extension of UTF-8 used to bridge UTF-8 and UCS2-plus-surrogates: while UTF8 is the modern encoding you have to interact with legacy systems, for UNIX's bags of bytes you may be able to assume UTF8 (possibly ill-formed) but a number of other legacy systems used UCS2 and added visible surrogates (rather than proper UTF-16) afterwards. Windows and NTFS, Java, UEFI, Javascript all work with UCS2-plus-surrogates. Having to interact with those systems from a UTF8-encoded world is an issue because they don't guarantee well-formed UTF-16, they might contain unpaired surrogates which can't be decoded to a codepoint allowed in UTF-8 or UTF-32 (neither allows unpaired surrogates, for obvious reasons). WTF8 extends UTF8 with unpaired surrogates (and unpaired surrogates only, paired surrogates from valid UTF16 are decoded and re-encoded to a proper UTF8-valid codepoint) which allows interaction with legacy UCS2 systems. WTF8 exists solely as an internal encoding (in-memory representation), but it's very useful there. It was initially created for Servo (which may need it to have an UTF8 internal representation yet properly interact with javascript), but turned out to first be a boon to Rust's OS/filesystem APIs on Windows. [0] http://exyr.org/2015/!!Con_WTF-8/slides.pdf http://exyr.org/2015/!!Con_WTF-8/slides.pdf
- tjradcliffe 11y ago
  > WTF8 exists solely as an internal encoding (in-memory representation) Today. Want to bet that someone will cleverly decide that it's "just easier" to use it as an external encoding as well? This kind of cat always gets out of the bag eventually.
  - Dylan16807 11y ago
    Better WTF8 than invalid UCS2-plus-surrogates. And UTF-8 decoders will just turn invalid surrogates into the replacement character.
PaulHoule 11y ago
I thought he was tackling the other problem which is that you frequently find web pages that have both UTF-8 codepoints and single bytes encoded as ISO-latin-1 or Windows-1252

The name might throw you off, but it's very much serious. It's like CESU-8 and Modified UTF-8, which both deal with various encoding issues in legacy systems by modifying UTF-8: https://en.wikipedia.org/wiki/UTF-8#Derivatives https://en.wikipedia.org/wiki/UTF-8#Derivatives Note the WTF-8 entry has only been there fore a few minutes, I just added it. It might be removed for non-notability.

TazeTSchnitzel 11y ago
s/Note/Note that/ s/fore/for/

masklinn 11y ago

The name is unserious but the project is very serious, its writer has responded to a few comments and linked to a presentation of his on the subject[0]. It's an extension of UTF-8 used to bridge UTF-8 and UCS2-plus-surrogates: while UTF8 is the modern encoding you have to interact with legacy systems, for UNIX's bags of bytes you may be able to assume UTF8 (possibly ill-formed) but a number of other legacy systems used UCS2 and added visible surrogates (rather than proper UTF-16) afterwards. Windows and NTFS, Java, UEFI, Javascript all work with UCS2-plus-surrogates. Having to interact with those systems from a UTF8-encoded world is an issue because they don't guarantee well-formed UTF-16, they might contain unpaired surrogates which can't be decoded to a codepoint allowed in UTF-8 or UTF-32 (neither allows unpaired surrogates, for obvious reasons). WTF8 extends UTF8 with unpaired surrogates (and unpaired surrogates only, paired surrogates from valid UTF16 are decoded and re-encoded to a proper UTF8-valid codepoint) which allows interaction with legacy UCS2 systems. WTF8 exists solely as an internal encoding (in-memory representation), but it's very useful there. It was initially created for Servo (which may need it to have an UTF8 internal representation yet properly interact with javascript), but turned out to first be a boon to Rust's OS/filesystem APIs on Windows. [0] http://exyr.org/2015/!!Con_WTF-8/slides.pdf http://exyr.org/2015/!!Con_WTF-8/slides.pdf

tjradcliffe 11y ago
> WTF8 exists solely as an internal encoding (in-memory representation) Today. Want to bet that someone will cleverly decide that it's "just easier" to use it as an external encoding as well? This kind of cat always gets out of the bag eventually.
- Dylan16807 11y ago
  Better WTF8 than invalid UCS2-plus-surrogates. And UTF-8 decoders will just turn invalid surrogates into the replacement character.

PaulHoule 11y ago

I thought he was tackling the other problem which is that you frequently find web pages that have both UTF-8 codepoints and single bytes encoded as ISO-latin-1 or Windows-1252