5 ms·

Excellent presentation! One improvement to consider is that many usages of "code point" should be "Unicode scalar value" instead. Basically, you don't want to u

by nayuki 5y ago

Excellent presentation! One improvement to consider is that many usages of "code point" should be "Unicode scalar value" instead. Basically, you don't want to use UTF-8 to encode UTF-16 surrogate code points (which are not scalar values). Fun fact, UTF-8's prefix scheme can cover up to 31 payload bits. See https://en.wikipedia.org/wiki/UTF-8#FSS-UTF https://en.wikipedia.org/wiki/UTF-8#FSS-UTF , section "FSS-UTF (1992) / UTF-8 (1993)". A manifesto that was much more important ~15 years ago when UTF-8 hadn't completely won yet: https://utf8everywhere.org/ https://utf8everywhere.org/

masklinn 5y ago
> Fun fact, UTF-8's prefix scheme can cover up to 31 payload bits. It’d probably be more correct to say that it was originally defined to cover 31 payload bits: you can easily complete the first byte to get 7 and 8 byte sequences (35 and 41 bits payloads). Alternatively, you could save the 11111111 leading byte to flag the following bytes as counts (5 bits each since you’d need a flag bit to indicate whether this was the last), then add the actual payload afterwards, this would give you an infinite-size payload, though it would make the payload size dynamic and streamed (where currently you can get the entire USV in two fetches, as the first byte tells you exactly how many continuation bytes you need).
- SethMLarson 5y ago
  Yeah the current definition is restricted to 4 octets in RFC 3629. Really interesting to see the history of ranges UTF-8 was able to cover.
CountSessine 5y ago
Basically, you don't want to use UTF-8 to encode UTF-16 surrogate code points The awful truth is that there is such a beast. UTF-8 wrapper with UTF-16 surrogate pairs. https://en.wikipedia.org/wiki/CESU-8 https://en.wikipedia.org/wiki/CESU-8
- nayuki 5y ago
  Is CESU-8 a synonym of WTF-8? https://en.wikipedia.org/wiki/UTF-8#WTF-8 https://en.wikipedia.org/wiki/UTF-8#WTF-8 ; https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/
  - LegionMammal978 5y ago
    No. Let UTF-8* denote UTF-8, except that surrogate code points are allowed and have no special meaning. To encode a string in CESU-8, each supplementary character is converted to a surrogate pair, leaving existing surrogates alone, then each codepoint is encoded as UTF-8*. To encode a string in WTF-8, each surrogate pair is converted to a supplementary character, leaving unpaired surrogates alone, then each codepoint is encoded as UTF-8*. So really, they have the opposite effect; CESU-8 always uses surrogate characters, whereas WTF-8 removes them if possible. Both are similar in that they directly UTF-8*-encode unpaired surrogates.
    - account42 5y ago
      The obvious advantage being that WTF-16 -> WTF-8 conversion maps all valid UTF-16 to the corresponding valid UTF-8 and only unpaired surrogates will produce invalid UTF-8 - but only invalid because UTF-8 explicitly disallows encoding those surrogates and not because the actual encoding differs, so you can almost always treat WTF-8 as UTF-8.

by nayuki 5y ago

masklinn 5y ago

> Fun fact, UTF-8's prefix scheme can cover up to 31 payload bits. It’d probably be more correct to say that it was originally defined to cover 31 payload bits: you can easily complete the first byte to get 7 and 8 byte sequences (35 and 41 bits payloads). Alternatively, you could save the 11111111 leading byte to flag the following bytes as counts (5 bits each since you’d need a flag bit to indicate whether this was the last), then add the actual payload afterwards, this would give you an infinite-size payload, though it would make the payload size dynamic and streamed (where currently you can get the entire USV in two fetches, as the first byte tells you exactly how many continuation bytes you need).

SethMLarson 5y ago
Yeah the current definition is restricted to 4 octets in RFC 3629. Really interesting to see the history of ranges UTF-8 was able to cover.

CountSessine 5y ago

Basically, you don't want to use UTF-8 to encode UTF-16 surrogate code points The awful truth is that there is such a beast. UTF-8 wrapper with UTF-16 surrogate pairs. https://en.wikipedia.org/wiki/CESU-8 https://en.wikipedia.org/wiki/CESU-8

nayuki 5y ago
Is CESU-8 a synonym of WTF-8? https://en.wikipedia.org/wiki/UTF-8#WTF-8 https://en.wikipedia.org/wiki/UTF-8#WTF-8 ; https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/
- LegionMammal978 5y ago
  No. Let UTF-8* denote UTF-8, except that surrogate code points are allowed and have no special meaning. To encode a string in CESU-8, each supplementary character is converted to a surrogate pair, leaving existing surrogates alone, then each codepoint is encoded as UTF-8*. To encode a string in WTF-8, each surrogate pair is converted to a supplementary character, leaving unpaired surrogates alone, then each codepoint is encoded as UTF-8*. So really, they have the opposite effect; CESU-8 always uses surrogate characters, whereas WTF-8 removes them if possible. Both are similar in that they directly UTF-8*-encode unpaired surrogates.
  - account42 5y ago
    The obvious advantage being that WTF-16 -> WTF-8 conversion maps all valid UTF-16 to the corresponding valid UTF-8 and only unpaired surrogates will produce invalid UTF-8 - but only invalid because UTF-8 explicitly disallows encoding those surrogates and not because the actual encoding differs, so you can almost always treat WTF-8 as UTF-8.