2 ms·
> UTF8 already is easily extensible to more bits, either 7 continuation bytes (and 42 bits), or infinite. Extending UTF-8 to 7 continuation bytes (or more) los
by cesarb 5y ago
> UTF8 already is easily extensible to more bits, either 7 continuation bytes (and 42 bits), or infinite.
Extending UTF-8 to 7 continuation bytes (or more) loses the useful property that the all-ones byte (0xFF) never happens in a valid UTF-8 string. Limiting it to 36 bits (6 continuation bytes) would be better.
- edflsafoiewq 5y agoWhy is that useful?
- mananaysiempre 5y agoYou can use FF as a sentinel byte internally (I think utf8proc actually does that?); given that FE never occurs, either, if you see the byte sequence corresponding to U+FEFF BYTE ORDER MARK in one of the other UTFs you can pretty much immediately tell it can’t possibly be UTF-8. (In general UTF-8, because of all the self-synchronization redundancy, has a very distinctive pattern that allows it to be detected with almost perfect reliability, and that is a frequent point of UTF-8 advocacy, which lends some irony to the fact that UTF-8 is the one encoding that Web browsers support but refuse to detect[1].) I don’t think there is any other advantage to excluding FF specifically, it’s not like we’re using punched paper tape. [1] https://hsivonen.fi/utf-8-detection/ https://hsivonen.fi/utf-8-detection/
- xigoi 5y agoYou can use 11000000 and 11000001 as sentinel bytes; since a sequence beginning with them can't possibly be minimal.