3 ms·
UTF-8 as defined (or restricted) is a prefix code, it gets all relevant information on the first read, and the rest on the (optional) second. Your scheme requir
by masklinn 5y ago
UTF-8 as defined (or restricted) is a prefix code, it gets all relevant information on the first read, and the rest on the (optional) second. Your scheme requires an unbounded number of reads.
> - It is easily extensible for more bits.
UTF8 already is easily extensible to more bits, either 7 continuation bytes (and 42 bits), or infinite. Neither of which is actually useful to its purposes.
> The last point is key: UTF-8 would need to invent a new prefix to go beyond 21 bits
UTF8 was defined as encoding 31 bits over 6 bytes. It was restricted to 21 bits (over 4 bytes) when unicode itself was restricted to 21 bits.
- cesarb 5y ago> UTF8 already is easily extensible to more bits, either 7 continuation bytes (and 42 bits), or infinite. Extending UTF-8 to 7 continuation bytes (or more) loses the useful property that the all-ones byte (0xFF) never happens in a valid UTF-8 string. Limiting it to 36 bits (6 continuation bytes) would be better.
- edflsafoiewq 5y agoWhy is that useful?
- mananaysiempre 5y agoYou can use FF as a sentinel byte internally (I think utf8proc actually does that?); given that FE never occurs, either, if you see the byte sequence corresponding to U+FEFF BYTE ORDER MARK in one of the other UTFs you can pretty much immediately tell it can’t possibly be UTF-8. (In general UTF-8, because of all the self-synchronization redundancy, has a very distinctive pattern that allows it to be detected with almost perfect reliability, and that is a frequent point of UTF-8 advocacy, which lends some irony to the fact that UTF-8 is the one encoding that Web browsers support but refuse to detect[1].) I don’t think there is any other advantage to excluding FF specifically, it’s not like we’re using punched paper tape. [1] https://hsivonen.fi/utf-8-detection/ https://hsivonen.fi/utf-8-detection/
- xigoi 5y agoYou can use 11000000 and 11000001 as sentinel bytes; since a sequence beginning with them can't possibly be minimal.
- cryptonector 5y agoAnd Unicode was restricted to 21 bits because of UTF-16. There is still the possibility of that restriction being lifted eventually.
- pierrebai 5y agoNo software decodes data by reading a stream byte-by-byte. Like I said in a previous comment, decoding 4 bytes using SIMD is possible and probably the best way to go. Furthermore, to actually decode, you need bit twiddling anyway, so you do need to do byte-processing. Finally, the inner loop of detecting character boundary is simpler: the UTF-8 scheme, due to the variable-length prefixes, requires to detect the first non-1 bits. It is probably written with a switch/case in C, vs two bit tests in my scheme. I'm not convinced the UTF-8 ends-up with a faster loop.
- wahern 5y ago> No software decodes data by reading a stream byte-by-byte. Maybe not in your bubble, but in my bubble this is the common case and highly optimized components using SIMD parsing is the exception. By byte-by-byte, I presume you mean logically, not invoking a syscall for every byte. A parsing library is often handed a buffer to parse. If every layer had it's own buffering, you'd end up with too much data copying, which compounds and can spill your CPU caches in ways that microbenchmarks won't reflect, particularly in a streaming pipeline. So you stick to simpler, straight-forward code and data paths, unless and until you know a particular component is a bottleneck. I've had much better results optimizing globally first before optimizing locally. You can usually go back and optimize locally whenever you want, but optimizing globally is more often a one-shot deal as it typically requires non-local (i.e. cross-component) analysis and refactors, which nobody has time for.