3 ms·
UTF-8 is a "fancy" way of representing 32 bit integers in a self-synchronising byte format. I can see an extension to 7 bytes (by making the leading byte 11111
by masklinn 6y ago
UTF-8 is a "fancy" way of representing 32 bit integers in a self-synchronising byte format.
I can see an extension to 7 bytes (by making the leading byte 11111110 which I think would still be unambiguous) going up to 37 bits, but at 8 bytes you'd start getting collisions in the byte patterns and would lose the self-synchronising properties.
- kstenerud 6y agoYou could in theory extend the pattern up to 42 bits: 0xxxxxxx 110xxxxx 10xxxxxx 1110xxxx 10xxxxxx 10xxxxxx 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 11111110 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 11111111 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
- masklinn 6y agoTrue, not sure why I figured FF was not an acceptable LBP. Possibly because I assumed you'd need a special continuation byte and all of them would be unavailable, but the existing continuation bytes work fine so that's just not correct. FF could even lead to special patterns of continuation bytes allowing for smuggling more data in there I guess.
- lifthrasiir 6y ago> 32 bit integers The original UTF-8 was at most 31 bits long (one bit in the leading byte, 6*5 = 30 bits in six subsequent bytes). > you'd start getting collisions in the byte patterns and would lose the self-synchronising properties I'm not sure why. The main requirement for self-synchronization in this case is that the leading byte doesn't appear anywhere else, which would be the case for the leading byte FF. It would require substantial modifications to go beyond 7 bytes (say, it would have alternative means to denote the length of following data bytes) but it is surely possible.
- deleted 6y ago[deleted]