3 ms·
If we ignore this artificial restriction, we can use arbitrary lengths, turning UTF-8 into varint encoding. I.e.: Oct.1 Oct.2 Oct.3 11111111 1
by drdaeman 13y ago
If we ignore this artificial restriction, we can use arbitrary lengths, turning UTF-8 into varint encoding.
I.e.:
Oct.1 Oct.2 Oct.3
11111111 110DDDDD DDDDDDDD (7 octets more)
\_________/
10 bits
("D"s represent data bits)
This is what'd be necessary to use UTF-8 if we'd need to go beyond the current limits of 0x010FFFF upper bound.
- cygx 13y agoIn your scheme, we'll end up with multiple leading bytes, though. Also, how do you plan to handle 9 leading bits instead of the 10 from your example? That would make the second byte start with 10, the marker for continuation bytes.
- drdaeman 13y agoOuch, you're right. Refreshed my memory of UTF-8 encoding and now it seems it's designed so it can't be made to express arbitrary values. Not sure why I imagined the opposite. My mistake.
- adsr 13y agoThe current limit (0x10FFFF) is the artificial restriction on UTF-8 to match UTF-16, not the upper bound. UTF-8 supports 2^31 code points with it's current encoding. So nothing needs to change until Unicode consists of 2.1 billion code points or so. In my mind that will only happen as a result of extreme carelessness.