4 ms·

I believe it depends on the encoding - UTF-32 has a limit of 2^32 characters, but UTF-8 could potentially expand infinitely (ignoring the current arbitrary 4-by

• cygx 13y ago
  That's incorrect - the UTF-8 scheme encodes the sequence length in the first byte as the number of bits set to 1 until the first 0. The original limit was 6 bytes encoding 31 bits, but this was lowered to 4 bytes encoding 21 bits. If I'm not mistaken, 7 bytes encoding 36 bits (or 8 bytes encoding 42 bits if you make the 0 optional) should be the hard limit.
  • drdaeman 13y ago
    If we ignore this artificial restriction, we can use arbitrary lengths, turning UTF-8 into varint encoding. I.e.: Oct.1 Oct.2 Oct.3 11111111 110DDDDD DDDDDDDD (7 octets more) \_________/ 10 bits ("D"s represent data bits) This is what'd be necessary to use UTF-8 if we'd need to go beyond the current limits of 0x010FFFF upper bound.
    • cygx 13y ago
      In your scheme, we'll end up with multiple leading bytes, though. Also, how do you plan to handle 9 leading bits instead of the 10 from your example? That would make the second byte start with 10, the marker for continuation bytes.
      • drdaeman 13y ago
        Ouch, you're right. Refreshed my memory of UTF-8 encoding and now it seems it's designed so it can't be made to express arbitrary values. Not sure why I imagined the opposite. My mistake.
    • adsr 13y ago
      The current limit (0x10FFFF) is the artificial restriction on UTF-8 to match UTF-16, not the upper bound. UTF-8 supports 2^31 code points with it's current encoding. So nothing needs to change until Unicode consists of 2.1 billion code points or so. In my mind that will only happen as a result of extreme carelessness.