4 ms·

"Designed as a single, global replacement for localised character sets, the Unicode standard is beautiful in its simplicity. In essence: collect all the charact

by gnosis 13y ago

"Designed as a single, global replacement for localised character sets, the Unicode standard is beautiful in its simplicity. In essence: collect all the characters in all the scripts known to humanity and number them in one single, canonical list. If new characters are invented or discovered, no problem, just add them to the list. The list isn’t an 8-bit list, or a 16-bit list, it’s just a list, with no limit on its length." Is this really true? My impression was that UTF-32 is a fixed-length encoding which uses 32 bits to encode all of Unicode. It seems that this means that Unicode can never have more code points than could fit in 32 bits. Right?

alexjeffrey 13y ago
I believe it depends on the encoding - UTF-32 has a limit of 2^32 characters, but UTF-8 could potentially expand infinitely (ignoring the current arbitrary 4-byte limit).
- cygx 13y ago
  That's incorrect - the UTF-8 scheme encodes the sequence length in the first byte as the number of bits set to 1 until the first 0. The original limit was 6 bytes encoding 31 bits, but this was lowered to 4 bytes encoding 21 bits. If I'm not mistaken, 7 bytes encoding 36 bits (or 8 bytes encoding 42 bits if you make the 0 optional) should be the hard limit.
  - drdaeman 13y ago
    If we ignore this artificial restriction, we can use arbitrary lengths, turning UTF-8 into varint encoding. I.e.: Oct.1 Oct.2 Oct.3 11111111 110DDDDD DDDDDDDD (7 octets more) \_________/ 10 bits ("D"s represent data bits) This is what'd be necessary to use UTF-8 if we'd need to go beyond the current limits of 0x010FFFF upper bound.
    - cygx 13y ago
      In your scheme, we'll end up with multiple leading bytes, though. Also, how do you plan to handle 9 leading bits instead of the 10 from your example? That would make the second byte start with 10, the marker for continuation bytes.
      - drdaeman 13y ago
        Ouch, you're right. Refreshed my memory of UTF-8 encoding and now it seems it's designed so it can't be made to express arbitrary values. Not sure why I imagined the opposite. My mistake.
    - adsr 13y ago
      The current limit (0x10FFFF) is the artificial restriction on UTF-8 to match UTF-16, not the upper bound. UTF-8 supports 2^31 code points with it's current encoding. So nothing needs to change until Unicode consists of 2.1 billion code points or so. In my mind that will only happen as a result of extreme carelessness.
cygx 13y ago
Unicode code points go up to 0x10FFFF, which fits in 21 bits. If this were ever to become a problem (which I don't see happening any time soon), the transition from UCS-2 to UTF-16 is prior art on how to pull off the extension of a coding space. Somewhat unrelated, but nevertheless worth mentioning as it's a common misconception: While UTF-32 is a fixed-length coding for Unicode characters, often, the more interesting unit is the grapheme cluster, effectively making UTF-32 into a variable-length coding.
ygra 13y ago
Unicode can never have more code points than could fit into 21 bits. Because Unicode is a 21-bit code. This has historical reasons because Unicode was a 16-bit code initially and it soon became apparent that 65536 characters are not enough, especially with the commitment of a) compatibility to every pre-existing character set and b) including historical scripts too¹ 21 bits was what emerged from expanding UCS-2 to UTF-16 via surrogate pairs and Unicode was reörganised into 17 “planes”, the first of which, the BMP, containing all code points allocated so far. UTF-32 then just was a simple encoding scheme that allows to have one code point per code unit that is also efficient to process. 21- or 24-bit code units would be unwieldy on most architectures (especially regarding unaligned memory access). ___________ ¹ Arguably the decision to include Emoji made a bigger dent in the code point space than hieroglyphs, Linear [AB], etc., though, but that came a litte later.
peteri 13y ago
Think you're right. The article could with a rewrite as Unicode used to be 16 bit until 1996 (according to Wikipedia) which explains why Java/Windows are really UTF-16 based. The more interesting question is if you're designing a new operating system would you pick UTF-8 or UTF-32 as the basis of your character system. Bearing in mind you need to normalise strings anyway for comparison purposes the general space efficiencies for UTF-8 for most systems seem tempting.
- lucian1900 13y ago
  Why would one not use UTF-8? You can't do byte comparisons or indexing, so all processing has to be linear anyway.

by gnosis 13y ago

alexjeffrey 13y ago

I believe it depends on the encoding - UTF-32 has a limit of 2^32 characters, but UTF-8 could potentially expand infinitely (ignoring the current arbitrary 4-byte limit).

cygx 13y ago
That's incorrect - the UTF-8 scheme encodes the sequence length in the first byte as the number of bits set to 1 until the first 0. The original limit was 6 bytes encoding 31 bits, but this was lowered to 4 bytes encoding 21 bits. If I'm not mistaken, 7 bytes encoding 36 bits (or 8 bytes encoding 42 bits if you make the 0 optional) should be the hard limit.
- drdaeman 13y ago
  If we ignore this artificial restriction, we can use arbitrary lengths, turning UTF-8 into varint encoding. I.e.: Oct.1 Oct.2 Oct.3 11111111 110DDDDD DDDDDDDD (7 octets more) \_________/ 10 bits ("D"s represent data bits) This is what'd be necessary to use UTF-8 if we'd need to go beyond the current limits of 0x010FFFF upper bound.
  - cygx 13y ago
    In your scheme, we'll end up with multiple leading bytes, though. Also, how do you plan to handle 9 leading bits instead of the 10 from your example? That would make the second byte start with 10, the marker for continuation bytes.
    - drdaeman 13y ago
      Ouch, you're right. Refreshed my memory of UTF-8 encoding and now it seems it's designed so it can't be made to express arbitrary values. Not sure why I imagined the opposite. My mistake.
  - adsr 13y ago
    The current limit (0x10FFFF) is the artificial restriction on UTF-8 to match UTF-16, not the upper bound. UTF-8 supports 2^31 code points with it's current encoding. So nothing needs to change until Unicode consists of 2.1 billion code points or so. In my mind that will only happen as a result of extreme carelessness.

cygx 13y ago

Unicode code points go up to 0x10FFFF, which fits in 21 bits. If this were ever to become a problem (which I don't see happening any time soon), the transition from UCS-2 to UTF-16 is prior art on how to pull off the extension of a coding space. Somewhat unrelated, but nevertheless worth mentioning as it's a common misconception: While UTF-32 is a fixed-length coding for Unicode characters, often, the more interesting unit is the grapheme cluster, effectively making UTF-32 into a variable-length coding.

ygra 13y ago

Unicode can never have more code points than could fit into 21 bits. Because Unicode is a 21-bit code. This has historical reasons because Unicode was a 16-bit code initially and it soon became apparent that 65536 characters are not enough, especially with the commitment of a) compatibility to every pre-existing character set and b) including historical scripts too¹ 21 bits was what emerged from expanding UCS-2 to UTF-16 via surrogate pairs and Unicode was reörganised into 17 “planes”, the first of which, the BMP, containing all code points allocated so far. UTF-32 then just was a simple encoding scheme that allows to have one code point per code unit that is also efficient to process. 21- or 24-bit code units would be unwieldy on most architectures (especially regarding unaligned memory access). ___________ ¹ Arguably the decision to include Emoji made a bigger dent in the code point space than hieroglyphs, Linear [AB], etc., though, but that came a litte later.

peteri 13y ago

Think you're right. The article could with a rewrite as Unicode used to be 16 bit until 1996 (according to Wikipedia) which explains why Java/Windows are really UTF-16 based. The more interesting question is if you're designing a new operating system would you pick UTF-8 or UTF-32 as the basis of your character system. Bearing in mind you need to normalise strings anyway for comparison purposes the general space efficiencies for UTF-8 for most systems seem tempting.

lucian1900 13y ago
Why would one not use UTF-8? You can't do byte comparisons or indexing, so all processing has to be linear anyway.