5 ms·

> For example, é (a single grapheme) is encoded in Unicode as e (U+0065 Latin Small Letter E) + ´ (U+0301 Combining Acute Accent). Two code points! It's a poo

> For example, é (a single grapheme) is encoded in Unicode as e (U+0065 Latin Small Letter E) + ´ (U+0301 Combining Acute Accent). Two code points! It's a poor and misleading example for it is definitely not how 'é' is encoded in 99.999% of all the text written in, say, french out there (french is the language where 'é' is the most common). 'é' is U+00F9, one codepoint, definitely not two. Now you could say: but it is also the two codepoints one. But that's precisely what makes Unicode the complete, total and utter clusterfuck that it is. And hence even an article explaining what every programmer should know about Unicode cannot even get the most basic example right. Which is honestly quite ironic.

wgjordan 3y ago
The author explains normalization in its own entire section several paragraphs later (Why is "Å" !== "Å" !== "Å"?).
spacechild1 3y ago
Next time read the whole article before accusing the author of incompetence! However, the author could have added a small note, e.g. "(Unicode normalization will be convered in a later section.)", to prevent knowledgable readers from rage quitting :)
ninkendo 3y ago
> Unicode the complete, total and utter clusterfuck that it is. Yikes, does it really deserve that much derision? They’re trying to standardize all written human language here. I think they’ve done a fantastic job. Pre-Unicode you had to worry about what code page a document had, and computers from different countries couldn’t interoperate. The work the consortium does is hugely important, and every decision has extremely complex tradeoffs. Composed characters makes a lot of sense, and there’s a lot of case to be made that it was the right call. The attitude of “this one thing I don’t like makes the whole thing a complete clusterfuck” is something I wish fewer engineers would have.
- rstuart4133 3y ago
  > Yikes, does it really deserve that much derision? To my simple mind it had one job: allocate every grapheme a number (code point). Had it done that, the 1/2 of the article warning you about the difficulty of iterating and modifying code points would have disappeared. But I guess it had a 2nd job: create a way of representing those numbers. The obvious way, u32, was difficult for ASCII users swallow as it quadrupled the space used for a string. The solution we settled on, UTF-8 didn't come form Unicode (or ISO). It came from Ken Thompson (the Ken Thompson, who created B, the predecessor of C), when he tired to make something workable for C. Unicode was the entity that ballsed up both of those tasks. It was a fork of ISO 10646. It's main contribution over 10646 was UCS-2 - ie 16 bits per character. That decision was so bad it had to be abandoned. Later they introduced the grapheme clusters rather than allocating a separate code point for each variant. I have no idea why, as it makes the programmers task far harder. Maybe they ran out of code points. How could they possibly run out of code points, given U32 has 4 billion of them and UTF-8 could potentially have more? Because they had to kludge their way around the USC-2 mistake to create UTF-16, and it's limited 1 million. Which leads us to the one thing in the article I disagree with: > The only downside of UTF-16 is that everything else is UTF-8, so it requires conversion every time a string is read from the network or from disk. No, that's not the only downside. There is one more: USC-2 / UTF-16 has the endianness problem. A 16 bit value needs two bytes to represent it, and you can write two bytes to storage in two ways - little endian or big endian. They didn't specify, so the same string can have two different representations on disk. They added the infamous BOM markers to distinguish between them. I could go on, but colour me singularly unimpressed with this mob.
  - lifthrasiir 3y ago
    > It was a fork of ISO 10646. It never was. The earlier draft of ISO/IEC 10646 bears absolutely no resemblance with the current 10646 and Unicode (for example, the first character in ISO/IEC DIS 10646:1990 was 0x20202020, which I believe is mapped to a space U+0020). Unicode had a much better design compared to 10646 so the final 10646 was retrofitted to Unicode instead. > It's main contribution over 10646 was UCS-2 - ie 16 bits per character. That decision was so bad it had to be abandoned. UCS-2 was already in 10646 in the draft stage. It had an even worse mechanism than surrogate pairs: escape sequences from ISO/IEC 2022 to switch groups and planes (upper 16 bits of code point). The standardized UCS-2 doesn't have them because of the merger of then-16-bit Unicode. > Later they introduced the grapheme clusters rather than allocating a separate code point for each variant. That sounds like that Unicode initially allocated separate code points for each variant. They didn't, or rather couldn't. An easy example is a Latin character with combining marks. There are a lot of combining marks in existence, some even defined before Unicode (yes, it's not the Unicode invention!), and sometimes a single character can have multiple marks. So Unicode only gave separate code points for compatibility, and otherwise resorted to the normalization mechanism that understands how to handle such cases. The concept of grapheme cluster naturally arises from the existence of normalization. Not in the strictest sense, but it can be thought as a closed set over normalization and concatenation, so that it roughly matches with user-perceived characters. So grapheme clusters were already there, only the precise algorithm was specified later. > To my simple mind it had one job: allocate every grapheme a number (code point). You can easily have more than 10M code points in this way. The current set of Hangul syllables, precomposed or not, is 125 * 95 * 138 = 1,638,750 characters. Latin characters with at most 3 combining marks (known to exist in the wild) would be probably in the same order of magnitude. Maybe now you can try, thanks to the computing power and all the information, but in 1990? Fat chance. > Maybe they ran out of code points. How could they possibly run out of code points, given U32 has 4 billion of them and UTF-8 could potentially have more? For last 20 years the rate was about 2,700 new code points per year. It would take more than 200 years to fill all other unassigned planes at this rate. And most "new" code points (in quantity) are for rare or ancient Han characters, which are technically unbounded but strongly bounded by existing ancient works and scholarly works to uncover them. I doubt there remain more than 100,000 potentially encodable Han characters.
crazygringo 3y ago
> definitely not how 'é' is encoded in 99.999% of all the text written in, say, french out there Maybe how it's input by the keyboard (I haven't checked) but not how it's output on the web or other documents. Plenty of text goes through Unicode normalization which may convert it to two codepoints.
drpixie 3y ago
> Unicode the complete, total and utter clusterfuck that it is. I'm inclined to agree - they took a big problem and made it into a big & complex problem. I propose a fix - immediately depreciate all multi-codepoint graphemes and provide single codepoint alternatives. That we should need to normalise something a basic as our text leaves soooo much room for problems. And the idea that some graphemes encode magically colour (emojis) ... :rolling_eyes_emoji:
rishav_sharan 3y ago
Unicode being an utter cluster fuck is my takeaway from this as well. Are there any alternatives to Unicode? Maybe something which has a single grapheme code point and uses utf32 for the encoding. Or anything else saner than what we have today?

wgjordan 3y ago

The author explains normalization in its own entire section several paragraphs later (Why is "Å" !== "Å" !== "Å"?).

spacechild1 3y ago

Next time read the whole article before accusing the author of incompetence! However, the author could have added a small note, e.g. "(Unicode normalization will be convered in a later section.)", to prevent knowledgable readers from rage quitting :)

ninkendo 3y ago

> Unicode the complete, total and utter clusterfuck that it is. Yikes, does it really deserve that much derision? They’re trying to standardize all written human language here. I think they’ve done a fantastic job. Pre-Unicode you had to worry about what code page a document had, and computers from different countries couldn’t interoperate. The work the consortium does is hugely important, and every decision has extremely complex tradeoffs. Composed characters makes a lot of sense, and there’s a lot of case to be made that it was the right call. The attitude of “this one thing I don’t like makes the whole thing a complete clusterfuck” is something I wish fewer engineers would have.

rstuart4133 3y ago
> Yikes, does it really deserve that much derision? To my simple mind it had one job: allocate every grapheme a number (code point). Had it done that, the 1/2 of the article warning you about the difficulty of iterating and modifying code points would have disappeared. But I guess it had a 2nd job: create a way of representing those numbers. The obvious way, u32, was difficult for ASCII users swallow as it quadrupled the space used for a string. The solution we settled on, UTF-8 didn't come form Unicode (or ISO). It came from Ken Thompson (the Ken Thompson, who created B, the predecessor of C), when he tired to make something workable for C. Unicode was the entity that ballsed up both of those tasks. It was a fork of ISO 10646. It's main contribution over 10646 was UCS-2 - ie 16 bits per character. That decision was so bad it had to be abandoned. Later they introduced the grapheme clusters rather than allocating a separate code point for each variant. I have no idea why, as it makes the programmers task far harder. Maybe they ran out of code points. How could they possibly run out of code points, given U32 has 4 billion of them and UTF-8 could potentially have more? Because they had to kludge their way around the USC-2 mistake to create UTF-16, and it's limited 1 million. Which leads us to the one thing in the article I disagree with: > The only downside of UTF-16 is that everything else is UTF-8, so it requires conversion every time a string is read from the network or from disk. No, that's not the only downside. There is one more: USC-2 / UTF-16 has the endianness problem. A 16 bit value needs two bytes to represent it, and you can write two bytes to storage in two ways - little endian or big endian. They didn't specify, so the same string can have two different representations on disk. They added the infamous BOM markers to distinguish between them. I could go on, but colour me singularly unimpressed with this mob.
- lifthrasiir 3y ago
  > It was a fork of ISO 10646. It never was. The earlier draft of ISO/IEC 10646 bears absolutely no resemblance with the current 10646 and Unicode (for example, the first character in ISO/IEC DIS 10646:1990 was 0x20202020, which I believe is mapped to a space U+0020). Unicode had a much better design compared to 10646 so the final 10646 was retrofitted to Unicode instead. > It's main contribution over 10646 was UCS-2 - ie 16 bits per character. That decision was so bad it had to be abandoned. UCS-2 was already in 10646 in the draft stage. It had an even worse mechanism than surrogate pairs: escape sequences from ISO/IEC 2022 to switch groups and planes (upper 16 bits of code point). The standardized UCS-2 doesn't have them because of the merger of then-16-bit Unicode. > Later they introduced the grapheme clusters rather than allocating a separate code point for each variant. That sounds like that Unicode initially allocated separate code points for each variant. They didn't, or rather couldn't. An easy example is a Latin character with combining marks. There are a lot of combining marks in existence, some even defined before Unicode (yes, it's not the Unicode invention!), and sometimes a single character can have multiple marks. So Unicode only gave separate code points for compatibility, and otherwise resorted to the normalization mechanism that understands how to handle such cases. The concept of grapheme cluster naturally arises from the existence of normalization. Not in the strictest sense, but it can be thought as a closed set over normalization and concatenation, so that it roughly matches with user-perceived characters. So grapheme clusters were already there, only the precise algorithm was specified later. > To my simple mind it had one job: allocate every grapheme a number (code point). You can easily have more than 10M code points in this way. The current set of Hangul syllables, precomposed or not, is 125 * 95 * 138 = 1,638,750 characters. Latin characters with at most 3 combining marks (known to exist in the wild) would be probably in the same order of magnitude. Maybe now you can try, thanks to the computing power and all the information, but in 1990? Fat chance. > Maybe they ran out of code points. How could they possibly run out of code points, given U32 has 4 billion of them and UTF-8 could potentially have more? For last 20 years the rate was about 2,700 new code points per year. It would take more than 200 years to fill all other unassigned planes at this rate. And most "new" code points (in quantity) are for rare or ancient Han characters, which are technically unbounded but strongly bounded by existing ancient works and scholarly works to uncover them. I doubt there remain more than 100,000 potentially encodable Han characters.

crazygringo 3y ago

> definitely not how 'é' is encoded in 99.999% of all the text written in, say, french out there Maybe how it's input by the keyboard (I haven't checked) but not how it's output on the web or other documents. Plenty of text goes through Unicode normalization which may convert it to two codepoints.

drpixie 3y ago

> Unicode the complete, total and utter clusterfuck that it is. I'm inclined to agree - they took a big problem and made it into a big & complex problem. I propose a fix - immediately depreciate all multi-codepoint graphemes and provide single codepoint alternatives. That we should need to normalise something a basic as our text leaves soooo much room for problems. And the idea that some graphemes encode magically colour (emojis) ... :rolling_eyes_emoji:

rishav_sharan 3y ago

Unicode being an utter cluster fuck is my takeaway from this as well. Are there any alternatives to Unicode? Maybe something which has a single grapheme code point and uses utf32 for the encoding. Or anything else saner than what we have today?