4 ms·

Unicode is just a cumulation of what already existed in one form or another, not an entirely new mistake. > 1. embedding invisible semantic information into th

Unicode is just a cumulation of what already existed in one form or another, not an entirely new mistake. > 1. embedding invisible semantic information into the encodings > 2. multiple encodings that mean the same thing Sure, ISO/IEC 8859 back then had no invisible characters nor composed characters, did it? [1] > 3. moved beyond standardizing existing alphabets, and wandered into a never-ending stream of people inventing all sorts of glyphs and campaigning to get them into Unicode > 4. set language back 2000 years by deciding that hieroglyphs were better than phonetics Emoji business is surely ugly (and I have complained about this a lot), but the very reason that Unicode has emoji is that Japanese telcos have exposed their priprietary encodings to the wild email and both Apple and Google had to implement them for the compatibility. Blame them, not Unicode. > 5. conflating 𝖋𝖔𝖓𝖙𝖘 with alphabets Yeah, you will be surprised to hear that Unicode encodes both the Latin letter A and Cyrillic letter A separately. (I think you had already said that they should be unified in the past, but I couldn't find that reply.) > 6. the crazy concept of "levels" of Unicode compatibility Most Unicode conformance rules are optional (past discussion: [2]). Also if you should display a Unicode text in your program and you are not using a stock library, it is up to you to decide what to support because they exist for the reason. No BiDi support? You now disregard right-to-left scripts (or left-to-right scripts if your program is for RTL users). No combining character support? Expect to lose a significant portion of potential users around the world. Probably a colored font support is fine to ignore, but that doesn't mean most other things can be ignored without the cost. [1] https://en.wikipedia.org/wiki/ISO/IEC_8859-8#Code_page_layout https://en.wikipedia.org/wiki/ISO/IEC_8859-8#Code_page_layou... [2] https://news.ycombinator.com/item?id=26904739 https://news.ycombinator.com/item?id=26904739

WalterBright 5y ago
> ISO/IEC 8859 back then had no invisible characters nor composed characters, did it? No reason to repeat mistakes. The whole point of Unicode was to fix it all. > No combining character support? There is no reason to support both combining chars and unique code points for the same glyph. Unicode should pick one and dump the other. > Most Unicode conformance rules are optional I know. This is what makes it crazy and nobody implements it right, or even knows what "right" is. > you will be surprised to hear that Unicode encodes both the Latin letter A and Cyrillic letter A separately They do more than that with visually indistinguishable code points. People use this to embed secret messages in innocuous looking text files that print identically. There's no round trip Unicode->paper->Unicode.
- coldacid 5y ago
  >No reason to repeat mistakes. The whole point of Unicode was to fix it all. The whole point of Unicode was to be The One True Encoding that served as a superset of all the others, so that anything expressible in any other character set could be expressed identically in it (after mapping code points, of course).
- lifthrasiir 5y ago
  > The whole point of Unicode was to fix it all. The whole point of Unicode was to provide a vendor-approved universal character set and encoding. Something ISO/IEC 2022 and the original draft of ISO/IEC 10646 (which was subsequently harmonized with Unicode) couldn't do. As I'll reiterate below, it is impossible to fix "mistakes" of the past because the human text is complex and these "mistakes" are a part of that. Unicode instead had more managable (and eventually successful) goal of maintaining the compatibility with existing character sets so that it can overtake them. > There is no reason to support both combining chars and unique code points for the same glyph. Unicode should pick one and dump the other. Your second claim that combining characters can replace unique code points doesn't make sense at all. If you have "combining" characters then you also need base characters to be combined. If these base characters can be used by its own then it is no different from the current Unicode. If these base characters can't be used by its own then some characters might have two code points assigned, one combinable and one not. That would be more complex than today and can cause issues in multiple scripts. I presume that you didn't think about that second claim and only thought about the first claim that unique code points can replace combining characters. This might be possible if you disregard several important scripts, and my native Korean is one of them. Pre-Unicode Korean and Hangul support in computers was always lousy and there had been three major approaches. The first is to PRECOMPOSE every common syllable. The second is to algorithmically COMPOSE consonants and vowels into a syllable block by having byte patterns calculated from jamos themselves. The third is similar to the second but RECOGNIZEs a row of composable jamos, so the byte length of each character is variable. All three approaches have been used multiple times in the history. By following your logic Hangul should be always encoded using PRECOMPOSE or COMPOSE. After all that's how Han characters (aka "CJK" "ideographs") are encoded, so it should be workable right? And yet we are still adding a slew of additional Han characters to Unicode every two or three year. Unicode 1.0 indeed had used PRECOMPOSE, but they had to switch to COMPOSE by 2.0 because additional characters kept being added. They are characters people do use, just less frequently. The adoption of COMPOSE made Hangul one of the biggest scripts in Unicode, encompassing 11,172 characters (and in part triggering the introduction of UTF-16). So we should be fine as is, right? No, because there are another beast called archaic Hangul used before the 1933 standardization. They are about as frequent as least used modern Hangul characters, but some of them are definitely in use. And here's a catch: if you implement archaic Hangul in Unicode using COMPOSE, there would be at least 1,638,750 characters of them [1]. I'm sure you are definitely not okay with that. Back then the most popular Korean word processor, nowadays called the Hangul Office, supported archaic Hangul natively. They had their own 16-bit encoding which used COMPOSE throughout modern Hangul, but their archaic Hangul encoding was very much mixed [2]. It was a combination of all three approaches above: some frequent archaic jamos are implemented just like modern Hangul using COMPOSE, remaining frequent archaic syllables got their own code a la PRECOMPOSE, and other archaic jamos are RECOGNIZEd as a character if consecutive. All of this mess because of the limited size of their encoding. And yet it was the best possible before Unicode. The modern Unicode encoding uses both COMPOSE (modern syllables only) and RECOGNIZE, and sequences in both approaches are considered equivalent. And that equivalence is not something made up just for Korean, the same approach is valid and used for many other scripts, so if you implement the Unicode normalization you've got most things right for archaic Korean. > This is what makes it crazy and nobody implements it right, or even knows what "right" is. You may want to believe that there is the "right" way, but there isn't. Unicode gives you a standardized set of algorithms and data for what they can (equivalence, normalization, collation, ...). Unicode is very fine for what they support, but it can't help you if something is out of their scope, including fonts (you were conflating this a lot, weren't you). For those things you don't have a single universal answer and your answer may change according to which languages, scripts or locales you support and how much your users can tolerate, among others. > People use this to embed secret messages in innocuous looking text files that print identically. There's no round trip Unicode->paper->Unicode. There is no round trip between any character encoding and paper. Assume that your text is monospaced and you see something like this: This line definitely doesn't have a space after this: Are you sure that there is no space after the colon? Even if we restrict ourselves to visual characters, you can do much of steganography without Unicode (font changes, whitespace counts, keming, intentional typos, alternative expressions...). You are making your own problem up. [1] https://charset.fandom.com/ko/wiki/%EC%9C%A0%EB%8B%88%EC%BD%94%EB%93%9C_%EC%98%9B%ED%95%9C%EA%B8%80_1638750%EC%9E%90 https://charset.fandom.com/ko/wiki/%EC%9C%A0%EB%8B%88%EC%BD%... [2] https://charset.fandom.com/ko/wiki/%ED%95%9C%EC%BB%B4_2%EB%B0%94%EC%9D%B4%ED%8A%B8_%EC%BD%94%EB%93%9C/%ED%95%9C%EA%B8%80 https://charset.fandom.com/ko/wiki/%ED%95%9C%EC%BB%B4_2%EB%B...
throw0101a 5y ago
> Emoji business is surely ugly (and I have complained about this a lot), but the very reason that Unicode has emoji is that Japanese telcos have exposed their priprietary encodings to the wild email and both Apple and Google had to implement them for the compatibility. Blame them, not Unicode. The initial set of emojis were brought in for compatibility purposes. The huge swaths of new emojis covering every conceivable animal, vegetable, and mineral created after that point is where problems started (IMHO).
- lifthrasiir 5y ago
  The very first version of emojis was already not just a compatibility encoding. If it were solely for compatibility every character could have been just named EMOJI COMPATIBILITY-HHHHH. Unicode (correctly) determined that the presence of initial emojis would trigger further needs and prepared basic encoding principles. No trademark characters, unify with existing characters whenever possible, country flags use generic regional indicators (there was already a specific request from national bodies at that time) and so on.

WalterBright 5y ago

> ISO/IEC 8859 back then had no invisible characters nor composed characters, did it? No reason to repeat mistakes. The whole point of Unicode was to fix it all. > No combining character support? There is no reason to support both combining chars and unique code points for the same glyph. Unicode should pick one and dump the other. > Most Unicode conformance rules are optional I know. This is what makes it crazy and nobody implements it right, or even knows what "right" is. > you will be surprised to hear that Unicode encodes both the Latin letter A and Cyrillic letter A separately They do more than that with visually indistinguishable code points. People use this to embed secret messages in innocuous looking text files that print identically. There's no round trip Unicode->paper->Unicode.

coldacid 5y ago
>No reason to repeat mistakes. The whole point of Unicode was to fix it all. The whole point of Unicode was to be The One True Encoding that served as a superset of all the others, so that anything expressible in any other character set could be expressed identically in it (after mapping code points, of course).
lifthrasiir 5y ago
> The whole point of Unicode was to fix it all. The whole point of Unicode was to provide a vendor-approved universal character set and encoding. Something ISO/IEC 2022 and the original draft of ISO/IEC 10646 (which was subsequently harmonized with Unicode) couldn't do. As I'll reiterate below, it is impossible to fix "mistakes" of the past because the human text is complex and these "mistakes" are a part of that. Unicode instead had more managable (and eventually successful) goal of maintaining the compatibility with existing character sets so that it can overtake them. > There is no reason to support both combining chars and unique code points for the same glyph. Unicode should pick one and dump the other. Your second claim that combining characters can replace unique code points doesn't make sense at all. If you have "combining" characters then you also need base characters to be combined. If these base characters can be used by its own then it is no different from the current Unicode. If these base characters can't be used by its own then some characters might have two code points assigned, one combinable and one not. That would be more complex than today and can cause issues in multiple scripts. I presume that you didn't think about that second claim and only thought about the first claim that unique code points can replace combining characters. This might be possible if you disregard several important scripts, and my native Korean is one of them. Pre-Unicode Korean and Hangul support in computers was always lousy and there had been three major approaches. The first is to PRECOMPOSE every common syllable. The second is to algorithmically COMPOSE consonants and vowels into a syllable block by having byte patterns calculated from jamos themselves. The third is similar to the second but RECOGNIZEs a row of composable jamos, so the byte length of each character is variable. All three approaches have been used multiple times in the history. By following your logic Hangul should be always encoded using PRECOMPOSE or COMPOSE. After all that's how Han characters (aka "CJK" "ideographs") are encoded, so it should be workable right? And yet we are still adding a slew of additional Han characters to Unicode every two or three year. Unicode 1.0 indeed had used PRECOMPOSE, but they had to switch to COMPOSE by 2.0 because additional characters kept being added. They are characters people do use, just less frequently. The adoption of COMPOSE made Hangul one of the biggest scripts in Unicode, encompassing 11,172 characters (and in part triggering the introduction of UTF-16). So we should be fine as is, right? No, because there are another beast called archaic Hangul used before the 1933 standardization. They are about as frequent as least used modern Hangul characters, but some of them are definitely in use. And here's a catch: if you implement archaic Hangul in Unicode using COMPOSE, there would be at least 1,638,750 characters of them [1]. I'm sure you are definitely not okay with that. Back then the most popular Korean word processor, nowadays called the Hangul Office, supported archaic Hangul natively. They had their own 16-bit encoding which used COMPOSE throughout modern Hangul, but their archaic Hangul encoding was very much mixed [2]. It was a combination of all three approaches above: some frequent archaic jamos are implemented just like modern Hangul using COMPOSE, remaining frequent archaic syllables got their own code a la PRECOMPOSE, and other archaic jamos are RECOGNIZEd as a character if consecutive. All of this mess because of the limited size of their encoding. And yet it was the best possible before Unicode. The modern Unicode encoding uses both COMPOSE (modern syllables only) and RECOGNIZE, and sequences in both approaches are considered equivalent. And that equivalence is not something made up just for Korean, the same approach is valid and used for many other scripts, so if you implement the Unicode normalization you've got most things right for archaic Korean. > This is what makes it crazy and nobody implements it right, or even knows what "right" is. You may want to believe that there is the "right" way, but there isn't. Unicode gives you a standardized set of algorithms and data for what they can (equivalence, normalization, collation, ...). Unicode is very fine for what they support, but it can't help you if something is out of their scope, including fonts (you were conflating this a lot, weren't you). For those things you don't have a single universal answer and your answer may change according to which languages, scripts or locales you support and how much your users can tolerate, among others. > People use this to embed secret messages in innocuous looking text files that print identically. There's no round trip Unicode->paper->Unicode. There is no round trip between any character encoding and paper. Assume that your text is monospaced and you see something like this: This line definitely doesn't have a space after this: Are you sure that there is no space after the colon? Even if we restrict ourselves to visual characters, you can do much of steganography without Unicode (font changes, whitespace counts, keming, intentional typos, alternative expressions...). You are making your own problem up. [1] https://charset.fandom.com/ko/wiki/%EC%9C%A0%EB%8B%88%EC%BD%94%EB%93%9C_%EC%98%9B%ED%95%9C%EA%B8%80_1638750%EC%9E%90 https://charset.fandom.com/ko/wiki/%EC%9C%A0%EB%8B%88%EC%BD%... [2] https://charset.fandom.com/ko/wiki/%ED%95%9C%EC%BB%B4_2%EB%B0%94%EC%9D%B4%ED%8A%B8_%EC%BD%94%EB%93%9C/%ED%95%9C%EA%B8%80 https://charset.fandom.com/ko/wiki/%ED%95%9C%EC%BB%B4_2%EB%B...

throw0101a 5y ago

> Emoji business is surely ugly (and I have complained about this a lot), but the very reason that Unicode has emoji is that Japanese telcos have exposed their priprietary encodings to the wild email and both Apple and Google had to implement them for the compatibility. Blame them, not Unicode. The initial set of emojis were brought in for compatibility purposes. The huge swaths of new emojis covering every conceivable animal, vegetable, and mineral created after that point is where problems started (IMHO).

lifthrasiir 5y ago
The very first version of emojis was already not just a compatibility encoding. If it were solely for compatibility every character could have been just named EMOJI COMPATIBILITY-HHHHH. Unicode (correctly) determined that the presence of initial emojis would trigger further needs and prepared basic encoding principles. No trademark characters, unify with existing characters whenever possible, country flags use generic regional indicators (there was already a specific request from national bodies at that time) and so on.