6 ms·

Name one writing system where you really need character composition. Even if there is one, these special cases should be handled outside of Unicode.

nottorp 3y ago
The problem is not that you need character composition for some writing systems. It's that there are no rules that would help with everything having an unique representation internally. Even "put the code points forming the composed character in descending numerical order" would be better than nothing. If it was there from the start. However, the Unicode commitee is too busy adding new emojis to make their standard sane.
- PeterisP 3y ago
  There are rules for that, Unicode has standards (not only formal, but easily usable in most software libraries) for canonical forms that will collapse all the variations to a single representation. But, of course, unicode can't define that the standard will cover only the canonical forms, and couldn't do that since the start, as it needed backwards compatibility with various pre-unicode encodings which had mutually incompatible principles, so it needed support for both composed and decomposed versions of the same characters.
  - nottorp 3y ago
    > for canonical formS There' your problem right there. Plural formS. It's not canonical if there are more of it.
asherah 3y ago
you can't not handle devanagari, tamil (or like half the scripts across the Indian subcontinent and oceania) or hangul. even the IPA, used by linguists every day, would be particularly bad to deal with if we couldn't write things like /á̤/, and some languages already don't have the precomposed diacritics for all letters (like ǿ), so the idea of a world with only precomposed letter forms is more of a exponential explosion in the character set
- nwellnhof 3y ago
  Hangul already has precomposed syllables in Unicode. We still have several hundred thousand unassigned codepoints to deal with diacritics.
- arp242 3y ago
  > so the idea of a world with only precomposed letter forms is more of a exponential explosion in the character set "Exponential explosion" is really putting it too strong; it's perfectly possible to just add ǿ and á̤ and a bunch of other things. The combinations aren't infinite here. The problem with e.g. Latin script isn't necessarily that combining characters exist, but that there's two ways to represent many things. That really is just a "mess": use either one system or the other, but not both. Hangul has similar problems. Devanagari doesn't have any pre-compose characters AFAIK, so that's fine. That's really the "mess": it's a hodgepodge of different systems, and you can't even know which system to use a lot of the time because it's not organised ("look it up in a large database"), and even taking in to account historical legacy I don't think it really needed to be like this (or is even an unfixable problem today, strictly speaking). At least they deprecated ligatures like ﬆ and ﬂ, although recently I did see ĳ being used in the wild.
  - WorldMaker 3y ago
    > The combinations aren't infinite here. They certainly are. Languages are a creative space driven by the human imagination. Give people enough time and they'll build new combinations for fun or for profit or for research or for trying to capture a spoken word/tone poem in just the right sort of exciting way. You may frown on "Zalgo text" [1] (and it is terrible for accessibility), but it speaks to a creative mood or three. The growing combinatorial explosion in Unicode's emoji space isn't an accident or something unique to emoji, but a characteristic that emoji are just as much a creative language as everything else Unicode encodes. The biggest difference is that it is a living language with a lot of visible creative work happening in contemporary writing as opposed to a language some monks centuries ago decided was "good enough" and school teachers long ago locked some of the creative tools in the figurative closets to keep their curriculum simpler and their days with fewer headaches. [1] https://en.wikipedia.org/wiki/Zalgo_text https://en.wikipedia.org/wiki/Zalgo_text
PetitPrince 3y ago
The intent of Unicode was to have a universal solution for humans. Excluding one case, even if it's remote, would defeat this mission statement.
layer8 3y ago
Thai, Arabic, Hebrew, and Devanagari are important examples, I believe.

nottorp 3y ago

The problem is not that you need character composition for some writing systems. It's that there are no rules that would help with everything having an unique representation internally. Even "put the code points forming the composed character in descending numerical order" would be better than nothing. If it was there from the start. However, the Unicode commitee is too busy adding new emojis to make their standard sane.

PeterisP 3y ago
There are rules for that, Unicode has standards (not only formal, but easily usable in most software libraries) for canonical forms that will collapse all the variations to a single representation. But, of course, unicode can't define that the standard will cover only the canonical forms, and couldn't do that since the start, as it needed backwards compatibility with various pre-unicode encodings which had mutually incompatible principles, so it needed support for both composed and decomposed versions of the same characters.
- nottorp 3y ago
  > for canonical formS There' your problem right there. Plural formS. It's not canonical if there are more of it.

asherah 3y ago

you can't not handle devanagari, tamil (or like half the scripts across the Indian subcontinent and oceania) or hangul. even the IPA, used by linguists every day, would be particularly bad to deal with if we couldn't write things like /á̤/, and some languages already don't have the precomposed diacritics for all letters (like ǿ), so the idea of a world with only precomposed letter forms is more of a exponential explosion in the character set

nwellnhof 3y ago
Hangul already has precomposed syllables in Unicode. We still have several hundred thousand unassigned codepoints to deal with diacritics.
arp242 3y ago
> so the idea of a world with only precomposed letter forms is more of a exponential explosion in the character set "Exponential explosion" is really putting it too strong; it's perfectly possible to just add ǿ and á̤ and a bunch of other things. The combinations aren't infinite here. The problem with e.g. Latin script isn't necessarily that combining characters exist, but that there's two ways to represent many things. That really is just a "mess": use either one system or the other, but not both. Hangul has similar problems. Devanagari doesn't have any pre-compose characters AFAIK, so that's fine. That's really the "mess": it's a hodgepodge of different systems, and you can't even know which system to use a lot of the time because it's not organised ("look it up in a large database"), and even taking in to account historical legacy I don't think it really needed to be like this (or is even an unfixable problem today, strictly speaking). At least they deprecated ligatures like ﬆ and ﬂ, although recently I did see ĳ being used in the wild.
- WorldMaker 3y ago
  > The combinations aren't infinite here. They certainly are. Languages are a creative space driven by the human imagination. Give people enough time and they'll build new combinations for fun or for profit or for research or for trying to capture a spoken word/tone poem in just the right sort of exciting way. You may frown on "Zalgo text" [1] (and it is terrible for accessibility), but it speaks to a creative mood or three. The growing combinatorial explosion in Unicode's emoji space isn't an accident or something unique to emoji, but a characteristic that emoji are just as much a creative language as everything else Unicode encodes. The biggest difference is that it is a living language with a lot of visible creative work happening in contemporary writing as opposed to a language some monks centuries ago decided was "good enough" and school teachers long ago locked some of the creative tools in the figurative closets to keep their curriculum simpler and their days with fewer headaches. [1] https://en.wikipedia.org/wiki/Zalgo_text https://en.wikipedia.org/wiki/Zalgo_text

PetitPrince 3y ago

The intent of Unicode was to have a universal solution for humans. Excluding one case, even if it's remote, would defeat this mission statement.

layer8 3y ago

Thai, Arabic, Hebrew, and Devanagari are important examples, I believe.