2 ms·
I’m going to defend this set of complaints: > “multiple encodings for the same code point”, “multiple code points for the same glyph”, and “an incomprehensible
by gumby 3y ago
I’m going to defend this set of complaints:
> “multiple encodings for the same code point”, “multiple code points for the same glyph”, and “an incomprehensible document describing all this nonsense”
One important factor for adoption was round trip idempotency. Unicode doesn’t simply encode every glyph but incorporates as many existing character sets as possible. (Of course “possible” is modulated by the fact there are/were competing encodings for the same language).
If you use a unicode-aware grep and use it to search for ‘.’, you want the result to be the same as `cat`, rather than having the result re-coded, objcopy-like.
Sure, this is less of a problem in 2023, but still a major problem that will continue indefinitely due to legacy files, legacy hardware, and small-resource devices. So there is no way in unicode to tell by inspection if ‘A’ is Latin or Greek..which is true also on the printed page.
Likewise the writing of different languages don’t merely have different appearance, but different semantics for the visually same shape. In some languages (e.g. French), diacritics are largely semantically silent: you can understand a sentence with them stripped out, though they can simplify reading in some cases by disambiguating some otherwise homonyms. In other languages (e.g. the German umlaut) they are simply contractions — fossils of medieval handwriting. Yet again, in Swedish they can be optional, like the acute accent or form completely different letters like ä. Capturing this is complicated and even discussing it requires, yes, an almost incomprehensible document.
And it has all been implemented but using those libraries is hard, because language processing is hard. In general I/O is hard.
Im not sure what you mean about “semantic meaning” but I do agree, mostly, with your other complaints.
- WalterBright 3y ago> semantic meaning There's one code point for the letter `a` and another code point for the mathematical symbol `a`, despite being visually identical. I.e. the code points have acquired semantic meaning. Spy agencies use this so different versions of a document look identical when printed. Thus, they can use this to track down the mole who leaked a sensitive document.