6 ms·
It's a pity that Unicode forgot its purpose and added: 1. formatting 2. fonts 3. semantic meaning 4. multiple encodings for the same code point 5. you vote
by WalterBright 3y ago
It's a pity that Unicode forgot its purpose and added:
1. formatting
2. fonts
3. semantic meaning
4. multiple encodings for the same code point
5. you vote for my invented character and I'll vote for your invented character
6. multiple code points for the same glyph
7. an incomprehensible document describing all this nonsense
8. an impossibility to implement it all
- lifthrasiir 3y agoThe only fault of Unicode is 4 (and we have eventually solved it anyway). Everything else is a natural complication from human writing systems.
- WalterBright 3y agoFonts, for example. That is a style, and should be set by the rendering instructions (like CSS) not the code point. Semantics: there's a code point for the letter 'a' and another code point for the mathematical symbol 'a', although the glyphs are identical. Unicode points should not have semantic meaning - semantic meaning comes from the context in which the points are used. You know, like the text in a printed book. And so on, for the other bullet points.
- lifthrasiir 3y agoI have already explained why mathematical alphanumeric symbols exist in the first place [1]. They are an extremely small portion of Unicode; you need way more examples. By the way, if we indeed decide to encode characters according to glyphs, we would have to distinguish a double-storey `a` from a single-storey `a` (among others). They have clearly different glyphs, so why shouldn't we? Think about that. [1] https://news.ycombinator.com/item?id=28294032 https://news.ycombinator.com/item?id=28294032
- WalterBright 3y ago> I have already explained But somehow mathematical books have done just fine without any need for such encodings. Meaning in written language is inferred from the context, not the symbol. This is because meanings are infinite in variety, and symbols are finite. > Think about that I did. That's what fonts are for. Fonts come in endless variety, and so cannot be encoded into Unicode. They are an orthogonal property of rendered text. And so is formatting - it's an orthogonal property.
- lifthrasiir 3y ago> But somehow mathematical books have done just fine without any need for such encodings. Printed books have no reason to think about encodings. In fact, they can use one-off metal types (or images nowadays) for unusual characters. In return printed books cannot be easily searched if the entry doesn't appear in prepared indices. > Fonts come in endless variety, and so cannot be encoded into Unicode. But some fonts convey semantics. For example a normal text does not distinguish two variants of `a`, but IPA does, so Unicode does encode a single-storey `ɑ` separately and unify a double-storey `a` to the normal lowercase `a`. (p. 299 of the Unicode standard 15.0) Semantics is arguably the only way to judge those cases. If fonts do not matter but glyphs do matter, how would know a particular glyph should be identically encoded to other glyph or not? I think you agree that a double-storey `a` and a single-storey `a` are same characters with different fonts. Semantics is how we decide the "same characters" part!
- WalterBright 3y ago> Printed books have no reason to think about encodings Exactly! It's the visual that matters. If you cannot print Unicode text on a sheet of paper and know its meaning, then Unicode has gone into a ditch. > printed books cannot be easily searched if the entry doesn't appear in prepared indices Are you going to invent a new encoding for every distinct use of the letter 'a'? That's not remotely possible. > But some fonts convey semantics People invent new fonts every day. Hell, I've invented fonts! Your mission to encode them all into Unicode is an instantly hopeless task. Again, semantics are divined from context. I know a mathematical 'i' from the english letter 'i' and the french letter 'i' and the bullet point 'i' and the electric current 'i' by context, and so do you and everyone else. I even use the letter 'A' to represent an Army in my computer game Empire. Nobody has a problem divining the meaning. (and B for Battleship, S for Submarine , etc.)
- derefr 3y ago> Semantics: there's a code point for the letter 'a' and another code point for the mathematical symbol 'a', although the glyphs are identical. Unicode points should not have semantic meaning - semantic meaning comes from the context in which the points are used. Just the opposite. A codepoint looks different in different fonts, and so things that are "identical" in one font won't be "identical" in another font. Only the semantics are the same. Printed "A" and cursive "A" are both, semantically, "latin capital A". Those semantics are very concrete and specific: they include its sort order, its conversion re: case-conversion, its joining/splitting behavior, its categoric matching in regexps, etc. But the codepoint "latin capital A" expresses no preference over whether it should be rendered as a printed/roman A or a cursive A. Codepoints don't define glyphs; they only define semantics! The glyphs associated with codepoints are conventional, following the codepoint's description! Further, codepoints tell accessibility technologies and other machine-agents what a codepoint "is." Greek letter π is text-to-speech'ed as a letter, and together with other greek letters, as a greek word. Mathematical symbol 𝜋 is text-to-speech'ed as the word "pi."
- WalterBright 3y ago> Codepoints don't define glyphs; they only define semantics! And this is the fundamental failure of Unicode post 16 bit code points. Unicode has set itself an absurdly impossible task.
- derefr 3y agoThis is how every character encoding ever has always worked. Text on computers — text over telegraph lines, even! — has always, implicitly, been a set of abstract standardized semiotic codels, with effectively-arbitrary per-OS/per-machine/per-font-family/per-style/per-size(!) graphical renditions. > Unicode has set itself an absurdly impossible task. Again: "Unicode" is not the problem here. "Human language" is the "absurdly impossible task", and has been, from ~40,000BCE. Unicode is just a formalist lens through which the scope of the task is made legible. (And I think you're being a slight bit scope-insensitive here. You should expect a project like "encoding all of human language" to be something that takes hundreds or thousands of years to get right. Unicode isn't a project, it's a megaproject, something that'll still be evolving fifteen human generations from now. But humanity as a whole has built many successful megaprojects! In the only-50-years-old realm of software engineering, we don't tend to rate things of this scope and scale as "possible" or "practical" or "tenable" — but Unicode fundamentally isn't a software engineering project; it's a language engineering project, like the project of the Académie Française, or of the creation of Simplified Chinese. Language engineering is, by definition, a project of indefinite length.)
- WalterBright 3y agoWhat should give pause in advocacy for the tarpit of Unicode is its unimplementability. That's a giant red flag that something went horribly wrong.
- lifthrasiir 3y agoBy this measure pretty much every common programming language is unimplementable and should be never advocated. People have implemented Unicode for decades. Only a few implemented "full" support (in some sense), but that's same for pretty much everything complex enough. Can you actually measure one meter [1] according to the International System of Units? [1] Or one foot. I remind you that all current standardized definition of "foot" is based on the SI.
- WalterBright 3y ago> People have implemented Unicode for decades Small subsets. No fonts remotely support all the characters. Prog languages do subsets.
- lifthrasiir 3y agoAs others pointed out, no single font is required to support all available Unicode characters. In fact it is just close to impossible anyway, because it is extremely hard to harmoize all typographic conventions across all scripts. Unicode mostly does the next best thing: it gives a uniform framework to handle multilingual texts without knowing specifics. Even though your font can't render "월터 브라이트" programs can normalize, compare, search and segment it just like it does with the text "Walter Bright". But if you like a pedantry, Noto fonts are close to be complete. They have to be used as a collection due to the OpenType limitation, but otherwise designed in unison. Most remaining characters are rarer CJK unified ideographs [1] but it is more about the priority issue; there are several fonts that cover the entire CJK unified ideographs as well. [1] https://notofonts.github.io/overview/ https://notofonts.github.io/overview/
- earthboundkid 3y agoEmoji are “fun” but it’s got to stop. These characters are going to be used until civilization collapses, so we can’t just keep being like “lol, let’s add a food that is only eaten in some regions and has been around for less than 100 years.”
- lifthrasiir 3y agoNot different from the fact that we have useless CC0/CC1 characters somewhere in Unicode. As long as bearable (i.e. not too many), having rare characters is okay.
- masklinn 3y agoEmojis are also great from a technical standpoint: because they are desirable to users the unicode consortium can (and has) leverage them to encourage text rendering engines to properly implement features necessary for various languages e.g. astral codepoints, ligatures, grapheme clusters, ...
- gumby 3y agoThey aren’t useless because they support file idempotency. Let’s say “important while almost entirely useless” and call it a day?
- zoky 3y agoBut I use the khlav kalash emoji all the time!
- WalterBright 3y agoDisplaying emoji is the job of the rendering engine, not the encoding.
- lifthrasiir 3y agoJapanese telcos disagreed. (Seriously. We have emojis in Unicode because of interoperabilities with them.)
- msla 3y ago> multiple encodings for the same code point I'm fairly sure a good fraction of these are due to the desire for compatibility: PDF: https://www.unicode.org/versions/Unicode15.0.0/ch02.pdf#G11062 https://www.unicode.org/versions/Unicode15.0.0/ch02.pdf#G110... > Conceptually, compatibility characters are characters that would not have been encoded in the Unicode Standard except for compatibility and round-trip convertibility with other standards. Such standards include international, national, and vendor character encoding standards. For the most part, these are widely used standards that pre-dated Unicode, but because continued interoperability with new standards and data sources is one of the primary design goals of the Unicode Standard, additional compatibility characters are added as the situation warrants. Round-trip convertibility is important for interoperability, which is important for getting the standard accepted. A data encoding standard that causes data loss is not acceptable, and trying to impose such on the world would have rightfully failed.
- WalterBright 3y ago> new standards ... added as the situation warrants Never ending madness. > impose such on the world The imposing is inflicted by the Unicode standard on everyone in the world who have no use for never ending new invented encodings for the same thing.
- derefr 3y ago> Never ending madness. [Unicode] didn't start the fire; it was always burning, since the world's been turning. (I.e. the "never-ending madness" is in the continual desire of human cultures to invent new characters with unique semantics and then have them faithfully digitally encoded in a way that carries those semantics. Being upset that this makes the management of a universal text encoding a never-ending job, is a bit like being upset that the Library of Congress always needs reorganizing and expanding because it keeps getting new books.)
- WalterBright 3y agoIt's different in that nobody expects anyone to implement their own copy of the LoC. This constant expansion of the character set puts an enormous burden on a very large collection of developers. The inevitable happens - they just don't bother - making full Unicode support a pipe dream.
- TacticalCoder 3y agoAnd on top of that people decided to start supporting Unicode everywhere, like in domain names, source code, filesystems, etc. So now we can have poop and animal symbols in filenames. And left-to-right overrides and countless homograph attacks. It's just sad.
- gumby 3y agoI’m going to defend this set of complaints: > “multiple encodings for the same code point”, “multiple code points for the same glyph”, and “an incomprehensible document describing all this nonsense” One important factor for adoption was round trip idempotency. Unicode doesn’t simply encode every glyph but incorporates as many existing character sets as possible. (Of course “possible” is modulated by the fact there are/were competing encodings for the same language). If you use a unicode-aware grep and use it to search for ‘.’, you want the result to be the same as `cat`, rather than having the result re-coded, objcopy-like. Sure, this is less of a problem in 2023, but still a major problem that will continue indefinitely due to legacy files, legacy hardware, and small-resource devices. So there is no way in unicode to tell by inspection if ‘A’ is Latin or Greek..which is true also on the printed page. Likewise the writing of different languages don’t merely have different appearance, but different semantics for the visually same shape. In some languages (e.g. French), diacritics are largely semantically silent: you can understand a sentence with them stripped out, though they can simplify reading in some cases by disambiguating some otherwise homonyms. In other languages (e.g. the German umlaut) they are simply contractions — fossils of medieval handwriting. Yet again, in Swedish they can be optional, like the acute accent or form completely different letters like ä. Capturing this is complicated and even discussing it requires, yes, an almost incomprehensible document. And it has all been implemented but using those libraries is hard, because language processing is hard. In general I/O is hard. Im not sure what you mean about “semantic meaning” but I do agree, mostly, with your other complaints.
- WalterBright 3y ago> semantic meaning There's one code point for the letter `a` and another code point for the mathematical symbol `a`, despite being visually identical. I.e. the code points have acquired semantic meaning. Spy agencies use this so different versions of a document look identical when printed. Thus, they can use this to track down the mole who leaked a sensitive document.