3 ms·

> Semantics: there's a code point for the letter 'a' and another code point for the mathematical symbol 'a', although the glyphs are identical. Unicode points s

by derefr 3y ago

> Semantics: there's a code point for the letter 'a' and another code point for the mathematical symbol 'a', although the glyphs are identical. Unicode points should not have semantic meaning - semantic meaning comes from the context in which the points are used. Just the opposite. A codepoint looks different in different fonts, and so things that are "identical" in one font won't be "identical" in another font. Only the semantics are the same. Printed "A" and cursive "A" are both, semantically, "latin capital A". Those semantics are very concrete and specific: they include its sort order, its conversion re: case-conversion, its joining/splitting behavior, its categoric matching in regexps, etc. But the codepoint "latin capital A" expresses no preference over whether it should be rendered as a printed/roman A or a cursive A. Codepoints don't define glyphs; they only define semantics! The glyphs associated with codepoints are conventional, following the codepoint's description! Further, codepoints tell accessibility technologies and other machine-agents what a codepoint "is." Greek letter π is text-to-speech'ed as a letter, and together with other greek letters, as a greek word. Mathematical symbol 𝜋 is text-to-speech'ed as the word "pi."

WalterBright 3y ago
> Codepoints don't define glyphs; they only define semantics! And this is the fundamental failure of Unicode post 16 bit code points. Unicode has set itself an absurdly impossible task.
- derefr 3y ago
  This is how every character encoding ever has always worked. Text on computers — text over telegraph lines, even! — has always, implicitly, been a set of abstract standardized semiotic codels, with effectively-arbitrary per-OS/per-machine/per-font-family/per-style/per-size(!) graphical renditions. > Unicode has set itself an absurdly impossible task. Again: "Unicode" is not the problem here. "Human language" is the "absurdly impossible task", and has been, from ~40,000BCE. Unicode is just a formalist lens through which the scope of the task is made legible. (And I think you're being a slight bit scope-insensitive here. You should expect a project like "encoding all of human language" to be something that takes hundreds or thousands of years to get right. Unicode isn't a project, it's a megaproject, something that'll still be evolving fifteen human generations from now. But humanity as a whole has built many successful megaprojects! In the only-50-years-old realm of software engineering, we don't tend to rate things of this scope and scale as "possible" or "practical" or "tenable" — but Unicode fundamentally isn't a software engineering project; it's a language engineering project, like the project of the Académie Française, or of the creation of Simplified Chinese. Language engineering is, by definition, a project of indefinite length.)

by derefr 3y ago

WalterBright 3y ago

> Codepoints don't define glyphs; they only define semantics! And this is the fundamental failure of Unicode post 16 bit code points. Unicode has set itself an absurdly impossible task.

derefr 3y ago
This is how every character encoding ever has always worked. Text on computers — text over telegraph lines, even! — has always, implicitly, been a set of abstract standardized semiotic codels, with effectively-arbitrary per-OS/per-machine/per-font-family/per-style/per-size(!) graphical renditions. > Unicode has set itself an absurdly impossible task. Again: "Unicode" is not the problem here. "Human language" is the "absurdly impossible task", and has been, from ~40,000BCE. Unicode is just a formalist lens through which the scope of the task is made legible. (And I think you're being a slight bit scope-insensitive here. You should expect a project like "encoding all of human language" to be something that takes hundreds or thousands of years to get right. Unicode isn't a project, it's a megaproject, something that'll still be evolving fifteen human generations from now. But humanity as a whole has built many successful megaprojects! In the only-50-years-old realm of software engineering, we don't tend to rate things of this scope and scale as "possible" or "practical" or "tenable" — but Unicode fundamentally isn't a software engineering project; it's a language engineering project, like the project of the Académie Française, or of the creation of Simplified Chinese. Language engineering is, by definition, a project of indefinite length.)