3 ms·

The only fault of Unicode is 4 (and we have eventually solved it anyway). Everything else is a natural complication from human writing systems.

WalterBright 3y ago
Fonts, for example. That is a style, and should be set by the rendering instructions (like CSS) not the code point. Semantics: there's a code point for the letter 'a' and another code point for the mathematical symbol 'a', although the glyphs are identical. Unicode points should not have semantic meaning - semantic meaning comes from the context in which the points are used. You know, like the text in a printed book. And so on, for the other bullet points.
- lifthrasiir 3y ago
  I have already explained why mathematical alphanumeric symbols exist in the first place [1]. They are an extremely small portion of Unicode; you need way more examples. By the way, if we indeed decide to encode characters according to glyphs, we would have to distinguish a double-storey `a` from a single-storey `a` (among others). They have clearly different glyphs, so why shouldn't we? Think about that. [1] https://news.ycombinator.com/item?id=28294032 https://news.ycombinator.com/item?id=28294032
  - WalterBright 3y ago
    > I have already explained But somehow mathematical books have done just fine without any need for such encodings. Meaning in written language is inferred from the context, not the symbol. This is because meanings are infinite in variety, and symbols are finite. > Think about that I did. That's what fonts are for. Fonts come in endless variety, and so cannot be encoded into Unicode. They are an orthogonal property of rendered text. And so is formatting - it's an orthogonal property.
    - lifthrasiir 3y ago
      > But somehow mathematical books have done just fine without any need for such encodings. Printed books have no reason to think about encodings. In fact, they can use one-off metal types (or images nowadays) for unusual characters. In return printed books cannot be easily searched if the entry doesn't appear in prepared indices. > Fonts come in endless variety, and so cannot be encoded into Unicode. But some fonts convey semantics. For example a normal text does not distinguish two variants of `a`, but IPA does, so Unicode does encode a single-storey `ɑ` separately and unify a double-storey `a` to the normal lowercase `a`. (p. 299 of the Unicode standard 15.0) Semantics is arguably the only way to judge those cases. If fonts do not matter but glyphs do matter, how would know a particular glyph should be identically encoded to other glyph or not? I think you agree that a double-storey `a` and a single-storey `a` are same characters with different fonts. Semantics is how we decide the "same characters" part!
      - WalterBright 3y ago
        > Printed books have no reason to think about encodings Exactly! It's the visual that matters. If you cannot print Unicode text on a sheet of paper and know its meaning, then Unicode has gone into a ditch. > printed books cannot be easily searched if the entry doesn't appear in prepared indices Are you going to invent a new encoding for every distinct use of the letter 'a'? That's not remotely possible. > But some fonts convey semantics People invent new fonts every day. Hell, I've invented fonts! Your mission to encode them all into Unicode is an instantly hopeless task. Again, semantics are divined from context. I know a mathematical 'i' from the english letter 'i' and the french letter 'i' and the bullet point 'i' and the electric current 'i' by context, and so do you and everyone else. I even use the letter 'A' to represent an Army in my computer game Empire. Nobody has a problem divining the meaning. (and B for Battleship, S for Submarine , etc.)
        lifthrasiir 3y ago
        > It's the visual that matters. I expect a good search function for my electronic book! > Again, semantics are divined from context. I gave an explicit counterexample. You can't distinguish an IPA `ɑ` (open back unrounded vowel) from an IPA `a` (open front unrounded vowel) with just a context---[hɑt] and [hat] are different pronunciations. So they have to be differently encoded, right? And due to its history, either the "normal" `a` glyph will exactly look like one of them, depending on the font. So we can't have perfectly distinct glyphs for distinct code points anyway. Maybe we need a better rationale, something like semantics...
- derefr 3y ago
  > Semantics: there's a code point for the letter 'a' and another code point for the mathematical symbol 'a', although the glyphs are identical. Unicode points should not have semantic meaning - semantic meaning comes from the context in which the points are used. Just the opposite. A codepoint looks different in different fonts, and so things that are "identical" in one font won't be "identical" in another font. Only the semantics are the same. Printed "A" and cursive "A" are both, semantically, "latin capital A". Those semantics are very concrete and specific: they include its sort order, its conversion re: case-conversion, its joining/splitting behavior, its categoric matching in regexps, etc. But the codepoint "latin capital A" expresses no preference over whether it should be rendered as a printed/roman A or a cursive A. Codepoints don't define glyphs; they only define semantics! The glyphs associated with codepoints are conventional, following the codepoint's description! Further, codepoints tell accessibility technologies and other machine-agents what a codepoint "is." Greek letter π is text-to-speech'ed as a letter, and together with other greek letters, as a greek word. Mathematical symbol 𝜋 is text-to-speech'ed as the word "pi."
  - WalterBright 3y ago
    > Codepoints don't define glyphs; they only define semantics! And this is the fundamental failure of Unicode post 16 bit code points. Unicode has set itself an absurdly impossible task.
    - derefr 3y ago
      This is how every character encoding ever has always worked. Text on computers — text over telegraph lines, even! — has always, implicitly, been a set of abstract standardized semiotic codels, with effectively-arbitrary per-OS/per-machine/per-font-family/per-style/per-size(!) graphical renditions. > Unicode has set itself an absurdly impossible task. Again: "Unicode" is not the problem here. "Human language" is the "absurdly impossible task", and has been, from ~40,000BCE. Unicode is just a formalist lens through which the scope of the task is made legible. (And I think you're being a slight bit scope-insensitive here. You should expect a project like "encoding all of human language" to be something that takes hundreds or thousands of years to get right. Unicode isn't a project, it's a megaproject, something that'll still be evolving fifteen human generations from now. But humanity as a whole has built many successful megaprojects! In the only-50-years-old realm of software engineering, we don't tend to rate things of this scope and scale as "possible" or "practical" or "tenable" — but Unicode fundamentally isn't a software engineering project; it's a language engineering project, like the project of the Académie Française, or of the creation of Simplified Chinese. Language engineering is, by definition, a project of indefinite length.)
WalterBright 3y ago
What should give pause in advocacy for the tarpit of Unicode is its unimplementability. That's a giant red flag that something went horribly wrong.
- lifthrasiir 3y ago
  By this measure pretty much every common programming language is unimplementable and should be never advocated. People have implemented Unicode for decades. Only a few implemented "full" support (in some sense), but that's same for pretty much everything complex enough. Can you actually measure one meter [1] according to the International System of Units? [1] Or one foot. I remind you that all current standardized definition of "foot" is based on the SI.
  - WalterBright 3y ago
    > People have implemented Unicode for decades Small subsets. No fonts remotely support all the characters. Prog languages do subsets.
    - lifthrasiir 3y ago
      As others pointed out, no single font is required to support all available Unicode characters. In fact it is just close to impossible anyway, because it is extremely hard to harmoize all typographic conventions across all scripts. Unicode mostly does the next best thing: it gives a uniform framework to handle multilingual texts without knowing specifics. Even though your font can't render "월터 브라이트" programs can normalize, compare, search and segment it just like it does with the text "Walter Bright". But if you like a pedantry, Noto fonts are close to be complete. They have to be used as a collection due to the OpenType limitation, but otherwise designed in unison. Most remaining characters are rarer CJK unified ideographs [1] but it is more about the priority issue; there are several fonts that cover the entire CJK unified ideographs as well. [1] https://notofonts.github.io/overview/ https://notofonts.github.io/overview/

WalterBright 3y ago

Fonts, for example. That is a style, and should be set by the rendering instructions (like CSS) not the code point. Semantics: there's a code point for the letter 'a' and another code point for the mathematical symbol 'a', although the glyphs are identical. Unicode points should not have semantic meaning - semantic meaning comes from the context in which the points are used. You know, like the text in a printed book. And so on, for the other bullet points.

lifthrasiir 3y ago
I have already explained why mathematical alphanumeric symbols exist in the first place [1]. They are an extremely small portion of Unicode; you need way more examples. By the way, if we indeed decide to encode characters according to glyphs, we would have to distinguish a double-storey `a` from a single-storey `a` (among others). They have clearly different glyphs, so why shouldn't we? Think about that. [1] https://news.ycombinator.com/item?id=28294032 https://news.ycombinator.com/item?id=28294032
- WalterBright 3y ago
  > I have already explained But somehow mathematical books have done just fine without any need for such encodings. Meaning in written language is inferred from the context, not the symbol. This is because meanings are infinite in variety, and symbols are finite. > Think about that I did. That's what fonts are for. Fonts come in endless variety, and so cannot be encoded into Unicode. They are an orthogonal property of rendered text. And so is formatting - it's an orthogonal property.
  - lifthrasiir 3y ago
    > But somehow mathematical books have done just fine without any need for such encodings. Printed books have no reason to think about encodings. In fact, they can use one-off metal types (or images nowadays) for unusual characters. In return printed books cannot be easily searched if the entry doesn't appear in prepared indices. > Fonts come in endless variety, and so cannot be encoded into Unicode. But some fonts convey semantics. For example a normal text does not distinguish two variants of `a`, but IPA does, so Unicode does encode a single-storey `ɑ` separately and unify a double-storey `a` to the normal lowercase `a`. (p. 299 of the Unicode standard 15.0) Semantics is arguably the only way to judge those cases. If fonts do not matter but glyphs do matter, how would know a particular glyph should be identically encoded to other glyph or not? I think you agree that a double-storey `a` and a single-storey `a` are same characters with different fonts. Semantics is how we decide the "same characters" part!
    - WalterBright 3y ago
      > Printed books have no reason to think about encodings Exactly! It's the visual that matters. If you cannot print Unicode text on a sheet of paper and know its meaning, then Unicode has gone into a ditch. > printed books cannot be easily searched if the entry doesn't appear in prepared indices Are you going to invent a new encoding for every distinct use of the letter 'a'? That's not remotely possible. > But some fonts convey semantics People invent new fonts every day. Hell, I've invented fonts! Your mission to encode them all into Unicode is an instantly hopeless task. Again, semantics are divined from context. I know a mathematical 'i' from the english letter 'i' and the french letter 'i' and the bullet point 'i' and the electric current 'i' by context, and so do you and everyone else. I even use the letter 'A' to represent an Army in my computer game Empire. Nobody has a problem divining the meaning. (and B for Battleship, S for Submarine , etc.)
      - lifthrasiir 3y ago
        > It's the visual that matters. I expect a good search function for my electronic book! > Again, semantics are divined from context. I gave an explicit counterexample. You can't distinguish an IPA `ɑ` (open back unrounded vowel) from an IPA `a` (open front unrounded vowel) with just a context---[hɑt] and [hat] are different pronunciations. So they have to be differently encoded, right? And due to its history, either the "normal" `a` glyph will exactly look like one of them, depending on the font. So we can't have perfectly distinct glyphs for distinct code points anyway. Maybe we need a better rationale, something like semantics...
derefr 3y ago
> Semantics: there's a code point for the letter 'a' and another code point for the mathematical symbol 'a', although the glyphs are identical. Unicode points should not have semantic meaning - semantic meaning comes from the context in which the points are used. Just the opposite. A codepoint looks different in different fonts, and so things that are "identical" in one font won't be "identical" in another font. Only the semantics are the same. Printed "A" and cursive "A" are both, semantically, "latin capital A". Those semantics are very concrete and specific: they include its sort order, its conversion re: case-conversion, its joining/splitting behavior, its categoric matching in regexps, etc. But the codepoint "latin capital A" expresses no preference over whether it should be rendered as a printed/roman A or a cursive A. Codepoints don't define glyphs; they only define semantics! The glyphs associated with codepoints are conventional, following the codepoint's description! Further, codepoints tell accessibility technologies and other machine-agents what a codepoint "is." Greek letter π is text-to-speech'ed as a letter, and together with other greek letters, as a greek word. Mathematical symbol 𝜋 is text-to-speech'ed as the word "pi."
- WalterBright 3y ago
  > Codepoints don't define glyphs; they only define semantics! And this is the fundamental failure of Unicode post 16 bit code points. Unicode has set itself an absurdly impossible task.
  - derefr 3y ago
    This is how every character encoding ever has always worked. Text on computers — text over telegraph lines, even! — has always, implicitly, been a set of abstract standardized semiotic codels, with effectively-arbitrary per-OS/per-machine/per-font-family/per-style/per-size(!) graphical renditions. > Unicode has set itself an absurdly impossible task. Again: "Unicode" is not the problem here. "Human language" is the "absurdly impossible task", and has been, from ~40,000BCE. Unicode is just a formalist lens through which the scope of the task is made legible. (And I think you're being a slight bit scope-insensitive here. You should expect a project like "encoding all of human language" to be something that takes hundreds or thousands of years to get right. Unicode isn't a project, it's a megaproject, something that'll still be evolving fifteen human generations from now. But humanity as a whole has built many successful megaprojects! In the only-50-years-old realm of software engineering, we don't tend to rate things of this scope and scale as "possible" or "practical" or "tenable" — but Unicode fundamentally isn't a software engineering project; it's a language engineering project, like the project of the Académie Française, or of the creation of Simplified Chinese. Language engineering is, by definition, a project of indefinite length.)

WalterBright 3y ago

What should give pause in advocacy for the tarpit of Unicode is its unimplementability. That's a giant red flag that something went horribly wrong.

lifthrasiir 3y ago
By this measure pretty much every common programming language is unimplementable and should be never advocated. People have implemented Unicode for decades. Only a few implemented "full" support (in some sense), but that's same for pretty much everything complex enough. Can you actually measure one meter [1] according to the International System of Units? [1] Or one foot. I remind you that all current standardized definition of "foot" is based on the SI.
- WalterBright 3y ago
  > People have implemented Unicode for decades Small subsets. No fonts remotely support all the characters. Prog languages do subsets.
  - lifthrasiir 3y ago
    As others pointed out, no single font is required to support all available Unicode characters. In fact it is just close to impossible anyway, because it is extremely hard to harmoize all typographic conventions across all scripts. Unicode mostly does the next best thing: it gives a uniform framework to handle multilingual texts without knowing specifics. Even though your font can't render "월터 브라이트" programs can normalize, compare, search and segment it just like it does with the text "Walter Bright". But if you like a pedantry, Noto fonts are close to be complete. They have to be used as a collection due to the OpenType limitation, but otherwise designed in unison. Most remaining characters are rarer CJK unified ideographs [1] but it is more about the priority issue; there are several fonts that cover the entire CJK unified ideographs as well. [1] https://notofonts.github.io/overview/ https://notofonts.github.io/overview/