12 ms·

> P.S. Having multiple Unicode values that exhibit identically when displayed are a huge veer-into-the-ditch mistake. I.e. the notion that code points should ha

> P.S. Having multiple Unicode values that exhibit identically when displayed are a huge veer-into-the-ditch mistake. I.e. the notion that code points should have semantic value is simply wrong. Should a cyrillic `а` and a latin `a` have the same code point? If they did then there's no consistent way to group those two alphabets, one or the other would end up with letters outside it's main grouping. And what happens if the shapes of those two characters divege over time? Do you have a breaking change to introduce a new code point for one of the languages, or do you just make it impossible to have a font that can be used for both latin and cyrillic at the same time? This isn't just a theoretical problem, by the way. There are characters in Chinese and Japanese that share a code point but the shapes of the characters aren't the same in the two languages.

WalterBright 3y ago
> Should a cyrillic `а` and a latin `a` have the same code point? Yes. Consider a book. Can you tell if it's a cyrillic or a latin `a`? Of course you can, because of the context. Unicode is about visible text, having hidden semantic meaning makes it something else. Besides, 'a' can have all kinds of semantic meanings - all depending on the context in which they are used. There is no way to encode all this into Unicode. > If they did then there's no consistent way to group those two alphabets Doesn't matter. > And what happens if the shapes of those two characters divege over time? If it actually looks different, then it becomes a different code point. > There are characters in Chinese and Japanese that share a code point but the shapes of the characters aren't the same in the two languages More evidence that the Unicode committee lost its way. Think of it this way. Printing Unicode text on a piece of paper, and then OCRing it back into Unicode, should be a lossless operation. Or another way - anyone should be able to tell what the code point value is by looking at the visual representation of it.
- boolemancer 3y ago
  > If it actually looks different, then it becomes a different code point. Which language gets the original code point and which one needs to change? And what happens to all of the existing text written with the original code point?
  - account42 3y ago
    Obviously we are stuck with Unicode as-is now, warts and all. But that doesn't mean that the original design was the right one.
- dullcrisp 3y ago
  How about an l and an I? Or a closed "a" and one with the little handle? A zero with a stroke, a zero without a stroke, and an O? I can see where you're coming from, but deduplicating every glyph from every culture based on which do or don't generally look the same when printed sounds like a tall order. And if all you want to record is the shape, you can use a PNG with OCR and bypass Unicode entirely.
  - WalterBright 3y ago
    > How about an l and an I? Or a closed "a" and one with the little handle? A zero with a stroke, a zero without a stroke, and an O? Those are font differences, not character differences. (Unicode has also failed by adding in some fonts. The nuttiness never ends.)
    - dullcrisp 3y ago
      But just as the “a” in alphabet and the «а» in азбука are only distinguishable by context, so too the “O” in SOS and the 0 in 90210 are often only distinguishable by context. I don’t see how you propose to determine which glyphs should be the same and which should be different in your system.
      - WalterBright 3y ago
        > so too the “O” in SOS and the 0 in 90210 are often only distinguishable by context Only if it's a bad font.
- tomjakubowski 3y ago
  How does your scheme handle Turkish "Dotless I"? Should there be different codepoints for serif and sans-serif versions of the same letter? After all, in some typefaces, "uppercase I" and "lowercase L" look just about the same.
  - WalterBright 3y ago
    I think you mean fonts. No, Unicode should not encode fonts. Nor should it encode italic, boldface, underline, line out, reverse video, point size, superscript, subscript, or colored. Those are all style attributes, best applied with a style sheet, not a code point.
    - ben_w 3y ago
      > superscript, subscript Very strongly disagree. Copy-pasting 10<sup>6</sup> turns into 106 rather than 10⁶.
      - account42 3y ago
        Depends where you copy and paste it. Encoding superscript variants is a fools errand because you really need an exponential number of variants for ALL characters to fully support possible combinations of styles. The only reason (a very limited selection of) superscript variants are in Unicode at all is for backwards compatibility with mistakes made by earlier encodings.

WalterBright 3y ago

> Should a cyrillic `а` and a latin `a` have the same code point? Yes. Consider a book. Can you tell if it's a cyrillic or a latin `a`? Of course you can, because of the context. Unicode is about visible text, having hidden semantic meaning makes it something else. Besides, 'a' can have all kinds of semantic meanings - all depending on the context in which they are used. There is no way to encode all this into Unicode. > If they did then there's no consistent way to group those two alphabets Doesn't matter. > And what happens if the shapes of those two characters divege over time? If it actually looks different, then it becomes a different code point. > There are characters in Chinese and Japanese that share a code point but the shapes of the characters aren't the same in the two languages More evidence that the Unicode committee lost its way. Think of it this way. Printing Unicode text on a piece of paper, and then OCRing it back into Unicode, should be a lossless operation. Or another way - anyone should be able to tell what the code point value is by looking at the visual representation of it.

boolemancer 3y ago
> If it actually looks different, then it becomes a different code point. Which language gets the original code point and which one needs to change? And what happens to all of the existing text written with the original code point?
- account42 3y ago
  Obviously we are stuck with Unicode as-is now, warts and all. But that doesn't mean that the original design was the right one.
dullcrisp 3y ago
How about an l and an I? Or a closed "a" and one with the little handle? A zero with a stroke, a zero without a stroke, and an O? I can see where you're coming from, but deduplicating every glyph from every culture based on which do or don't generally look the same when printed sounds like a tall order. And if all you want to record is the shape, you can use a PNG with OCR and bypass Unicode entirely.
- WalterBright 3y ago
  > How about an l and an I? Or a closed "a" and one with the little handle? A zero with a stroke, a zero without a stroke, and an O? Those are font differences, not character differences. (Unicode has also failed by adding in some fonts. The nuttiness never ends.)
  - dullcrisp 3y ago
    But just as the “a” in alphabet and the «а» in азбука are only distinguishable by context, so too the “O” in SOS and the 0 in 90210 are often only distinguishable by context. I don’t see how you propose to determine which glyphs should be the same and which should be different in your system.
    - WalterBright 3y ago
      > so too the “O” in SOS and the 0 in 90210 are often only distinguishable by context Only if it's a bad font.
tomjakubowski 3y ago
How does your scheme handle Turkish "Dotless I"? Should there be different codepoints for serif and sans-serif versions of the same letter? After all, in some typefaces, "uppercase I" and "lowercase L" look just about the same.
- WalterBright 3y ago
  I think you mean fonts. No, Unicode should not encode fonts. Nor should it encode italic, boldface, underline, line out, reverse video, point size, superscript, subscript, or colored. Those are all style attributes, best applied with a style sheet, not a code point.
  - ben_w 3y ago
    > superscript, subscript Very strongly disagree. Copy-pasting 10<sup>6</sup> turns into 106 rather than 10⁶.
    - account42 3y ago
      Depends where you copy and paste it. Encoding superscript variants is a fools errand because you really need an exponential number of variants for ALL characters to fully support possible combinations of styles. The only reason (a very limited selection of) superscript variants are in Unicode at all is for backwards compatibility with mistakes made by earlier encodings.