4 ms·

How to say you don't know what Unicode is for without saying it. 1, 2, 4, 5, 6, and, unfortunately, 8, all fall under "ability to encode written text from all

by p_l 2y ago

How to say you don't know what Unicode is for without saying it. 1, 2, 4, 5, 6, and, unfortunately, 8, all fall under "ability to encode written text from all human languages". And that includes historical. Some of the issues (5 & 6) are due semantic difference even if the resulting glyph looks the same. Unfortunately you can't expect programmers to understand pesky little thing like languages having different writing, so you end up with normalisation to handle the fact that one system sent "a + ogonek accent" and another (properly) sent "a with ogonek" (these print the same but are semantically different!), and now you need to figure out normalisation in order to be able to compare strings. 7. just like 8 are down to proposal of specific new forms of writing to add to Unicode. Elvish had one since 1997 but only now got a tentative "we will talk about it". Klingon, which is IIRC more complete language including native speakers (...weird things happened sometimes) does not have outside of private use area. Emojis were added because they were used with incompatible encodings first, even before unicode happened, and without including something like SIXEL into unicode they were unrepresentable (and with SIXEL would lose semantic information)

news_to_me 2y ago
> "a + ogonek accent" and another (properly) sent "a with ogonek" (these print the same but are semantically different!) How can these possibly be semantically different? Isn’t the point of combining characters to create semantic characters that are the combination of those parts?
- p_l 2y ago
  There's a semantic difference between "accented letter" and "different letter that happens to visually look like another language's accented letter". "Ą" in polish is not "A" with some accent. And the idea behind unicode was to preserve human written text, including keeping track of things like "this is letter A1 with an accent, but this is letter A2 that looks visually similar to A1 with accent but is different semantically". Of course then worries about code page size resulted in the stupidity of Han unification, so Unicode is a bit broken.
  - Dylan16807 2y ago
    Unless there's some nuance I'm missing, I think you're reading too much into the word "accent". Especially because the codepoint is actually called "Combining Ogonek". And for anyone writing in Cyrillic, it's actually more accurate to use the combining form, even as its own letter, because the only precomposed form technically uses a latin A. But my main point is that I do not think there is supposed to be any semantic difference in Unicode based on whether you use precomposed or decomposed code points.
  - eviks 2y ago
    But it is precisely "a with some accent", you just have two ways to encode it for
    - p_l 2y ago
      "Ą" is a separate letter in polish alfabet, not an accented variant of "A". There are writing systems where combining accents are used to represent just variation on a letter. Use of combining characters for "Ą" (and "Ć" and "Ł" and many other so-called "polish letters") is, at best, a historical artefact of trying to write them in deficient encodings.
      - eviks 2y ago
        It doesn't matter that it's a separate letter in an alphabet, you're denying the obvious - it IS an accented (or ogonek'ed) variant of A, and you can achieve this in Unicode in 2 ways: having one id for a precomposed variant and composing the variant from two ids. There is no semantic difference, just an encoding one, the end result looks the same and means the same thing (well, to a point, it still depends on the context - like what language you mean - but within the same context it's the same thing and there are even Unicode rules to treat it the same like in search etc.) And precomposed is just the same historical deficiency - you could've just as well designed a more compact encoding with no precomposed letters, only combinations
        pests 2y ago
WalterBright 2y ago
> How to say you don't know what Unicode is for without saying it. I know what its original mission was, which was a character set. It's been mangled beyond recognition - by including semantic information which is in the purview of context, and presentation information (italics, fonts) which is in the purview of markup languages and layout information (backwards text) which is also in the purview of markup. > you can't expect programmers to understand pesky little thing like languages having different writing, But you're requiring programmers to understand all the complicated normalization rules? Normalization is a totally unnecessary feature. Just use the normalized code points. Done. > these print the same but are semantically different! Think about what this means. How ever did people manage to read and understand printed books? The semantic meaning comes from the context, not the glyph. For example, I can use `a` to mean the `ahh` sound, or the `ayy` sound, or mean a variable in algebra. How can I know which? The context. It is totally impossible to add every meaning a glyph has. > would lose semantic information Unicode is supposed to be a character set. That's it. Characters do not have semantic information without context. Oh, and here's some drawkcab text I wrote without any help from Unicode at all. I had to add some code into the D compiler to reject Unicode text direction "characters" because they can be used to invisibly insert malware into ordinary code. Adding toy "languages" should be for people having fun, not Unicode.

by p_l 2y ago

news_to_me 2y ago

> "a + ogonek accent" and another (properly) sent "a with ogonek" (these print the same but are semantically different!) How can these possibly be semantically different? Isn’t the point of combining characters to create semantic characters that are the combination of those parts?

p_l 2y ago
There's a semantic difference between "accented letter" and "different letter that happens to visually look like another language's accented letter". "Ą" in polish is not "A" with some accent. And the idea behind unicode was to preserve human written text, including keeping track of things like "this is letter A1 with an accent, but this is letter A2 that looks visually similar to A1 with accent but is different semantically". Of course then worries about code page size resulted in the stupidity of Han unification, so Unicode is a bit broken.
- Dylan16807 2y ago
  Unless there's some nuance I'm missing, I think you're reading too much into the word "accent". Especially because the codepoint is actually called "Combining Ogonek". And for anyone writing in Cyrillic, it's actually more accurate to use the combining form, even as its own letter, because the only precomposed form technically uses a latin A. But my main point is that I do not think there is supposed to be any semantic difference in Unicode based on whether you use precomposed or decomposed code points.
- eviks 2y ago
  But it is precisely "a with some accent", you just have two ways to encode it for
  - p_l 2y ago
    "Ą" is a separate letter in polish alfabet, not an accented variant of "A". There are writing systems where combining accents are used to represent just variation on a letter. Use of combining characters for "Ą" (and "Ć" and "Ł" and many other so-called "polish letters") is, at best, a historical artefact of trying to write them in deficient encodings.
    - eviks 2y ago
      It doesn't matter that it's a separate letter in an alphabet, you're denying the obvious - it IS an accented (or ogonek'ed) variant of A, and you can achieve this in Unicode in 2 ways: having one id for a precomposed variant and composing the variant from two ids. There is no semantic difference, just an encoding one, the end result looks the same and means the same thing (well, to a point, it still depends on the context - like what language you mean - but within the same context it's the same thing and there are even Unicode rules to treat it the same like in search etc.) And precomposed is just the same historical deficiency - you could've just as well designed a more compact encoding with no precomposed letters, only combinations
      - pests 2y ago

WalterBright 2y ago

> How to say you don't know what Unicode is for without saying it. I know what its original mission was, which was a character set. It's been mangled beyond recognition - by including semantic information which is in the purview of context, and presentation information (italics, fonts) which is in the purview of markup languages and layout information (backwards text) which is also in the purview of markup. > you can't expect programmers to understand pesky little thing like languages having different writing, But you're requiring programmers to understand all the complicated normalization rules? Normalization is a totally unnecessary feature. Just use the normalized code points. Done. > these print the same but are semantically different! Think about what this means. How ever did people manage to read and understand printed books? The semantic meaning comes from the context, not the glyph. For example, I can use `a` to mean the `ahh` sound, or the `ayy` sound, or mean a variable in algebra. How can I know which? The context. It is totally impossible to add every meaning a glyph has. > would lose semantic information Unicode is supposed to be a character set. That's it. Characters do not have semantic information without context. Oh, and here's some drawkcab text I wrote without any help from Unicode at all. I had to add some code into the D compiler to reject Unicode text direction "characters" because they can be used to invisibly insert malware into ordinary code. Adding toy "languages" should be for people having fun, not Unicode.