7 ms·
So, one might wonder why these homo-graphs have different code points. After all the French A and the English A share the same code point. It's really difficul
by wodenokoto 11y ago
So, one might wonder why these homo-graphs have different code points. After all the French A and the English A share the same code point.
It's really difficult to do the right thing here. If Greek question marks share code point with semi-colon, it obstructs search and replace for question marks.
Subtle differences in how Japanese and Chinese are written has led to differently written characters sharing the same code point. It's nice that you can easily look up most Japanese characters in a Chinese dictionary and see how they are used in China, but it has become frustratingly hard to get subtleties in their written form right. The Chinese version may have the line strike through another line, while the Japanese only has it touching.
I honestly don't know how to go about posting how to same code points have different written forms!
But it seems like it would be nice if code editors warned about text outside ascii. You usually only want that in strings and comments.
- ant6n 11y agonow that we gave up on ucs-2, we could re-encode those overloaded Japenese/Chinese characters as separate Japenese and Chinese characters on astral planes like the supplemental multilingual one.
- rspeer 11y agoThey're doing that. That's what a lot of plane 2 is.
- lazyjones 11y ago> It's really difficult to do the right thing here. If Greek question marks share code point with semi-colon, it obstructs search and replace for question marks. Context is the key here. Greek text doesn't use the semicolon for other purposes and searching/replacing such single characters in source code is a terrible idea anyway (think comments, string literals...). So what is the prohibitive failure scenario here? Indistinguishable (for humans) characters with different code points were a stupid idea, it's fine to abuse it in order to point out that fact.
- wodenokoto 11y agoI don't know enough about Greek to deeply comment on it. some people do consider using the same code point for apostrophe and citations problematic (it's definitely annoying when doing word segmentation) we also have breaking and non-breaking spaces, as well as tabs. We luckily didn't follow typewriter conventions and collapsed several alphabetic characters! A semicolon is not considered sentence final, whereas a question mark usually is. This makes it easier for software to auto capitalize. So at least that's possible a use case. There's also the possibility that Greek requires the top dot to be square or circle, meaning it might in fact have subtle differences in print.
- lazyjones 11y ago> A semicolon is not considered sentence final, whereas a question mark usually is. This makes it easier for software to auto capitalize. So at least that's possible a use case. Such software will need and have a language setting anyway (e.g. for hyphenation). It doesn't have to and cannot rely on code points alone, so the characters (or rather, different uses of the semicolon) needn't have different codepoints.
- retbull 11y agoWhat if someone is quoting something in a different language for a translation or similar?
- DarkUranium 11y agoTo be fair, that would break hyphenation as well if both languages use a Latin alphabet (give or take a few carons and such).
- thaumasiotes 11y ago> some people do consider using the same code point for apostrophe and citations problematic (it's definitely annoying when doing word segmentation) And those people are a menace. "Ball bearings" is a single word with a space in the middle. The only way you're going to get reliable word segmentation is with a natural language parser and a lexicon with an entry for "ball bearing". At that point, you're already recognizing "aren't" as a word; the punctuation isn't really relevant. On the other hand, if you're not particularly upset about messing up space-including words, there's no real reason to be upset about apostrophe-including ones either.
- Gigablah 11y ago> The Chinese version may have the line strike through another line, while the Japanese only has it touching. Case in point: 每 (chinese) vs 毎 (japanese).
- rootbear 11y agoI had to zoom in to my screen just to see the difference between those. I know that I am perhaps not well informed on this issue, but I also think that having multiple code points for what most people would perceive as the "same" character is not a good idea.
- Grue3 11y agoIt's a matter of the font size. At the size they're usually displayed, the characters look quite different [1] [2]. The real problem is that there's no way to mark a character as Chinese or Japanese on HN. So if it actually was the same character it would've been displayed identically. A common character that looks very different in Japanese and Chinese is 直 [3] and there's only one Unicode codepoint for it. [1] https://en.wiktionary.org/wiki/%E6%AF%8F https://en.wiktionary.org/wiki/%E6%AF%8F [2] https://en.wiktionary.org/wiki/%E6%AF%8E https://en.wiktionary.org/wiki/%E6%AF%8E [3] https://en.wiktionary.org/wiki/%E7%9B%B4 https://en.wiktionary.org/wiki/%E7%9B%B4
- mratzloff 11y agoIt's not the same. It's written differently.
- AnthonyMouse 11y ago> It's really difficult to do the right thing here. There is a fairly good solution to this for things like code editors and URLs or search strings in browsers: If a string contains a non-ASCII code point that is a homograph for an ASCII character, swap the text and background colors.
- wodenokoto 11y agoI meant the right approach to designating code points.
- Allan_Smithee 11y agoIn today's lesson, wodenokoto learned that real life is infinitely more complicated than San Francisco startups would have you believe.
- omaranto 11y agoThere is no "French A" and no "English A", I believe both languages call their alphabet the Roman (or Latin) alphabet. So those really are the same letter.
- thaumasiotes 11y agoThe Chinese could call their writing "the Roman alphabet" too (they don't), and label 闩 "A". So what?
- kps 11y agoThe Chinese could do that, but they'd be wrong. ‘French A’ and ‘English A’ have undiverged continuity with ‘Roman A’. Chinese writing is neither descended from Roman nor an alphabet.
- thaumasiotes 11y ago> ‘French A’ and ‘English A’ have undiverged continuity with ‘Roman A’. This is true, but it's equally true that they both have undiverged continuity with 'Greek Α', which has its own code point.
- bcoates 11y agoThe reason for a lot of homographs with ASCII is that there are old code pages that have entire non-US alphabets and punction sets in the 128-255 range, and regular ASCII in the 0-127 range. It's a design goal of Unicode to support exact round-trip transformation with any code page ever in use in the real world, so they can't unify two characters that appear at different code points in a greek codepage without breaking things, even if they're always graphically indistinguishable in a font.
- dragonwriter 11y ago> But it seems like it would be nice if code editors warned about text outside ascii. You usually only want that in strings and comments. That's somewhat language dependent, though its true for most uses of most popular languages.