7 ms·

I am not sure if native speakers are involved in Unicode standardization at all. Personal example. In Georgian (a few millions native speakers) we had histori

by adontz 6y ago

I am not sure if native speakers are involved in Unicode standardization at all. Personal example. In Georgian (a few millions native speakers) we had historically three different independent alphabets: mtavruli, nuskhuri and mkhedruli. All represent same letters, but different ways to write them. Alphabets are represented in Unicode under different code points. Only mkhedruli is used currently (last few centuries) and therefore most fonts have glyphs for mkhedruli only. Also there is no concept of capitalization in Georgian, there is no "a" and "A", we have only "ა", one does not start a sentence, a word or a personal name with some variant of a letter. Anna is ანნა in Georgian, the first and the last letters are written the same way. Now, because of some reasons, letters of mtavruli alphabet are registered as uppercase versions of letters of mkhedruli alphabet. Mind blown. And since well written software uses rules defined by Unicode standard, via ICU library or some platform specific routines, many desktop and mobile applications as well as major web-sites are hard to use. First letters of sentences are capitalized, but since mtavruli glyphs are missing from fonts, they are rendered as ⍰. So instead of მაგათი დედაც ვატირე. we see ⍰აგათი დედაც ვატირე. and have to guess what was the first letter. But who cares, if it is not used in U.S. right?

CharlotteBuff 6y ago
The uppercase mapping of Mkhedruli is Mtavruli in Unicode, but the titlecase mapping is not. If you use the uppercase mapping to automatically capitalise the first letter of a sentence, you are not following the recommendations of the Unicode standard.
- Mindless2112 6y ago
  TIL about ToTitleCase. An easy-to-visualize example of the difference is ToUpperCase("ǳ") => "Ǳ" ToTitleCase("ǳ") => "ǲ"
- techdragon 6y ago
  But how many programmers are even aware of the fact Unicode has both an Uppercase and Titlecase mapping? Let alone that they can do different things! I routinely see code that treats titlecase as “set individual character in this Unicode string to its upper case equivalent”... It wouldn’t surprise me at all to find out this kind of titlecase vs uppercase mistake is a very widespread issue.
  - TeMPOraL 6y ago
    I for one never heard of "titlecase" before. I guess that's what I get from learning text encoding from APIs exposed in programming languages.
- wereHamster 6y ago
  Interesting, I didn't know that there was a distinction between uppercase and titlecase. Perhaps unsurprisingly, unicode has a FAQ around just that: https://unicode.org/faq/casemap_charprop.html https://unicode.org/faq/casemap_charprop.html
- adontz 6y ago
  The thing is one cannot type mtavruli letters. No modern keyboard layout which I am aware of allows that. If you press "s" key it types "ს" (S), but if you press "shift+s" it types "შ" (SH), which is entirely different letter. Georgian alphabet has 33 letters, 7 more letters than Latin, so letters ჭღშჟძჩთ are mapped to shift+something keystrokes. Russian language has 33 letters too, but Russian layouts override 7 punctuation keys ,.;'[]~ so pressing "," will not type a comma, but a letter "б", pressing "]" will type "ъ", so Russian layouts remap punctuation symbols to shift+number keystrokes. Georgian layouts keep punctuation key assignments same as for English and use shift+letter for additional letters. Modern fonts may have mtavruli styled glyphs, that is true, but they are placed on mkhedruli code points anyway, which is exactly the problem, no widely used fonts support this case switch anyway and render ⍰.
  - ALan-ge 6y ago
    In Arch-based Linux distros you can use Caps Lock. When Caps Lock is active, each letter comes out capitalized (including shift+ combinations). Mtavruli has already been supported by the following fonts: Google Noto (Android 11 DP, Arch Linux), SegoeUI, Calibri (Windows 2019 update), Helvetica Neue (macOS Catalina, iOS 13).
jcranmer 6y ago
Doing a bit of research, this mkhedruli/mtavruli case relationship was only added in Unicode 11, which means you can go and read the documents you used to justify it. And it seems that the Georgians themselves were the ones who pushed for this change: https://unicode.org/L2/L2016/16034-n4707-georgian.pdf https://unicode.org/L2/L2016/16034-n4707-georgian.pdf. It seems this isn't a case of "ignorant Westerners screwing up Unicode" but "ignorant programmers who don't understand how Unicode works screwing it up," as noted in the sibling comment.
- knolax 6y ago
  Or "ignorant programmers who don't understand how text works or can't bother to support it properly getting to decide character encoding standards". We have English speaking programmers swearing by ASCII despite it not having all the characters needed for English (I can't find it now but there was a report by the Library of Congress that came to that conclusion). I suspect their Georgian counterparts have similar attitudes towards text.
  - _vbdg 6y ago
    > not having all the characters needed for English I've heard this argument a few times, and each time it's just as contrived, particularly if you stick to American English. While I'm happy to acknowledge it's not good enough for other languages, ASCII is plenty good for English. Unless you're trying to type cafe with an accented e to be fancy or type that old conjoined-ae character, ASCII is fine for everyday usage. See the fact that most Americans have never installed a separate keyboard layout.
    - catalogia 6y ago
      The 'long s' (ſ) was still in fairly common use in English less than 200 years ago. Consequently there are a lot of historically relevant documents written in English that can't be faithfully reproduced using ASCII. This may not be relevant to most people, but it seems plausibly relevant to libraries or archivists (for instance the Library of Congress.) Example: https://upload.wikimedia.org/wikipedia/commons/7/79/Bill_of_Rights_Pg1of1_AC.jpg https://upload.wikimedia.org/wikipedia/commons/7/79/Bill_of_...
lifthrasiir 6y ago
> I am not sure if native speakers are involved in Unicode standardization at all. Definitely. For Georgian alphabets that were one of the original scripts included in Unicode 1.0, as an example, the reference fonts were designed by Georgian type designers Anton Dumbadze and Irakli Garibashvili. If something was very off they must have noticed that.
_vbdg 6y ago
> who cares, if it is not used in U.S. right? This isn't intended to sound rude, but why should an American care? Dismissing the argument that for better or worse many people hold doesn't do any good. Making it might convince them otherwise.
- dang 6y ago
  Please don't take HN threads further into nationalistic flamewar.
  - _vbdg 6y ago
    That strikes me as much closer to linguistic than nationalistic; there's more than one nation that speaks English. Could you clarify what about that statement was flame-y? All I did was ask that adontz make the argument rather than making a dismissive statement.
    - dang 6y ago
      You took a sarcastic question that was being asked from a defensive place and reflected it back to the person in an amplified way that was guaranteed to rub salt in their wound.

by adontz 6y ago

CharlotteBuff 6y ago

The uppercase mapping of Mkhedruli is Mtavruli in Unicode, but the titlecase mapping is not. If you use the uppercase mapping to automatically capitalise the first letter of a sentence, you are not following the recommendations of the Unicode standard.

Mindless2112 6y ago
TIL about ToTitleCase. An easy-to-visualize example of the difference is ToUpperCase("ǳ") => "Ǳ" ToTitleCase("ǳ") => "ǲ"
techdragon 6y ago
But how many programmers are even aware of the fact Unicode has both an Uppercase and Titlecase mapping? Let alone that they can do different things! I routinely see code that treats titlecase as “set individual character in this Unicode string to its upper case equivalent”... It wouldn’t surprise me at all to find out this kind of titlecase vs uppercase mistake is a very widespread issue.
- TeMPOraL 6y ago
  I for one never heard of "titlecase" before. I guess that's what I get from learning text encoding from APIs exposed in programming languages.
wereHamster 6y ago
Interesting, I didn't know that there was a distinction between uppercase and titlecase. Perhaps unsurprisingly, unicode has a FAQ around just that: https://unicode.org/faq/casemap_charprop.html https://unicode.org/faq/casemap_charprop.html
adontz 6y ago
The thing is one cannot type mtavruli letters. No modern keyboard layout which I am aware of allows that. If you press "s" key it types "ს" (S), but if you press "shift+s" it types "შ" (SH), which is entirely different letter. Georgian alphabet has 33 letters, 7 more letters than Latin, so letters ჭღშჟძჩთ are mapped to shift+something keystrokes. Russian language has 33 letters too, but Russian layouts override 7 punctuation keys ,.;'[]~ so pressing "," will not type a comma, but a letter "б", pressing "]" will type "ъ", so Russian layouts remap punctuation symbols to shift+number keystrokes. Georgian layouts keep punctuation key assignments same as for English and use shift+letter for additional letters. Modern fonts may have mtavruli styled glyphs, that is true, but they are placed on mkhedruli code points anyway, which is exactly the problem, no widely used fonts support this case switch anyway and render ⍰.
- ALan-ge 6y ago
  In Arch-based Linux distros you can use Caps Lock. When Caps Lock is active, each letter comes out capitalized (including shift+ combinations). Mtavruli has already been supported by the following fonts: Google Noto (Android 11 DP, Arch Linux), SegoeUI, Calibri (Windows 2019 update), Helvetica Neue (macOS Catalina, iOS 13).

jcranmer 6y ago

Doing a bit of research, this mkhedruli/mtavruli case relationship was only added in Unicode 11, which means you can go and read the documents you used to justify it. And it seems that the Georgians themselves were the ones who pushed for this change: https://unicode.org/L2/L2016/16034-n4707-georgian.pdf https://unicode.org/L2/L2016/16034-n4707-georgian.pdf. It seems this isn't a case of "ignorant Westerners screwing up Unicode" but "ignorant programmers who don't understand how Unicode works screwing it up," as noted in the sibling comment.

knolax 6y ago
Or "ignorant programmers who don't understand how text works or can't bother to support it properly getting to decide character encoding standards". We have English speaking programmers swearing by ASCII despite it not having all the characters needed for English (I can't find it now but there was a report by the Library of Congress that came to that conclusion). I suspect their Georgian counterparts have similar attitudes towards text.
- _vbdg 6y ago
  > not having all the characters needed for English I've heard this argument a few times, and each time it's just as contrived, particularly if you stick to American English. While I'm happy to acknowledge it's not good enough for other languages, ASCII is plenty good for English. Unless you're trying to type cafe with an accented e to be fancy or type that old conjoined-ae character, ASCII is fine for everyday usage. See the fact that most Americans have never installed a separate keyboard layout.
  - catalogia 6y ago
    The 'long s' (ſ) was still in fairly common use in English less than 200 years ago. Consequently there are a lot of historically relevant documents written in English that can't be faithfully reproduced using ASCII. This may not be relevant to most people, but it seems plausibly relevant to libraries or archivists (for instance the Library of Congress.) Example: https://upload.wikimedia.org/wikipedia/commons/7/79/Bill_of_Rights_Pg1of1_AC.jpg https://upload.wikimedia.org/wikipedia/commons/7/79/Bill_of_...

lifthrasiir 6y ago

> I am not sure if native speakers are involved in Unicode standardization at all. Definitely. For Georgian alphabets that were one of the original scripts included in Unicode 1.0, as an example, the reference fonts were designed by Georgian type designers Anton Dumbadze and Irakli Garibashvili. If something was very off they must have noticed that.

_vbdg 6y ago

> who cares, if it is not used in U.S. right? This isn't intended to sound rude, but why should an American care? Dismissing the argument that for better or worse many people hold doesn't do any good. Making it might convince them otherwise.

dang 6y ago
Please don't take HN threads further into nationalistic flamewar.
- _vbdg 6y ago
  That strikes me as much closer to linguistic than nationalistic; there's more than one nation that speaks English. Could you clarify what about that statement was flame-y? All I did was ask that adontz make the argument rather than making a dismissive statement.
  - dang 6y ago
    You took a sarcastic question that was being asked from a defensive place and reflected it back to the person in an amplified way that was guaranteed to rub salt in their wound.