8 ms·

In the 90s I worked on a project to digitize land registration in Taiwan. In order to record deeds and property transfers, we needed to enter people's names an

In the 90s I worked on a project to digitize land registration in Taiwan. In order to record deeds and property transfers, we needed to enter people's names and official registered addresses into the computer system. The problem was that some people used non-traditional writing variants for their names, and some of their birthplaces were tiny places in China with weird names. Someone might write their name with a two-dot water radical instead of three-dot radical. We would print it out in the normal font, and the people would lose their minds, saying that it was wrong. Chinese people can be superstitious about the number of strokes in their name, so adding a stroke might make it unlucky, so they would not buy the property. The customer went to the agency responsible for managing the big character set, https://en.wikipedia.org/wiki/CNS_11643 https://en.wikipedia.org/wiki/CNS_11643 Despite having more characters than anything else on earth, it didn't have those variants. The agency said they would not encode them, because they were not real characters, just printing differences. The solution was for the staff in the office to use a "font maker" program to create a custom font with these characters. Then they could print out the deeds using a Chinese variant of Adobe Acrobat, and everyone was happy.

EvanAnderson 4y ago
Yikes. If somebody hasn’t written a “falsehoods programmers believe about human writing systems” document this would make for a good start.
- cestith 4y ago
  It deserves its own entry in "falsehoods programmers believe about names" lists too.
  - lmkg 4y ago
    It's already there, #11 "People’s names are all mapped in Unicode code points."
    - Scarblac 4y ago
      The falsehood here is thinking that if you can encode the name into the right code points, and you have a font that can print them, the result will be acceptable to the people whose name it is. They had that, but needed a font that used a different number of strokes for the characters because of the superstition.
      - lmkg 4y ago
        One could argue they're facets of the same issue. Although in the spirit of the original list, they would probably get split into separate line items. On further review, I think this is als similar to #12 & #13 on the list: "names are case-sensitive," and "names are not case-sensitive." To generalize that to include non-Western alphabets: display variations of the same character are significant, and display variations of the same character are not significant. This of course goes back to the evergreen philosophical question "what even is a character, anyways?" Since we've found a case where two characters which are the same character are not the same character. Are they distinct characters or typographical variants? Yesn't: one would want them unified for searching, but distinct for printing. But regardless of what they are, these characters/variants only show up in names. Names tend to retain archaic (or extinct) language variations longer than speech, which is the reason for rule #11, which is at least part of the problem.
        cestith 4y ago
        I fully agree with this second, expanded take of yours. Some names are both represented and not represented by the Unicode simultaneously. This suggests there should be variant versions of characters, but that becomes an even thornier combinatorics (and sorting/collation, and lookalike characters) issue than what already exists.
      - corrral 4y ago
        More generally, the notion that human culture, systems, and behaviors can be mapped, losslessly and without causing harm, to something a computer understands. I think these language examples are so good, as examples, because all aspects of them are clear and easy to follow. I think computerization of business and society and the systems that make them work, causes immense amounts of this kind of friction and pain all the time, in ways that are much harder to understand, explain, or catalog (which is precisely why it's such a big problem, though as far as I know it's received little attention) [EDIT] To distill it, I think that trying to make a computer a "source of truth" rather than a tool, tends to do substantial violence to the "truth".
- cardiffspaceman 4y ago
  "If the character isn't in Unicode it's in CNS-11643" apparently is also false.
duxup 4y ago
That sounds equally fascinating, and a little madding.
- kurthr 4y ago
  Yep, and with pictographic writing systems it's a lot more common than latin... but even here we have X Æ A-12 Musk, and Prince's name symbol. Heck, my initials are totally non-standard.
deleted 4y ago
[deleted]
jetrink 4y ago
That's a great story. The inability to represent a name with standard characters reminds me of when Prince changed his name to a symbol and they had to send all of the media floppy disks containing a custom font with a single character. https://nymag.com/intelligencer/2016/04/princes-legendary-floppy-disk-symbol-font.html https://nymag.com/intelligencer/2016/04/princes-legendary-fl...
- mdp2021 4y ago
  Are you acquainted with Freur (which means, "Underworld 0.5" - Rick Smith and Karl Hyde in the '80s)? "Freur", or, "The squiggle we chose as the name for a band but that CBS Records insisted should at least have a pronunciation". I see it is not in Unicode (well, you can never really know if you do not try), nor I can find pieces to reconstruct it. The "freur" in foreground: https://d4q8jbdc3dbnf.cloudfront.net/user/6885/edb290c6183aca5aad1199f6fcbe8db0.jpg https://d4q8jbdc3dbnf.cloudfront.net/user/6885/edb290c6183ac...
agumonkey 4y ago
Forgot which country (iran, turkey..) but one diacritic on a phone text got a girl killed because it altered the meaning one word. Turning the sentence from loving to threatening or insulting.
- kps 4y ago
  https://gizmodo.com/a-cellphones-missing-dot-kills-two-people-puts-three-m-382026 https://gizmodo.com/a-cellphones-missing-dot-kills-two-peopl...
  - jwilk 4y ago
    Disussed on HN in 2008: https://news.ycombinator.com/item?id=226853 https://news.ycombinator.com/item?id=226853 (18 comments)
- asveikau 4y ago
  That sounds terrible, however, it's important to remember that diacritics don't get people killed, the person who decides to kill ultimately needs to stop themselves.
  - pixl97 4y ago
    Ya, I don't see that happening in authoritarian countries. As a contrived example if you had a symbol for 'happy' you want to be very cautious that it doesn't get converted to 'gay' because in your language gay and happy mean the same thing, in some repressive regime it means the leadership gets to execute you with the approval of the law.
    - zarzavat 4y ago
      A recent example is that "Let's go [gun emoji] him" could be interpreted as either harmless fun, or conspiracy to murder, depending on if the recipient's phone displays that as a water pistol or a real gun. Edit: weirdly HN refuses to display that emoji.
77pt77 4y ago
> Chinese people can be superstitious about the number of strokes in their name, so adding a stroke might make it unlucky Why am I not surprised in the slightest?
Findecanor 4y ago
I've been told that this is also an issue in Japan, except the reason might more often be a matter of pride than superstition. It is supposedly one reason (of a few) why fax machines are still in common use in Japan. Later versions of Unicode support "Variation Forms" of Han characters as a way to be able to encode different variations. They are encoded as a Variation Selector code (U+E01000 and up) after the Han character. The forms are listed separate from Unicode versions in the "Ideographic Variation Database" <https://www.unicode.org/ivd/ https://www.unicode.org/ivd/>. So far, it contains characters from a couple of Japanese dictionaries, a Korean and one from Macao/Hong Kong.
- hinkley 4y ago
  I knew someone who added an accent character to their name because everyone pronounced it wrong. She met someone bilingual who shot back that if she wants it pronounced that way she needs to add an aigue. So she did, and everyone still pronounced her name wrong. In fact going any place with her very nearly became an “are we living in a simulation” crisis for me because the number of times she would say her name and the other person would say it back incorrectly was… upsetting. The degree to which some people butchered her name, especially combining half of her first and last name into a completely different name, made us joke about buggy NPCs. I could imagine how in some cultures writing it incorrectly hurts as much as pronouncing it incorrectly. Or possibly moreso in places where multiple plausible pronunciations have to be negotiated via an introduction, which is the case in China, is it not?
  - teknopaul 4y ago
    In Poland people have a neat life hack for that problem. They have other names for non-polish folk to use. Eg pawek, tomek, bartek rather than have people mangle their real name. My name got changed when I moved to Spain and it never bothered me, while I have met people who took great offence at the use of standard nicks that they had not explicitly sanctioned in advance. I know a guy who makes a new name up for everyone he meets. Like or lump it. If you are too sensitive about your name, you risk people not using it at all.
lostlogin 4y ago
There was a great thread on HH about names and falsehoods programmers believe. You’ve added to it, as custom fonts wasn’t one covered. I think it’s this thread: https://news.ycombinator.com/item?id=18567548 https://news.ycombinator.com/item?id=18567548 Edit: and it’s there, #11.

EvanAnderson 4y ago

Yikes. If somebody hasn’t written a “falsehoods programmers believe about human writing systems” document this would make for a good start.

cestith 4y ago
It deserves its own entry in "falsehoods programmers believe about names" lists too.
- lmkg 4y ago
  It's already there, #11 "People’s names are all mapped in Unicode code points."
  - Scarblac 4y ago
    The falsehood here is thinking that if you can encode the name into the right code points, and you have a font that can print them, the result will be acceptable to the people whose name it is. They had that, but needed a font that used a different number of strokes for the characters because of the superstition.
    - lmkg 4y ago
      One could argue they're facets of the same issue. Although in the spirit of the original list, they would probably get split into separate line items. On further review, I think this is als similar to #12 & #13 on the list: "names are case-sensitive," and "names are not case-sensitive." To generalize that to include non-Western alphabets: display variations of the same character are significant, and display variations of the same character are not significant. This of course goes back to the evergreen philosophical question "what even is a character, anyways?" Since we've found a case where two characters which are the same character are not the same character. Are they distinct characters or typographical variants? Yesn't: one would want them unified for searching, but distinct for printing. But regardless of what they are, these characters/variants only show up in names. Names tend to retain archaic (or extinct) language variations longer than speech, which is the reason for rule #11, which is at least part of the problem.
      - cestith 4y ago
        I fully agree with this second, expanded take of yours. Some names are both represented and not represented by the Unicode simultaneously. This suggests there should be variant versions of characters, but that becomes an even thornier combinatorics (and sorting/collation, and lookalike characters) issue than what already exists.
    - corrral 4y ago
      More generally, the notion that human culture, systems, and behaviors can be mapped, losslessly and without causing harm, to something a computer understands. I think these language examples are so good, as examples, because all aspects of them are clear and easy to follow. I think computerization of business and society and the systems that make them work, causes immense amounts of this kind of friction and pain all the time, in ways that are much harder to understand, explain, or catalog (which is precisely why it's such a big problem, though as far as I know it's received little attention) [EDIT] To distill it, I think that trying to make a computer a "source of truth" rather than a tool, tends to do substantial violence to the "truth".
cardiffspaceman 4y ago
"If the character isn't in Unicode it's in CNS-11643" apparently is also false.

duxup 4y ago

That sounds equally fascinating, and a little madding.

kurthr 4y ago
Yep, and with pictographic writing systems it's a lot more common than latin... but even here we have X Æ A-12 Musk, and Prince's name symbol. Heck, my initials are totally non-standard.

deleted 4y ago

[deleted]

jetrink 4y ago

That's a great story. The inability to represent a name with standard characters reminds me of when Prince changed his name to a symbol and they had to send all of the media floppy disks containing a custom font with a single character. https://nymag.com/intelligencer/2016/04/princes-legendary-floppy-disk-symbol-font.html https://nymag.com/intelligencer/2016/04/princes-legendary-fl...

mdp2021 4y ago
Are you acquainted with Freur (which means, "Underworld 0.5" - Rick Smith and Karl Hyde in the '80s)? "Freur", or, "The squiggle we chose as the name for a band but that CBS Records insisted should at least have a pronunciation". I see it is not in Unicode (well, you can never really know if you do not try), nor I can find pieces to reconstruct it. The "freur" in foreground: https://d4q8jbdc3dbnf.cloudfront.net/user/6885/edb290c6183aca5aad1199f6fcbe8db0.jpg https://d4q8jbdc3dbnf.cloudfront.net/user/6885/edb290c6183ac...

agumonkey 4y ago

Forgot which country (iran, turkey..) but one diacritic on a phone text got a girl killed because it altered the meaning one word. Turning the sentence from loving to threatening or insulting.

kps 4y ago
https://gizmodo.com/a-cellphones-missing-dot-kills-two-people-puts-three-m-382026 https://gizmodo.com/a-cellphones-missing-dot-kills-two-peopl...
- jwilk 4y ago
  Disussed on HN in 2008: https://news.ycombinator.com/item?id=226853 https://news.ycombinator.com/item?id=226853 (18 comments)
asveikau 4y ago
That sounds terrible, however, it's important to remember that diacritics don't get people killed, the person who decides to kill ultimately needs to stop themselves.
- pixl97 4y ago
  Ya, I don't see that happening in authoritarian countries. As a contrived example if you had a symbol for 'happy' you want to be very cautious that it doesn't get converted to 'gay' because in your language gay and happy mean the same thing, in some repressive regime it means the leadership gets to execute you with the approval of the law.
  - zarzavat 4y ago
    A recent example is that "Let's go [gun emoji] him" could be interpreted as either harmless fun, or conspiracy to murder, depending on if the recipient's phone displays that as a water pistol or a real gun. Edit: weirdly HN refuses to display that emoji.

77pt77 4y ago

> Chinese people can be superstitious about the number of strokes in their name, so adding a stroke might make it unlucky Why am I not surprised in the slightest?

Findecanor 4y ago

I've been told that this is also an issue in Japan, except the reason might more often be a matter of pride than superstition. It is supposedly one reason (of a few) why fax machines are still in common use in Japan. Later versions of Unicode support "Variation Forms" of Han characters as a way to be able to encode different variations. They are encoded as a Variation Selector code (U+E01000 and up) after the Han character. The forms are listed separate from Unicode versions in the "Ideographic Variation Database" <https://www.unicode.org/ivd/ https://www.unicode.org/ivd/>. So far, it contains characters from a couple of Japanese dictionaries, a Korean and one from Macao/Hong Kong.

hinkley 4y ago
I knew someone who added an accent character to their name because everyone pronounced it wrong. She met someone bilingual who shot back that if she wants it pronounced that way she needs to add an aigue. So she did, and everyone still pronounced her name wrong. In fact going any place with her very nearly became an “are we living in a simulation” crisis for me because the number of times she would say her name and the other person would say it back incorrectly was… upsetting. The degree to which some people butchered her name, especially combining half of her first and last name into a completely different name, made us joke about buggy NPCs. I could imagine how in some cultures writing it incorrectly hurts as much as pronouncing it incorrectly. Or possibly moreso in places where multiple plausible pronunciations have to be negotiated via an introduction, which is the case in China, is it not?
- teknopaul 4y ago
  In Poland people have a neat life hack for that problem. They have other names for non-polish folk to use. Eg pawek, tomek, bartek rather than have people mangle their real name. My name got changed when I moved to Spain and it never bothered me, while I have met people who took great offence at the use of standard nicks that they had not explicitly sanctioned in advance. I know a guy who makes a new name up for everyone he meets. Like or lump it. If you are too sensitive about your name, you risk people not using it at all.

lostlogin 4y ago

There was a great thread on HH about names and falsehoods programmers believe. You’ve added to it, as custom fonts wasn’t one covered. I think it’s this thread: https://news.ycombinator.com/item?id=18567548 https://news.ycombinator.com/item?id=18567548 Edit: and it’s there, #11.