21 ms·

Very similar problem to one described started my exodus from Google services. I also have non-latin characters in my name however I knew it was always an issue

• musicale 5y ago
  > the best course of action would be to change my name That's usually easier than getting a company to fix their software.
• nullspace 5y ago
  > the best course of action would be to change my name As someone who has been told this, for other reasons, I empathize. My reaction has always been - "Your system can't even handle names, you need to fix it". Edit: I wish there was a library / service that helped you handle all sorts of edge cases in names, so that you don' t have to worry about it. Just use a user-id, and set / get a name from a lib / service that can actually handle it.
  • web007 5y ago
    I believe that library / service is called UTF-8. These days everything should be stored as bare UTF-8 data (or utf8mb4 if you're MySQL) and presented without anything else. Don't parse it, don't slice-and-dice it, don't prepend or append titles or honorifics or suffixes, don't make assumptions about length or content beyond "must be > 0 as a whole" and DEFINITELY don't use it as an identifier. Treat it as a non-unique opaque token and you'll be fine greater than 99% of the time. There are people with no last name. There are people with two or three or twelve middle names. There are people with a number for a last name. There are people with a symbol for their entire name. Take what they give you and use it and be done with it.
    • lmm 5y ago
      Not good enough, thanks to Han unification - if you do that you'll mangle Japanese names.
      • indigo945 5y ago
        "Mangle" is an exaggeration. Japanese names will look correct after Han unification both to Japanese people on a Japanese computer and to Chinese people on a Chinese computer. (All other combinations fail.)
        • lmm 5y ago
          > Japanese names will look correct after Han unification both to Japanese people on a Japanese computer and to Chinese people on a Chinese computer. Only if you are displaying them in a way that respects the computer's preferences (most websites and programs, especially American websites and programs, don't) and those preferences are set correctly. And certainly if you have text blocks that contain both Chinese and Japanese names you will always mangle at least one of them.
        • jack1243star 5y ago
          And we bilinguals can spam <span lang="..."> to try force the correct font on our blogs. Ugh.
      • web007 5y ago
        TIL, https://en.wikipedia.org/wiki/Han_unification#Examples_of_language-dependent_glyphs https://en.wikipedia.org/wiki/Han_unification#Examples_of_la... It looks like there's no general solution possible with Han unification. If you have any two of ZH and JA and KO and VI in a page, you will fail to display one of them correctly for certain characters unless (as in that wiki page) you add a LANG attribute for each element they are contained within. Personally, I would use the browser's language or user locale to set the page language and give up. Then in Japan the local (Japanese) names would look fine, and same for China, Korea and Vietnam. Local consistency versus complicated perfection (tracking the input language as well as tokens and using them everywhere), and I could blame the browser for doing poorly at its impossible job. One possible "perfect" fix would be to store the token and <span lang=...>$token</span> as well. The only place the non-wrapped version would be used is plaintext email or SMS, either of which are beyond lost causes for other reasons. Doing it with an embedded SPAN tag presents its own problems with sanitization, as well as guaranteeing it's always wrong if the input language was specified incorrectly when the token was populated, versus as above where it would be corrected to the local-optimal version if the user locale overrides it.
  • dymk 5y ago
    Has that reaction ever resulted in the other party fixing their system in a timely manner?
    • kingcharles 5y ago
      In my experience, never. You are considered the bug, not their system.
      • PeterisP 5y ago
        Here's a relevant recent EU court case of a person arguing with their bank that their name should be represented properly including the accented 'é', as the GDPR asserts a right to have mistakes of personal data corrected. The bank argued that it's impossible due to a legacy system using EBCDIC encoding and would be expensive to change. The appeals court affirmed that the customer has the right to get mistakes in their personal data corrected, and it is the duty of the bank to do so even if it is expensive. https://gdprhub.eu/index.php?title=Court_of_Appeal_of_Brussels_-_2019/AR/1006 https://gdprhub.eu/index.php?title=Court_of_Appeal_of_Brusse...
        • sushsjsuauahab 5y ago
          This is an insanely nightmarish precedent. We need to take computer systems less seriously. In a human-interface world, these issues are avoided thanks to simple human intuition.
          • unionpivo 5y ago
            I disagree. We are in this mess because nobody took it serious enough for long time. Now that serious money is on the table, it might actually get solved and fixed once and for all. We/us in tech had 30 years to fix our shit on our own. We didn't, that's the result.
        • GoblinSlayer 5y ago
          I once did it, encoded UTF-8 text in a legacy text because the system interface didn't support unicode, and decoded on the other end. Used • prefix as a marker of encoded string... now that I think about it, it could be just old good BOM. MIME standard also has extensive experience in packing arbitrary data in 7-bit clean text.
• mjevans 5y ago
  This is exactly why I hate the way Python3 handles Unicode. EVERY language should _try_ to handle Unicode such that if a data sequence were valid before it remains valid after. NONE should ever FORCE validation, since sometimes, like in the article's case, the correct answer is GIGO. Just pass it through and hope it continues to work. Sometimes the error is trying to enforce that validation.
  • geofft 5y ago
    Python 3 usually handles this correctly, and I'm a little bit confused what's going on in the article, exactly. For UNIX path names (and other OS data like environment variables), Python uses the "surrogateescape" error handling method, which does exactly what you ask. Any byte sequence can be converted to a string. If it decodes as valid UTF-8, it will do that. If it hits a byte that does not decode as valid UTF-8 (necessarily a byte >= 128), it will map it to code points U+DC80 through U+DCFF. These are in a reserved ranges of code points ("surrogates", which make it possible to represent code points > 0xFFFF in UTF-16), and they can't show up in actual Unicode text (i.e., there is no UTF-8 encoding of them, strictly speaking, and if you applied the UTF-8 encoding algorithm to a code point in the U+D800 to U+DFFF range, you would get bytes that aren't valid UTF-8). On the way out, this is reversed. So you get the results you expect if your filenames are in UTF-8, but since UNIX has no requirement that filenames are indeed UTF-8 (the only constraint is they can't contain NUL or ASCII-forward-slash), the bytes are preserved in a funky-looking format in Python and you get the exact same output on the other end. See https://www.python.org/dev/peps/pep-0383/ https://www.python.org/dev/peps/pep-0383/ for more on what's going on. The tl;dr for users of Python is that if you want to interact with, say, subprocess output as mostly-normal strings (instead of bytes) but you want to be robust to non-UTF-8 bytes, you should do something like subprocess.check_output(["some", "command"], errors="surrogateescape") You don't need to do this for APIs that directly interact with pathnames, because they do it already. You just need to do it for things like subprocess output and file contents that Python doesn't know you want to handle in this way. ... On Windows, however, path names must be valid Unicode and are stored in UTF-16. So the idea of a "ł" that doesn't decode properly shouldn't even happen! Mikołaj's home directory ought to be a very boring (and valid) 004d 0069 006b 006f 0142 0061 006a on disk. Windows doesn't enforce that file paths are valid UTF-16 though (specifically, the surrogate code points are only supposed to show up in a certain way, but nothing enforces that and you can have random surrogates on disk), and hence Rust, which internally represents all strings in UTF-8, has a solution ("WTF-8") that's basically the inverse of surrogateescape - it uses extrapolated-UTF-8-encoding-of-surrogates to handle unpaired surrogates. http://simonsapin.github.io/wtf-8/ http://simonsapin.github.io/wtf-8/ But it seems very odd to me that the directory C:\Users\Mikołaj would actually contain any of those, and if it doesn't, I would expect it to very easily turn into a Python Unicode string. Maybe this is from a Python version before https://www.python.org/dev/peps/pep-0529/ https://www.python.org/dev/peps/pep-0529/ , which is claimed to "fail to round-trip characters outside of the user's active code page"? Maybe this is from a Python version after that change and it's wrong?
• perl4ever 5y ago
  A Polish relative of mine used to just gave an arbitrary substitute name (e.g. "Dave Smith") for restaurant reservations, because even if they could write his last name, they wouldn't be able to pronounce it. My sibling has a name that has an accent, and just enters it with the plain letter most of the time. The name was once rare and "ethnic", but became popular a generation later so people know how to pronounce it regardless. Our parents gave us two middle names, wanting to preserve our grandmothers' surnames, but also in the spirit of "Bobby Tables", having ambivalent feelings about the computerization of society tending towards inflexibility. According to: https://en.wikipedia.org/wiki/Naming_customs_of_Hispanic_America https://en.wikipedia.org/wiki/Naming_customs_of_Hispanic_Ame... ...misunderstanding of naming customs in the US has actually led to significant consequences due to last names not matching on legal documents. I remember reading a story about how there are people in China whose name incorporates a character that is obscure enough, the authorities are trying to eliminate it and get them to change their name. If I recall correctly, Chinese has a particular problem with characters that are part of names that have been around forever, but are no longer used for ordinary writing.
  • em-bee 5y ago
    a friend of mine has that problem. his company is owned by his wife because his name can't appear in legal documents and he refuses to change it. his approach was that he petitioned unicode to include that character. i don't think the authorities are actively trying to eliminate those characters but simply don't want to go through the effort to track down and have those characters added to the standard. the process takes years and in the mean time you have to live with the inconvenience. also most of the people faced with the problem would not even know how.
    • sugarkjube 5y ago
      > his name can't appear in legal documents maybe has the benefit that he also can't receive speeding tickets? > his approach was that he petitioned unicode to include that character. Sounds like the right approach (although not the easiest). if unicode can include thousands of smileys, the least they can do is include actual characters used in people names.