9 ms·

My name is Léon, with the acute accent on the e. I usually leave the accent when I need to enter my name somewhere digitally, since in about 50% of the cases it

by LeonM 9y ago

My name is Léon, with the acute accent on the e. I usually leave the accent when I need to enter my name somewhere digitally, since in about 50% of the cases it's not handled correctly. It usually ends up as L'on. Even in the travel world it goes wrong all the time. You'd expect large international travel organisations (yes, talking to you Tui!) to be able to handle UTF8 names since many of their customers and locations will have special characters, but no. I once was nearly refused to board an airplane because the name on my ticket did not match the one on my passport...

makecheck 9y ago
String-matching is really scary in Unicode, especially since the exact form of the string matters with respect to composition — and that’s before you even consider that some characters just plain look like others or even are the same glyph. And strings can contain things like zero-width spaces that look like nothing at all. Sure, there are recommended practices but there have been enough mistakes already (or lazy programmers) that it is hard to be confident that any string with “interesting” symbols in it is exactly what it appears to be. And there have been security problems related to the fact that many interfaces expect the user to know exactly what they’re reading, much less the programmer.
- Klathmon 9y ago
  It almost sounds like there could be a lot of benefit from a "homophones check" system but for Unicode glyphs (perhaps with a variable amount of "closeness") being built into Unicode handling libraries. Like how "е" looks identical to "e", which looks close to "ė" if you aren't careful, which might be mistaken for "é" in smaller fonts even though all 4 letters are different unicode glyphs. Being able to say "ɡrеɡ" is the same as "greg" even though 3 of the 4 characters are actually different would be extremely useful in some cases, and in others would be extremely incorrect, so giving the developer the ability to say how "exact" they need their checks to be in a "native" and easy way might go a long way toward not only making this problem more "obvious" but also toward forcing them to be explicit about what they are checking for.
  - asveikau 9y ago
    This comparison is language specific. Do you group c with ç and č? In English you would. In France, Portugal, Serbia, the Baltic States or Czech republic you may not.
    - greglindahl 9y ago
      Are there any word pairs that switch ç for č or vice versa?
      - asveikau 9y ago
        I wouldn't think so, because the languages I know of using ç do not use č and vice versa. There are definitely pairs with "plain" C in a bunch of those. The ė mentioned is distinct from e in Lithuanian.
    - Klathmon 9y ago
      I was just trying to think of a way to actually deal with this in a way that it's up to the developer to decide what they should be matching. Even in those languages you still might want to treat an OCR of a "c" with "ç" in some cases, or you might want to treat them identical when moving from a system that only accepted ASCII in the past to one that is fully unicode compliant. And even in English there are situations where "ç" should not be treated as "c" (like in URL resolving or other scenarios where an exact match is needed). It's not technically correct, but it's "more correct". And giving the developer the ability to determine where along that "scale" of exact-ness they want to be might help.
    - ygra 9y ago
      It's also font-specific. Is т a homoglyph of T or m, for example? There isn't really a good way to solve this because restricting systems to only use ASCII (which also has homoglyphs, e.g. 0/O, 1/l, I/l, ...) is very user-unfriendly.
      - algesten 9y ago
        I got to write an address database matcher once covering all of scandinavia and poland. For suggest functions, it turns out people fully expect to be able to write some sort of ascii normalisation and still get a match, i.e. the address has ż in it, but people want to type a plain z. And the rules for this are not entirely obvious. A swede would totally expect being able to write ö instead of the norwegian ø when doing routing across the border.
- skookumchuck 9y ago
  It's a huge mistake for Unicode to have two different code points having the same glyph. There should not be semantic meaning that disappears when something is printed. We're going to be suffering for that mistake for a looong time.
  - scrollaway 9y ago
    > There should not be semantic meaning that disappears when something is printed. I can see why you'd say that, but who decides whether it should?
    - skookumchuck 9y ago
      The Unicode Committee. I think they lost the point of what Unicode was somewhere along the line.
  - alanh 9y ago
    Disagree. A good example of the opposite mistake is the “Turkish i” problem. Basically they have a version of I with and without a dot — for both lowercase and uppercase — so algorithms that uppercase i to I break Turkish by removing the dot. If the Turkish i were a unique code point, the algorithms would not mess it up.
    - skookumchuck 9y ago
      Uppercasing is heavily dependent on context. Even the ASCII characters are context dependent. a+b=d+d Shouldn't be uppercased. It's an insoluble problem to put contextual semantic info into Unicode characters, because individual characters have no context.
    - josefx 9y ago
      Then you have the german ß (sharp S) which does not have an upper case version. While ISO added one for whatever reason the official upper case is two letters consisting of either "SS" or "SZ". So you have three different ways to upper case ß one which is guaranteed to be wrong in any official context and two which lower case to "ss" or "sz" and not back to ß. That is one big ouch, especially to the ISO standard adding that invalid upper case variation. Languages are messy, best don't try to transform your input text in any way.
- amptorn 9y ago
  > String-matching is really scary in Unicode Isn't this why NFKC normalization exists?
  - mjevans 9y ago
    Unicode has four different normalization forms. Different forms are useful for different intended outcomes. You, as the programmer, need to understand each of them and why you want to use them. Brief overview: https://en.wikipedia.org/wiki/Unicode_equivalence#Normalization https://en.wikipedia.org/wiki/Unicode_equivalence#Normalizat... More technical details: http://www.unicode.org/reports/tr15/ http://www.unicode.org/reports/tr15/ I --THINK-- offhand, that NFKC is what you want to use when preparing a password input for processing/comparison (it's lossless, but to a specific point). I also --THINK-- that NFC is the form you want to use when retaining source glyph language distinctions. From the stackoverflow hits: https://stackoverflow.com/questions/16173328/what-unicode-normalization-and-other-processing-is-appropriate-for-passwords-w https://stackoverflow.com/questions/16173328/what-unicode-no... I agree with the destructive (pre computation/comparison) operation and that either of the NFKD or NFKC forms should be used (since they destroy non-printing differences for visually compatible characters; a more user friendly approach). The 'C' forms are always more condensed (accents are packed in to a single character where possible), and thus of higher entropy per input byte. It is my belief that this form is likely to be less susceptible to attacks. The 'D' forms seem like good choices for /editors/ where the precise nature of a character might be altered by adding or removing accents. (Most human input boxes; during the input/edit process)
    - eridius 9y ago
      > The 'C' forms are always more condensed (accents are packed in to a single character where possible), and thus of higher entropy per input byte. It is my belief that this form is likely to be less susceptible to attacks. What sort of attacks are you talking about? > The 'D' forms seem like good choices for /editors/ where the precise nature of a character might be altered by adding or removing accents. (Most human input boxes; during the input/edit process) Editors shouldn't care about C vs D. The reason being, once you've typed the grapheme cluster, it's supposed to act in an editor as if it's a single "character" regardless of whether it's made from one codepoint or several. This means that if I type é then arrow keys and the delete key will operate it on it exactly the same whether it's composed or decomposed.
enkrs 9y ago
My shipping address is "Roņu street, Liepāja" and in the other end of our city is "Rožu street, Liepāja" If the address gets somehow mangled, as it often does, the shipping label could read "Ro?u street" and the postal offices guess on the address being "Roņu" or "Rožu" is as good as anyones. I too usually just write "Ronu street"
- coldtea 9y ago
  How about adding the zip code for disambiguation? Because even with no characters messed up, many cities have 2 or more streets with the same name.
  - jbg_ 9y ago
    Depending on the size of the city, there's a good chance these two differently-named streets are in the same postal code.
  - tetraodonpuffer 9y ago
    in many parts of the world zip codes are city-wide (regardless of city size), so that wouldn't help
    - markdown 9y ago
      And some countries don't use zip-codes at all.
      - coldtea 9y ago
        And at least one country (Singapore) has building-level zip codes
        theBobBob 9y ago
        Ireland goes even further and each apartment in an apartment block would have its own unique code.
  - 9y ago
ensiferum 9y ago
Pro tip, write your name as it is in your passport on the <<<<<<< line
deleted 9y ago
[deleted]

by LeonM 9y ago

makecheck 9y ago

String-matching is really scary in Unicode, especially since the exact form of the string matters with respect to composition — and that’s before you even consider that some characters just plain look like others or even are the same glyph. And strings can contain things like zero-width spaces that look like nothing at all. Sure, there are recommended practices but there have been enough mistakes already (or lazy programmers) that it is hard to be confident that any string with “interesting” symbols in it is exactly what it appears to be. And there have been security problems related to the fact that many interfaces expect the user to know exactly what they’re reading, much less the programmer.

Klathmon 9y ago
It almost sounds like there could be a lot of benefit from a "homophones check" system but for Unicode glyphs (perhaps with a variable amount of "closeness") being built into Unicode handling libraries. Like how "е" looks identical to "e", which looks close to "ė" if you aren't careful, which might be mistaken for "é" in smaller fonts even though all 4 letters are different unicode glyphs. Being able to say "ɡrеɡ" is the same as "greg" even though 3 of the 4 characters are actually different would be extremely useful in some cases, and in others would be extremely incorrect, so giving the developer the ability to say how "exact" they need their checks to be in a "native" and easy way might go a long way toward not only making this problem more "obvious" but also toward forcing them to be explicit about what they are checking for.
- asveikau 9y ago
  This comparison is language specific. Do you group c with ç and č? In English you would. In France, Portugal, Serbia, the Baltic States or Czech republic you may not.
  - greglindahl 9y ago
    Are there any word pairs that switch ç for č or vice versa?
    - asveikau 9y ago
      I wouldn't think so, because the languages I know of using ç do not use č and vice versa. There are definitely pairs with "plain" C in a bunch of those. The ė mentioned is distinct from e in Lithuanian.
  - Klathmon 9y ago
    I was just trying to think of a way to actually deal with this in a way that it's up to the developer to decide what they should be matching. Even in those languages you still might want to treat an OCR of a "c" with "ç" in some cases, or you might want to treat them identical when moving from a system that only accepted ASCII in the past to one that is fully unicode compliant. And even in English there are situations where "ç" should not be treated as "c" (like in URL resolving or other scenarios where an exact match is needed). It's not technically correct, but it's "more correct". And giving the developer the ability to determine where along that "scale" of exact-ness they want to be might help.
  - ygra 9y ago
    It's also font-specific. Is т a homoglyph of T or m, for example? There isn't really a good way to solve this because restricting systems to only use ASCII (which also has homoglyphs, e.g. 0/O, 1/l, I/l, ...) is very user-unfriendly.
    - algesten 9y ago
      I got to write an address database matcher once covering all of scandinavia and poland. For suggest functions, it turns out people fully expect to be able to write some sort of ascii normalisation and still get a match, i.e. the address has ż in it, but people want to type a plain z. And the rules for this are not entirely obvious. A swede would totally expect being able to write ö instead of the norwegian ø when doing routing across the border.
skookumchuck 9y ago
It's a huge mistake for Unicode to have two different code points having the same glyph. There should not be semantic meaning that disappears when something is printed. We're going to be suffering for that mistake for a looong time.
- scrollaway 9y ago
  > There should not be semantic meaning that disappears when something is printed. I can see why you'd say that, but who decides whether it should?
  - skookumchuck 9y ago
    The Unicode Committee. I think they lost the point of what Unicode was somewhere along the line.
- alanh 9y ago
  Disagree. A good example of the opposite mistake is the “Turkish i” problem. Basically they have a version of I with and without a dot — for both lowercase and uppercase — so algorithms that uppercase i to I break Turkish by removing the dot. If the Turkish i were a unique code point, the algorithms would not mess it up.
  - skookumchuck 9y ago
    Uppercasing is heavily dependent on context. Even the ASCII characters are context dependent. a+b=d+d Shouldn't be uppercased. It's an insoluble problem to put contextual semantic info into Unicode characters, because individual characters have no context.
  - josefx 9y ago
    Then you have the german ß (sharp S) which does not have an upper case version. While ISO added one for whatever reason the official upper case is two letters consisting of either "SS" or "SZ". So you have three different ways to upper case ß one which is guaranteed to be wrong in any official context and two which lower case to "ss" or "sz" and not back to ß. That is one big ouch, especially to the ISO standard adding that invalid upper case variation. Languages are messy, best don't try to transform your input text in any way.
amptorn 9y ago
> String-matching is really scary in Unicode Isn't this why NFKC normalization exists?
- mjevans 9y ago
  Unicode has four different normalization forms. Different forms are useful for different intended outcomes. You, as the programmer, need to understand each of them and why you want to use them. Brief overview: https://en.wikipedia.org/wiki/Unicode_equivalence#Normalization https://en.wikipedia.org/wiki/Unicode_equivalence#Normalizat... More technical details: http://www.unicode.org/reports/tr15/ http://www.unicode.org/reports/tr15/ I --THINK-- offhand, that NFKC is what you want to use when preparing a password input for processing/comparison (it's lossless, but to a specific point). I also --THINK-- that NFC is the form you want to use when retaining source glyph language distinctions. From the stackoverflow hits: https://stackoverflow.com/questions/16173328/what-unicode-normalization-and-other-processing-is-appropriate-for-passwords-w https://stackoverflow.com/questions/16173328/what-unicode-no... I agree with the destructive (pre computation/comparison) operation and that either of the NFKD or NFKC forms should be used (since they destroy non-printing differences for visually compatible characters; a more user friendly approach). The 'C' forms are always more condensed (accents are packed in to a single character where possible), and thus of higher entropy per input byte. It is my belief that this form is likely to be less susceptible to attacks. The 'D' forms seem like good choices for /editors/ where the precise nature of a character might be altered by adding or removing accents. (Most human input boxes; during the input/edit process)
  - eridius 9y ago
    > The 'C' forms are always more condensed (accents are packed in to a single character where possible), and thus of higher entropy per input byte. It is my belief that this form is likely to be less susceptible to attacks. What sort of attacks are you talking about? > The 'D' forms seem like good choices for /editors/ where the precise nature of a character might be altered by adding or removing accents. (Most human input boxes; during the input/edit process) Editors shouldn't care about C vs D. The reason being, once you've typed the grapheme cluster, it's supposed to act in an editor as if it's a single "character" regardless of whether it's made from one codepoint or several. This means that if I type é then arrow keys and the delete key will operate it on it exactly the same whether it's composed or decomposed.

enkrs 9y ago

My shipping address is "Roņu street, Liepāja" and in the other end of our city is "Rožu street, Liepāja" If the address gets somehow mangled, as it often does, the shipping label could read "Ro?u street" and the postal offices guess on the address being "Roņu" or "Rožu" is as good as anyones. I too usually just write "Ronu street"

coldtea 9y ago
How about adding the zip code for disambiguation? Because even with no characters messed up, many cities have 2 or more streets with the same name.
- jbg_ 9y ago
  Depending on the size of the city, there's a good chance these two differently-named streets are in the same postal code.
- tetraodonpuffer 9y ago
  in many parts of the world zip codes are city-wide (regardless of city size), so that wouldn't help
  - markdown 9y ago
    And some countries don't use zip-codes at all.
    - coldtea 9y ago
      And at least one country (Singapore) has building-level zip codes
      - theBobBob 9y ago
        Ireland goes even further and each apartment in an apartment block would have its own unique code.
- 9y ago

ensiferum 9y ago

Pro tip, write your name as it is in your passport on the <<<<<<< line

deleted 9y ago

[deleted]