3 ms·

He's rendering normalized text, but normalization is only for string comparisons... I don't understand why emoji are width 1 either.. really the EastAsianWidth

He's rendering normalized text, but normalization is only for string comparisons... I don't understand why emoji are width 1 either.. really the EastAsianWidth.txt from the Unicode standard needs to match fixed with terminal emulators. I've been dealing with all of this recently in JOE: http://sourceforge.net/p/joe-editor/mercurial/ci/default/tree/NEWS.md http://sourceforge.net/p/joe-editor/mercurial/ci/default/tre... In particular JOE now finally renders combining characters correctly. It now stores a string for each character cell which includes the start character and any following combining characters. If any of them change, JOE re-emits the entire sequence. But which characters are combining characters? I expect \p{Mn} and \p{Me}, but U+1160 - U+11FF needs to be included as well but isn't. It's crazy that these are not counted as combining characters. Now I'm going to have to check how zero-width joiner is handled in terminal emulators. JOE is not changing the start character after a joiner into a combining character, ugh..

jrochkind1 11y ago
Well, there are multiple normalization forms in unicode. The OP isn't clear about this. (Perhaps because the python library he's using also isn't as clear as it ought to be? I dunno) 'compatibility' normalizations are mainly for comparison (including indexing/search/retrieval) and sorting, although there might be other uses. But indeed you should not expect a 'compatibility' normalization to render the same as the not-normalized input that produced it under normalization. The 'canonical' normalization outputs ought to render the same as de-normalized input, but rendering systems don't always get it quite right. For the web, the WWW consortium recommends a canonical normalization. The Unicode documentation on normalization forms is actually pretty readable and straightforward, for being a somewhat confusing topic. http://unicode.org/reports/tr15/ http://unicode.org/reports/tr15/ > Normalization Forms KC and KD [compatibility normalizations] must not be blindly applied to arbitrary text. Because they erase many formatting distinctions, they will prevent round-trip conversion to and from many legacy character sets, and unless supplanted by formatting markup, they may remove distinctions that are important to the semantics of the text. It is best to think of these Normalization Forms as being like uppercase or lowercase mappings: useful in certain contexts for identifying core meanings, but also performing modifications to the text that may not always be appropriate. The compatibility normalizations are pretty damn useful for indexing/search/retrieval though. Anyone storing non-ascii text in Solr or ElasticSearch (etc) probably wants to be familiar with them -- as a general rule of thumb, you probably want to do a compatibility normalization before indexing and again on query input.
- eevee 11y ago
  OP here. You say this all, and yet, if I google for "unicode strip accents"... Top-voted answer uses NFD, one below it uses NFKD: http://stackoverflow.com/questions/517923/what-is-the-best-way-to-remove-accents-in-a-python-unicode-string http://stackoverflow.com/questions/517923/what-is-the-best-w... NFKD: http://www.perlmonks.org/?node_id=835238 http://www.perlmonks.org/?node_id=835238 NFD: http://www.perlmonks.org/?node_id=1105025 http://www.perlmonks.org/?node_id=1105025 NFD: http://www.perlmonks.org/?node_id=485681 http://www.perlmonks.org/?node_id=485681 NFD: http://drillio.com/en/software/java/remove-accent-diacritic/ http://drillio.com/en/software/java/remove-accent-diacritic/ NFKD: https://gist.github.com/j4mie/557354 https://gist.github.com/j4mie/557354 Two and a half of the first six results blindly apply NFKD to arbitrary text. All of them use normalization. Sad state of affairs.
  - taejo 11y ago
    "Strip accents" is not a well-defined operation outside of a specific locale. Does "Ö" have an accent or not? In German, yes: it's an O with an umlaut. In English, yes: it's an O with some funny dots on it (heavy metal umlauts?). In the "New Yorker" dialect of English, it's an O with a dieresis. But in Hungarian, Finnish, Turkish, and many others, it's not: it's the letter between O and P, or between O and Ő, or after Z, or... If you do want to do this, you should know that it only makes sense in your own locale, and you shouldn't be surprised that the methods are somewhat ad-hoc (I'm not saying you shouldn't do this: I've done it myself).
    - ygra 11y ago
      In German, history of the letter and rules even dictate that ö should be written as oe in such cases (that's what it evolved from and that's what the two dots are; e.g. it's not a diaeresis in German, despite looking the same).
  - jrochkind1 11y ago
    Some of what you find googling is just wrong. Dealing with global characters is confusing, people get it wrong a lot, and suggest wrong answers. But it's true, as far as i know, that there's no unicode standard way to 'strip accents', which is unfortunate because we sometimes do need to do it. Even if 'strip accents' is locale dependent, and may have no sensible answer in some locales, I think there are sensible ways to do it in some locales (certainly in English, for Latin characters at least), and I wish there were a recognized best practice standard for doing it that could be implemented identically in various languages (maybe there is and I don't know it?). There are unicode standard ways to compare/sort strings ignoring accents, in at least some locales, which might get you there if you reverse engineered them and took them further. At any rate, at the end of the day, you can't simply talk about 'unicode normalization' without talking about the four different unicode normalization forms (canonical and compatibility; decomposed and composed) -- if you do, you are definitely getting something wrong. And also, unicode normalization forms are definitely _not_ intended to 'strip accents', that is not what they are for, they aren't the solution to that, even if the compatibility normalizations do it in some cases.

jrochkind1 11y ago

Well, there are multiple normalization forms in unicode. The OP isn't clear about this. (Perhaps because the python library he's using also isn't as clear as it ought to be? I dunno) 'compatibility' normalizations are mainly for comparison (including indexing/search/retrieval) and sorting, although there might be other uses. But indeed you should not expect a 'compatibility' normalization to render the same as the not-normalized input that produced it under normalization. The 'canonical' normalization outputs ought to render the same as de-normalized input, but rendering systems don't always get it quite right. For the web, the WWW consortium recommends a canonical normalization. The Unicode documentation on normalization forms is actually pretty readable and straightforward, for being a somewhat confusing topic. http://unicode.org/reports/tr15/ http://unicode.org/reports/tr15/ > Normalization Forms KC and KD [compatibility normalizations] must not be blindly applied to arbitrary text. Because they erase many formatting distinctions, they will prevent round-trip conversion to and from many legacy character sets, and unless supplanted by formatting markup, they may remove distinctions that are important to the semantics of the text. It is best to think of these Normalization Forms as being like uppercase or lowercase mappings: useful in certain contexts for identifying core meanings, but also performing modifications to the text that may not always be appropriate. The compatibility normalizations are pretty damn useful for indexing/search/retrieval though. Anyone storing non-ascii text in Solr or ElasticSearch (etc) probably wants to be familiar with them -- as a general rule of thumb, you probably want to do a compatibility normalization before indexing and again on query input.

eevee 11y ago
OP here. You say this all, and yet, if I google for "unicode strip accents"... Top-voted answer uses NFD, one below it uses NFKD: http://stackoverflow.com/questions/517923/what-is-the-best-way-to-remove-accents-in-a-python-unicode-string http://stackoverflow.com/questions/517923/what-is-the-best-w... NFKD: http://www.perlmonks.org/?node_id=835238 http://www.perlmonks.org/?node_id=835238 NFD: http://www.perlmonks.org/?node_id=1105025 http://www.perlmonks.org/?node_id=1105025 NFD: http://www.perlmonks.org/?node_id=485681 http://www.perlmonks.org/?node_id=485681 NFD: http://drillio.com/en/software/java/remove-accent-diacritic/ http://drillio.com/en/software/java/remove-accent-diacritic/ NFKD: https://gist.github.com/j4mie/557354 https://gist.github.com/j4mie/557354 Two and a half of the first six results blindly apply NFKD to arbitrary text. All of them use normalization. Sad state of affairs.
- taejo 11y ago
  "Strip accents" is not a well-defined operation outside of a specific locale. Does "Ö" have an accent or not? In German, yes: it's an O with an umlaut. In English, yes: it's an O with some funny dots on it (heavy metal umlauts?). In the "New Yorker" dialect of English, it's an O with a dieresis. But in Hungarian, Finnish, Turkish, and many others, it's not: it's the letter between O and P, or between O and Ő, or after Z, or... If you do want to do this, you should know that it only makes sense in your own locale, and you shouldn't be surprised that the methods are somewhat ad-hoc (I'm not saying you shouldn't do this: I've done it myself).
  - ygra 11y ago
    In German, history of the letter and rules even dictate that ö should be written as oe in such cases (that's what it evolved from and that's what the two dots are; e.g. it's not a diaeresis in German, despite looking the same).
- jrochkind1 11y ago
  Some of what you find googling is just wrong. Dealing with global characters is confusing, people get it wrong a lot, and suggest wrong answers. But it's true, as far as i know, that there's no unicode standard way to 'strip accents', which is unfortunate because we sometimes do need to do it. Even if 'strip accents' is locale dependent, and may have no sensible answer in some locales, I think there are sensible ways to do it in some locales (certainly in English, for Latin characters at least), and I wish there were a recognized best practice standard for doing it that could be implemented identically in various languages (maybe there is and I don't know it?). There are unicode standard ways to compare/sort strings ignoring accents, in at least some locales, which might get you there if you reverse engineered them and took them further. At any rate, at the end of the day, you can't simply talk about 'unicode normalization' without talking about the four different unicode normalization forms (canonical and compatibility; decomposed and composed) -- if you do, you are definitely getting something wrong. And also, unicode normalization forms are definitely _not_ intended to 'strip accents', that is not what they are for, they aren't the solution to that, even if the compatibility normalizations do it in some cases.