4 ms·

If you limit yourself to the West Europe languages, you can just go with the stdlib: >>> import unicodedata >>> print( unicodedata.normalize('NFKD', "

If you limit yourself to the West Europe languages, you can just go with the stdlib: >>> import unicodedata >>> print( unicodedata.normalize('NFKD', "éèêàùçÇ").encode('ascii','ignore')) eeeaucC But if you need to project (transliterate to ascii) Arabic, Russian or Chinese, unidecode is close to black magic: >>> from unidecode import unidecode >>> unidecode("北亰") 'Bei Jing ' Anyway, always remember that str.encode(), str.decode(), open() and many other related callables have an "errors" parameters that allow you to deal with unkown solutions when encoding or decoding: >>> print("Père Noël".encode("ascii", errors="ignore")) b'Pre Nol' >>> print("Père Noël".encode("ascii", errors="replace")) b'P?re No?l' I'll conclude with the mandatory "use Python 3" (3.7 if you can, it has many utf8 fixes: https://vstinner.github.io/posix-locale.html https://vstinner.github.io/posix-locale.html), since you'll be in a world of pain if you deal with non-ascii in Python 2, and EOL is next year :) Tic, Toc...

titanix2 8y ago
Just fixing the French here: it’s Père Noël, not Pére Noël.
- sametmax 8y ago
  'Tin la honte.
kuschku 8y ago
Unidecode may handle Chinese fine, but it definitely handles Western European languages wrong. DIN 5007 Var. 2 specifies that for the purposes of sorting, ö is replaced with oe. This also applies to ä (ae), ü (ue) and ß (ss). This same replacement rule is also used on passports and IDs. Unidecode does not handle this correctly. The unidecode author wrote about this: > In German, there's the typographical convention that an umlaut (the double-dots on: ä ö ü) can be written as an "-e", like with "Schön" becoming "Schoen". But Unidecode doesn't do that-- I have Unidecode simply drop the umlaut accent and give back "Schon". > (I chose this not because I'm a big meanie, but because generally changing "ü" to "ue" is disastrous for all text that's not in German. Finnish "Hyvää päivää" would turn into "Hyvaeae paeivaeae". And I discourage you from being yet another German who emails me, trying to impel me to consider a typographical nicety of German to be more important than all other languages.)
- vesinisa 8y ago
  What do you mean by a Western European language? Germany is usually considered geographically part of Central Europe. Finnish (and Estonian) are of course not Germanic nor even Indo-European languages. But OTOH Swedish, a Germanic language, also uses umlauts and considers them separate letters in collation (Finnish collation rules are actually adopted from Swedish). The most 'correct' mapping of Swedish "Lära sig höra" would be "Lara sig hora". Is Swedish not a Western European language?
- FabHK 8y ago
  Unidecode is a projection into the ASCII plane, not your favourite language specific preferred transliteration of certain characters. As such, calling it "definitely [...] wrong" is somewhat overblown, in particular since the author of the library explicitly addresses it.
- gumby 8y ago
  I agree with this -- simply dropping the umlaut is adequately clear, weird as it may look to me, just as dropping a circumflex is preferred to adding an 's' after it in French. Adding an e doesn't produce a reversible result (there are, for example, ordinary German words written with ae, oe, and ss). Every such projection is a lossy projection and is expected to possibly generate in an ambiguous result that must be interpreted through context. Hence the strange but comprehensible ,,Godel'' and ,,Malmo''. Cook is not claiming that his code replaces the need for Unicode! (There is a minor linguistic irony that the origin of the umlaut was scribe's shorthand when an E vowel inflection was turned into tiny E written above a letter, almost like a ligature, which became a pair of dots. But that character was then used in other languages differently, much as a loanword from a different language usually changes its meaning in the new language).
vesinisa 8y ago
> if you need to project (transliterate to ascii) Chinese unidecode is close to black magic That might be a dangerous assumption. The project page explicitly states: > Transliteration of languages like Chinese is a very complex issue and this library does not even attempt to address it. It draws the line at context-free character-by-character mapping. I.e. it is not black magic; just a mapping of Unicode characters to static ASCII transliterations. The results will certainly be incorrect in some contexts.
- waynecochran 8y ago
  Perhaps for some applications, e.g. hash keys, consistency is more important than correctness.
avian 8y ago
Unidecode maintainer here. I really need to add some commentary to that "Bei Jing" example in the README. Unidecode doesn't do language-specific transliteration and really works best for user-invisible things, like database identifiers or normalization. CJK characters in particular are very problematic, since they must be transliterated differently depending on the locale. Over the years I have received many angry mails from people that were deeply offended by an error in transliteration they saw in an URL or something. Unihandecode is a fork of Unidecode that tries to address this: https://github.com/miurahr/unihandecode https://github.com/miurahr/unihandecode
- lotyrin 8y ago
  One of those angry users here -- e.g. making flash card decks for Japanese on Memrise where they generate URLs based on some choice set of Chinese transliterations for no discernible reason.
- johndcook 8y ago
  Thank you for maintaining this package. I wish I'd run across it sooner.
mci 8y ago
Fredrik Lundh describes a DIY patch on top of unicodedata here: http://effbot.org/zone/unicode-convert.htm http://effbot.org/zone/unicode-convert.htm It solves the ä->ae problem and its ilk for Western European languages. Time has devoured my comment on that post, which extended the transliteration table to Eastern European languages and proposed to use mnemonic names like int(u'\N{Latin capital letter AE}') instead of 0xc6.
- akie 8y ago
  ä -> ae is a German transliteration that would not be recognized by a non-German (e.g. a Dutchman).
  - hjek 8y ago
    While a similar transliteration (ø-> oe) would be understood by Danes, it can create misunderstandings: køn (pretty) != koen (the cow), søn (son) != soen (the sow), røde (red) != roede (rowed), tør (dry) != toer (a two) and rør (pipe) != roer (rower).
    - Aloha 8y ago
      Context though is the missing piece to all of those, all languages, particularly english however is context driven.

titanix2 8y ago

Just fixing the French here: it’s Père Noël, not Pére Noël.

sametmax 8y ago
'Tin la honte.

kuschku 8y ago

Unidecode may handle Chinese fine, but it definitely handles Western European languages wrong. DIN 5007 Var. 2 specifies that for the purposes of sorting, ö is replaced with oe. This also applies to ä (ae), ü (ue) and ß (ss). This same replacement rule is also used on passports and IDs. Unidecode does not handle this correctly. The unidecode author wrote about this: > In German, there's the typographical convention that an umlaut (the double-dots on: ä ö ü) can be written as an "-e", like with "Schön" becoming "Schoen". But Unidecode doesn't do that-- I have Unidecode simply drop the umlaut accent and give back "Schon". > (I chose this not because I'm a big meanie, but because generally changing "ü" to "ue" is disastrous for all text that's not in German. Finnish "Hyvää päivää" would turn into "Hyvaeae paeivaeae". And I discourage you from being yet another German who emails me, trying to impel me to consider a typographical nicety of German to be more important than all other languages.)

vesinisa 8y ago
What do you mean by a Western European language? Germany is usually considered geographically part of Central Europe. Finnish (and Estonian) are of course not Germanic nor even Indo-European languages. But OTOH Swedish, a Germanic language, also uses umlauts and considers them separate letters in collation (Finnish collation rules are actually adopted from Swedish). The most 'correct' mapping of Swedish "Lära sig höra" would be "Lara sig hora". Is Swedish not a Western European language?
FabHK 8y ago
Unidecode is a projection into the ASCII plane, not your favourite language specific preferred transliteration of certain characters. As such, calling it "definitely [...] wrong" is somewhat overblown, in particular since the author of the library explicitly addresses it.
gumby 8y ago
I agree with this -- simply dropping the umlaut is adequately clear, weird as it may look to me, just as dropping a circumflex is preferred to adding an 's' after it in French. Adding an e doesn't produce a reversible result (there are, for example, ordinary German words written with ae, oe, and ss). Every such projection is a lossy projection and is expected to possibly generate in an ambiguous result that must be interpreted through context. Hence the strange but comprehensible ,,Godel'' and ,,Malmo''. Cook is not claiming that his code replaces the need for Unicode! (There is a minor linguistic irony that the origin of the umlaut was scribe's shorthand when an E vowel inflection was turned into tiny E written above a letter, almost like a ligature, which became a pair of dots. But that character was then used in other languages differently, much as a loanword from a different language usually changes its meaning in the new language).

vesinisa 8y ago

> if you need to project (transliterate to ascii) Chinese unidecode is close to black magic That might be a dangerous assumption. The project page explicitly states: > Transliteration of languages like Chinese is a very complex issue and this library does not even attempt to address it. It draws the line at context-free character-by-character mapping. I.e. it is not black magic; just a mapping of Unicode characters to static ASCII transliterations. The results will certainly be incorrect in some contexts.

waynecochran 8y ago
Perhaps for some applications, e.g. hash keys, consistency is more important than correctness.

avian 8y ago

Unidecode maintainer here. I really need to add some commentary to that "Bei Jing" example in the README. Unidecode doesn't do language-specific transliteration and really works best for user-invisible things, like database identifiers or normalization. CJK characters in particular are very problematic, since they must be transliterated differently depending on the locale. Over the years I have received many angry mails from people that were deeply offended by an error in transliteration they saw in an URL or something. Unihandecode is a fork of Unidecode that tries to address this: https://github.com/miurahr/unihandecode https://github.com/miurahr/unihandecode

lotyrin 8y ago
One of those angry users here -- e.g. making flash card decks for Japanese on Memrise where they generate URLs based on some choice set of Chinese transliterations for no discernible reason.
johndcook 8y ago
Thank you for maintaining this package. I wish I'd run across it sooner.

mci 8y ago

Fredrik Lundh describes a DIY patch on top of unicodedata here: http://effbot.org/zone/unicode-convert.htm http://effbot.org/zone/unicode-convert.htm It solves the ä->ae problem and its ilk for Western European languages. Time has devoured my comment on that post, which extended the transliteration table to Eastern European languages and proposed to use mnemonic names like int(u'\N{Latin capital letter AE}') instead of 0xc6.

akie 8y ago
ä -> ae is a German transliteration that would not be recognized by a non-German (e.g. a Dutchman).
- hjek 8y ago
  While a similar transliteration (ø-> oe) would be understood by Danes, it can create misunderstandings: køn (pretty) != koen (the cow), søn (son) != soen (the sow), røde (red) != roede (rowed), tør (dry) != toer (a two) and rør (pipe) != roer (rower).
  - Aloha 8y ago
    Context though is the missing piece to all of those, all languages, particularly english however is context driven.