3 ms·

Unidecode may handle Chinese fine, but it definitely handles Western European languages wrong. DIN 5007 Var. 2 specifies that for the purposes of sorting, ö is

Unidecode may handle Chinese fine, but it definitely handles Western European languages wrong. DIN 5007 Var. 2 specifies that for the purposes of sorting, ö is replaced with oe. This also applies to ä (ae), ü (ue) and ß (ss). This same replacement rule is also used on passports and IDs. Unidecode does not handle this correctly. The unidecode author wrote about this: > In German, there's the typographical convention that an umlaut (the double-dots on: ä ö ü) can be written as an "-e", like with "Schön" becoming "Schoen". But Unidecode doesn't do that-- I have Unidecode simply drop the umlaut accent and give back "Schon". > (I chose this not because I'm a big meanie, but because generally changing "ü" to "ue" is disastrous for all text that's not in German. Finnish "Hyvää päivää" would turn into "Hyvaeae paeivaeae". And I discourage you from being yet another German who emails me, trying to impel me to consider a typographical nicety of German to be more important than all other languages.)

vesinisa 8y ago
What do you mean by a Western European language? Germany is usually considered geographically part of Central Europe. Finnish (and Estonian) are of course not Germanic nor even Indo-European languages. But OTOH Swedish, a Germanic language, also uses umlauts and considers them separate letters in collation (Finnish collation rules are actually adopted from Swedish). The most 'correct' mapping of Swedish "Lära sig höra" would be "Lara sig hora". Is Swedish not a Western European language?
FabHK 8y ago
Unidecode is a projection into the ASCII plane, not your favourite language specific preferred transliteration of certain characters. As such, calling it "definitely [...] wrong" is somewhat overblown, in particular since the author of the library explicitly addresses it.
gumby 8y ago
I agree with this -- simply dropping the umlaut is adequately clear, weird as it may look to me, just as dropping a circumflex is preferred to adding an 's' after it in French. Adding an e doesn't produce a reversible result (there are, for example, ordinary German words written with ae, oe, and ss). Every such projection is a lossy projection and is expected to possibly generate in an ambiguous result that must be interpreted through context. Hence the strange but comprehensible ,,Godel'' and ,,Malmo''. Cook is not claiming that his code replaces the need for Unicode! (There is a minor linguistic irony that the origin of the umlaut was scribe's shorthand when an E vowel inflection was turned into tiny E written above a letter, almost like a ligature, which became a pair of dots. But that character was then used in other languages differently, much as a loanword from a different language usually changes its meaning in the new language).

vesinisa 8y ago

What do you mean by a Western European language? Germany is usually considered geographically part of Central Europe. Finnish (and Estonian) are of course not Germanic nor even Indo-European languages. But OTOH Swedish, a Germanic language, also uses umlauts and considers them separate letters in collation (Finnish collation rules are actually adopted from Swedish). The most 'correct' mapping of Swedish "Lära sig höra" would be "Lara sig hora". Is Swedish not a Western European language?

FabHK 8y ago

Unidecode is a projection into the ASCII plane, not your favourite language specific preferred transliteration of certain characters. As such, calling it "definitely [...] wrong" is somewhat overblown, in particular since the author of the library explicitly addresses it.

gumby 8y ago

I agree with this -- simply dropping the umlaut is adequately clear, weird as it may look to me, just as dropping a circumflex is preferred to adding an 's' after it in French. Adding an e doesn't produce a reversible result (there are, for example, ordinary German words written with ae, oe, and ss). Every such projection is a lossy projection and is expected to possibly generate in an ambiguous result that must be interpreted through context. Hence the strange but comprehensible ,,Godel'' and ,,Malmo''. Cook is not claiming that his code replaces the need for Unicode! (There is a minor linguistic irony that the origin of the umlaut was scribe's shorthand when an E vowel inflection was turned into tiny E written above a letter, almost like a ligature, which became a pair of dots. But that character was then used in other languages differently, much as a loanword from a different language usually changes its meaning in the new language).