3 ms·

I was working on the remove diacritics feature of Drupal this weekend so I am quite fresh with these affairs. "Any self-respecting" -- this is very, very hard.

by chx 6y ago

I was working on the remove diacritics feature of Drupal this weekend so I am quite fresh with these affairs. "Any self-respecting" -- this is very, very hard. If you are Danish, your keyboard most likely has a way to enter Ø and you would expect Ø and O to be handled as two different letters. However, if an English article mentions the Øresund bridge, it's not unreasonable to expect a search on Oresund to match it. Or, in Hungarian, "kor" means period or age, "kór" means disease and "kör" means circle... but searching for "kor" in Chrome finds all of them. And Unicode won't save you: Ø does not have decomposition rules but ó and ö both do. So the common practice of decomposing - removing marks - recomposing route can either be too little or too much. The upcoming solution for Drupal will allow users to edit remove diacritics rules per language, no other way to do it. This is Libreoffice taking that path, I know Firefox does the same: https://cgit.freedesktop.org/libreoffice/core/commit/?h=libreoffice-5-3&id=18bc169b4727744f35227532078cbf7c9558bc9a https://cgit.freedesktop.org/libreoffice/core/commit/?h=libr... see the icu::Transliterator::createInstance("NFD; [:M:] Remove; NFC", part.

corty 6y ago
My ideal (but no one has implemented this yet) search would have a slider for strict/lax. In strict mode, only equivalent forms would match, so composed and decomposed would match as per unicode rules, but nothing else. As you get laxer, you obey locale-specific replacements like ö -> oe in German or å -> aa in Danish. Even more lax one could have homoglyphs match, so latin a would match kyrillic а. Then maybe α matches a and β matches b and ß, and of course C matches ℂ and weirdness like that. That already sounds like a few years of research to get it working world-wide. In the last stage, a full DWIM-AI would do the matches. ;)
- mcswell 6y ago
  Some of this (Latin and Cyrillic A/a, for example) is relevant to e.g. Internationalized Domain Name issues (https://en.wikipedia.org/wiki/IDN_homograph_attack https://en.wikipedia.org/wiki/IDN_homograph_attack). I once worked on what we called DYM tools (for "Did You Mean"). The goal was to assist native English speakers who were learning a second language find words they heard, in that language's electronic dictionary. We knew, for example, that native speakers of English have difficulty distinguishing the dental and retroflex consonants of Hindi, so the DYM allowed mismatches of those. A perfect spelling was at the top of the list returned from the dictionary, while a misspelling--such as writing a dental for a retroflex--resulted in the mismatched word being a little lower in the list. We tailored our DYMs for particular target languages (and always for native speakers of English). As you say, getting something like this to work for multiple languages at the same time would be difficult.

by chx 6y ago

corty 6y ago

My ideal (but no one has implemented this yet) search would have a slider for strict/lax. In strict mode, only equivalent forms would match, so composed and decomposed would match as per unicode rules, but nothing else. As you get laxer, you obey locale-specific replacements like ö -> oe in German or å -> aa in Danish. Even more lax one could have homoglyphs match, so latin a would match kyrillic а. Then maybe α matches a and β matches b and ß, and of course C matches ℂ and weirdness like that. That already sounds like a few years of research to get it working world-wide. In the last stage, a full DWIM-AI would do the matches. ;)

mcswell 6y ago
Some of this (Latin and Cyrillic A/a, for example) is relevant to e.g. Internationalized Domain Name issues (https://en.wikipedia.org/wiki/IDN_homograph_attack https://en.wikipedia.org/wiki/IDN_homograph_attack). I once worked on what we called DYM tools (for "Did You Mean"). The goal was to assist native English speakers who were learning a second language find words they heard, in that language's electronic dictionary. We knew, for example, that native speakers of English have difficulty distinguishing the dental and retroflex consonants of Hindi, so the DYM allowed mismatches of those. A perfect spelling was at the top of the list returned from the dictionary, while a misspelling--such as writing a dental for a retroflex--resulted in the mismatched word being a little lower in the list. We tailored our DYMs for particular target languages (and always for native speakers of English). As you say, getting something like this to work for multiple languages at the same time would be difficult.