4 ms·

Sometimes it makes sense to reduce to Unicode confusables. For example the Greek letter Big Alpha looks like uppercase A. Or some characters look very similar

Sometimes it makes sense to reduce to Unicode confusables. For example the Greek letter Big Alpha looks like uppercase A. Or some characters look very similar like the slash and the fraction slash. Yes, Unicode has separate scalar values for them. There are Open Source tools to handle confusables. This is in addition to the search specified by Unicode.

wanderingstan 3y ago
I wrote such a library for Python here: https://github.com/wanderingstan/Confusables https://github.com/wanderingstan/Confusables My use case was to thwart spammers in our company’s channels, but I suppose it could be used to also normalize accent encoding issues. Basically converts a phrase into a regular expression matching confusables. E.g. "ℍ℮1೦" would match "Hello"
- _nalply 3y ago
  Interesting. What would you think about this approach: reduce each character to a standard form which is the same for all characters in the same confusable group? Then match all search input to this standard form. This means "ℍ℮1l೦" is converted to "Hello" before searching, for example.
  - wanderingstan 3y ago
    It’s been a long time since I wrote this, but I think the issue with that approach is the possibility of one character being confusable with more than one letter. I.e. there may not be a single correct form to reduce to.
wyldfire 3y ago
> For example the Greek letter Big Alpha looks like uppercase A. If they're truly drawn the same (are they?) then why have a distinct encoding?
- adzm 3y ago
  > If they're truly drawn the same (are they?) then why have a distinct encoding? They may be drawn the same or similar in some typefaces but not all.
- schoen 3y ago
  One argument would be that you can apply functions to change their case. For example in Python >>> "Ᾰ̓ΡΕΤΉ".lower() 'ᾰ̓ρετή' >>> "AWESOME".lower() 'awesome' The Greek Α has lowercase form α, whereas the Roman A has lowercase form a. Another argument would be that you want a distinct encoding in order to be able to sort properly. Suppose we used the same codepoint (U+0050) for everything that looked like P. Then Greek Ρόδος would sort before Greek Δήλος because Roman P is numerically prior to Greek Δ in Unicode, even though Ρ comes later than Δ in the Greek alphabet.
  - mmoskal 3y ago
    Apparently this works very well, except for a single letter, Turkish I. Turkish has two version of 'i' and Unicode folks decided to use the Latin 'i' for lowercase dotted i, and Latin 'I' for uppercase dot-less I (and have two new code points for upper-case dotted I and lower-case dot-less I). Now, 'I'.lower() depends on your locale. A cause for a number of security exploits and lots of pain in regular expression engines. edit: Well, apparently 'I'.lower() doesn't depend on locale (so it's incorrect for Turkish languages); in JS you have to do 'I'.toLocaleLowerCase('tr-TR'). Regexps don't support it in neither.
- ninkendo 3y ago
  To me, it depends on what you think Unicode’s priorities should be. Let’s consider the opposite approach, that any letters that render the same should collapse to the same code point. What about Cherokee letter “go” (Ꭺ) versus the Latin A? What if they’re not precisely the same? Should lowercase l and capital I have the same encoding? What about the Roman numeral for 1 versus the letter I? Doesn’t it depend on the font too? How exactly do you draw the line? If Unicode sets out to say “no two letters that render the same shall ever have different encodings”, all it takes is one counterexample to break software. And I don’t think we’d ever get everyone to agree on whether certain letters should be distinct or not. Look at Han unification (and how poorly it was received) for examples of this. To me it’s much more sane to say that some written languages have visual overlap in their glyphs, and that’s to be expected, and if you want to prevent two similar looking strings from being confused with one another, you’re going to have to deploy an algorithm to de-dupe them. (Unicode even has an official list of this called “confusables”, devoted to helping you solve this.)

wanderingstan 3y ago

I wrote such a library for Python here: https://github.com/wanderingstan/Confusables https://github.com/wanderingstan/Confusables My use case was to thwart spammers in our company’s channels, but I suppose it could be used to also normalize accent encoding issues. Basically converts a phrase into a regular expression matching confusables. E.g. "ℍ℮1೦" would match "Hello"

_nalply 3y ago
Interesting. What would you think about this approach: reduce each character to a standard form which is the same for all characters in the same confusable group? Then match all search input to this standard form. This means "ℍ℮1l೦" is converted to "Hello" before searching, for example.
- wanderingstan 3y ago
  It’s been a long time since I wrote this, but I think the issue with that approach is the possibility of one character being confusable with more than one letter. I.e. there may not be a single correct form to reduce to.

wyldfire 3y ago

> For example the Greek letter Big Alpha looks like uppercase A. If they're truly drawn the same (are they?) then why have a distinct encoding?

adzm 3y ago
> If they're truly drawn the same (are they?) then why have a distinct encoding? They may be drawn the same or similar in some typefaces but not all.
schoen 3y ago
One argument would be that you can apply functions to change their case. For example in Python >>> "Ᾰ̓ΡΕΤΉ".lower() 'ᾰ̓ρετή' >>> "AWESOME".lower() 'awesome' The Greek Α has lowercase form α, whereas the Roman A has lowercase form a. Another argument would be that you want a distinct encoding in order to be able to sort properly. Suppose we used the same codepoint (U+0050) for everything that looked like P. Then Greek Ρόδος would sort before Greek Δήλος because Roman P is numerically prior to Greek Δ in Unicode, even though Ρ comes later than Δ in the Greek alphabet.
- mmoskal 3y ago
  Apparently this works very well, except for a single letter, Turkish I. Turkish has two version of 'i' and Unicode folks decided to use the Latin 'i' for lowercase dotted i, and Latin 'I' for uppercase dot-less I (and have two new code points for upper-case dotted I and lower-case dot-less I). Now, 'I'.lower() depends on your locale. A cause for a number of security exploits and lots of pain in regular expression engines. edit: Well, apparently 'I'.lower() doesn't depend on locale (so it's incorrect for Turkish languages); in JS you have to do 'I'.toLocaleLowerCase('tr-TR'). Regexps don't support it in neither.
ninkendo 3y ago
To me, it depends on what you think Unicode’s priorities should be. Let’s consider the opposite approach, that any letters that render the same should collapse to the same code point. What about Cherokee letter “go” (Ꭺ) versus the Latin A? What if they’re not precisely the same? Should lowercase l and capital I have the same encoding? What about the Roman numeral for 1 versus the letter I? Doesn’t it depend on the font too? How exactly do you draw the line? If Unicode sets out to say “no two letters that render the same shall ever have different encodings”, all it takes is one counterexample to break software. And I don’t think we’d ever get everyone to agree on whether certain letters should be distinct or not. Look at Han unification (and how poorly it was received) for examples of this. To me it’s much more sane to say that some written languages have visual overlap in their glyphs, and that’s to be expected, and if you want to prevent two similar looking strings from being confused with one another, you’re going to have to deploy an algorithm to de-dupe them. (Unicode even has an official list of this called “confusables”, devoted to helping you solve this.)