5 ms·

Unicode Utilities: Confusables

97 points by simonpure 4y ago

jwilk 4y ago
ICU has API for confusables: https://unicode-org.github.io/icu-docs/apidoc/released/icu4j/com/ibm/icu/text/SpoofChecker.html https://unicode-org.github.io/icu-docs/apidoc/released/icu4j... https://unicode-org.github.io/icu-docs/apidoc/released/icu4c/uspoof_8h.html https://unicode-org.github.io/icu-docs/apidoc/released/icu4c... PyICU (≥ 1.9) has bindings for this (class icu.SpoofChecker).
jancsika 4y ago
Too bad there isn't a standard lib that will flag any non-ASCII characters to be rendered big like those oversized sprites in the "giant" level of Super Mario 3. So you scroll through source code and hit this weirdly spaced line with a big "h" character and go-- oh-- that's some buggy crap in there. Then maybe have a map for ranges that you want to include in your allowable set, and we're all good to go. :)
- jabbany 4y ago
  I think the rust compiler kind of does this: https://mobile.twitter.com/hmemcpy/status/1151890968477667328?lang=en https://mobile.twitter.com/hmemcpy/status/115189096847766732...
  - jancsika 4y ago
    Ooh, good job, Rust!
  - jeroenhd 4y ago
    Only a partial set, though: https://github.com/rust-lang/rust/blob/master/compiler/rustc_parse/src/lexer/unicode_chars.rs https://github.com/rust-lang/rust/blob/master/compiler/rustc...
    - estebank 4y ago
      That's only the lexer check. We have further lints that run after parsing to detect Unicode confusables in identifiers as well, as well as two lints for text flow codepoints that can change what the code looks like in editors (one for comments and one for string literals).
      - rurban 4y ago
        Yeah, but Rust does it properly by following UTS-39, and not blindly checking for just confusable matches.
- II2II 4y ago
  Better yet, highlight characters out of context. Not everyone writes in languages that are fully representable in ASCII, yet confusable characters are still an issue.
gcau 4y ago
If its of interest, I'm the author of the javascript/npm package 'confusables', which does the same thing (and also in the reverse) Source: https://github.com/gc/confusables https://github.com/gc/confusables Demo: https://confusables.gc.codes/ https://confusables.gc.codes/
danbruc 4y ago
Total raw values: 148,068,974,592,000 Too many raw items to process. Any good guesses what this is about? EDIT: Got it, the number of strings confusable with the input.
hilbert42 4y ago
Potentially this is a big problem. Especially with OCR, transliterations from and across different languages, characters missing their diacriticals, etc. Whilst I was aware of the problem I wasn't aware that it's as big as it is. Those of us who use Latin scripts are reasonably familiar with the common ones such as o, O and 0 [l/c, u/c alpha & zero], but there's some tricky ones even in these Latin scripts that many of us get wrong. As I discovered a while back - but I can't remember how - most of us get the World War Two abbreviation WWII wrong (myself included). The 'II' is not two alpha characters as we almost inevitability use but rather it should be the Unicode characters for Roman numerals. Even then I cannot remember if the correct transliteration for Arabic numeral 2 is supposed to to be Roman numeral '1' used twice/repeated or if Roman numeral '2' actually has its own Roman numerical glyph (I suspect the latter is correct). There are many more instances like this too, the dash, minus sign, en and em dash for instance. Yes, I could look them up but it's a nuisance to do so on-the-fly and that's the whole point/trouble. It seems to me we need much better proofing tools that would flag errors or potential errors. I reckon we've been very poorly served in this regard in that there no simple software tools available of the quality we need. It's not only symbols or characters we need to correct but also typos that don't show up on spelling checkers such as for and fro and the big troublemakers it's and its. Spelling and grammar checkers should automatically highlight or flag such words whether their usage is correct or not.
- nerdponx 4y ago
  Unicode does have both upper- and lower-case Roman numerals, U+2160 to U+2188. But I doubt that the Romans had any clearer of a sense than you of whether "II" was one letter or two. It probably varied across eras and regions, and most people probably didn't think one way or another about it. It's probably not such a big deal to write II when you meant U+2161. However, I definitely do agree that we are badly lacking in good software tools to help us enrich the set of letters that we can easily use. Imagine if you could type "II" and your text editor could offer to replace it with "Ⅱ". On the other hand, font support is a limiting factor. For most purposes, II should looks similar to Ⅱ. But in our current world, you are at the mercy of whatever fonts that happen to implement Ⅱ, which might be totally different from how II looks.
jonstewart 4y ago
Confusables are used by attackers to make malicious domains and URLs apppear innocuous. If you show such things to users, it’s good to highlight confusables.
raffy 4y ago
Unfortunately, this is only single-character confusables. There are an enormous number of permutations missing (ѐ [450] vs è [E8]). The confusable matching isn't even reflexive (A confuse B, but B doesn't confuse A). I've been developing a normalization library for Ethereum Name Service: https://adraffy.github.io/ens-normalize.js/test/resolver.html#Mab%C3%81%EF%B8%8E%CC%A5%CC%A6%C2%AD%F0%9F%88%9A%EF%B8%8Fa.eth https://adraffy.github.io/ens-normalize.js/test/resolver.htm...
- dhosek 4y ago
  There are the NKFC and NKFD normalizations which can resolve most of the confusing things so that, e.g., Ⅵ will become VI, Å will become Å, etc. although it doesn’t resolve B and В to the same character.
- petesergeant 4y ago
  I wonder at what point you need to give up and just OCR a rendering
- leni536 4y ago
  Honestly, for a brand new name service I would just stick to ASCII to sidestep the whole issue.
  - thematrixturtle 4y ago
    Because fuck those people who want to use non-English names, amirite? Seriously, it's 2022, we have better solutions than ASCII by now. And for what it's worthful, even in ASCII some chars like l, I and 1 are quite confusable.
    - cmroanirgo 4y ago
      DNS support is no where near Unicode. At best we get local language support for non-ASCII domains, but everyone else will see it as punycode. Of course, phishing is one good reason why it's this. > Internationalized domain names are stored in the Domain Name System (DNS) as ASCII strings using Punycode transcription. https://en.wikipedia.org/wiki/Internationalized_domain_name https://en.wikipedia.org/wiki/Internationalized_domain_name https://en.wikipedia.org/wiki/Punycode https://en.wikipedia.org/wiki/Punycode
mikotodomo 4y ago
Isn't it dangerous to be posting this? Sorry if this is a stupid question.
- shaggie76 4y ago
  Spam bots weaponized this long ago I'm afraid.
gumby 4y ago
One useful table that AFAIK isn’t in the Unicode consortium’s tables is “characters that exist primarily or only for round trip compatibility”. An example would be the “Roman numerals” set that are discussed elsewhere in this HN discussion. Others include “ghost characters” and obsolete keitai emojis from the 90s that only appear in obsolete character tables. Mostly these should be normalised to modern / canonical versions.
rurban 4y ago
Note that this table is of extremely bad quality and can NOT be used in production without patches. See https://github.com/rurban/libu8ident/#confusables https://github.com/rurban/libu8ident/#confusables
wanderingstan 4y ago
Years ago I created a Python library for generating regex to match confusables: https://github.com/wanderingstan/Confusables https://github.com/wanderingstan/Confusables For example, "𝓗℮𝐥1೦" matches "Hello". "Hello" gets turned into the following regex of character classes: [H\Ｈ\ℋ\ℌ\ℍ\𝐇\𝐻\𝑯\𝓗\𝕳\𝖧\𝗛\𝘏\𝙃\𝙷\Η\𝚮\𝛨\𝜢\𝝜\𝞖\Ⲏ\Н\Ꮋ\ᕼ\ꓧ\𐋏\Ⱨ\Ң\Ħ\Ӊ\Ӈ] [e\℮\ｅ\ℯ\ⅇ\𝐞\𝑒\𝒆\𝓮\𝔢\𝕖\𝖊\𝖾\𝗲\𝘦\𝙚\𝚎\ꬲ\е\ҽ\ɇ\ҿ] [l\‎\|\∣\⏽\￨1\‎\۱\𐌠\‎\𝟏\𝟙\𝟣\𝟭\𝟷I\Ｉ\Ⅰ\ℐ\ℑ\𝐈\𝐼\𝑰\𝓘\𝕀\𝕴\𝖨\𝗜\𝘐\𝙄\𝙸\Ɩ\ｌ\ⅼ\ℓ\𝐥\𝑙\𝒍\𝓁\𝓵\𝔩\𝕝\𝖑\𝗅\𝗹\𝘭\𝙡\𝚕\ǀ\Ι\𝚰\𝛪\𝜤\𝝞\𝞘\Ⲓ\І\Ӏ\‎\‎\‎\‎\‎\‎\‎\‎\ⵏ\ᛁ\ꓲ\𖼨\𐊊\𐌉\‎\‎\ł\ɭ\Ɨ\ƚ\ɫ\‎\‎\‎\‎\ŀ\Ŀ\ᒷ\🄂\⒈\‎\⒓\㏫\㋋\㍤\⒔\㏬\㍥\⒕\㏭\㍦\⒖\㏮\㍧\⒗\㏯\㍨\⒘\㏰\㍩\⒙\㏱\㍪\⒚\㏲\㍫\ǉ\Ĳ\‖\∥\Ⅱ\ǁ\‎\𐆙\⒒\Ⅲ\𐆘\㏪\㋊\㍣\Ю\⒑\㏩\㋉\㍢\ʪ\₶\Ⅳ\Ⅸ\ɮ\ʫ\㏠\㋀\㍙] [l\‎\|\∣\⏽\￨1\‎\۱\𐌠\‎\𝟏\𝟙\𝟣\𝟭\𝟷I\Ｉ\Ⅰ\ℐ\ℑ\𝐈\𝐼\𝑰\𝓘\𝕀\𝕴\𝖨\𝗜\𝘐\𝙄\𝙸\Ɩ\ｌ\ⅼ\ℓ\𝐥\𝑙\𝒍\𝓁\𝓵\𝔩\𝕝\𝖑\𝗅\𝗹\𝘭\𝙡\𝚕\ǀ\Ι\𝚰\𝛪\𝜤\𝝞\𝞘\Ⲓ\І\Ӏ\‎\‎\‎\‎\‎\‎\‎\‎\ⵏ\ᛁ\ꓲ\𖼨\𐊊\𐌉\‎\‎\ł\ɭ\Ɨ\ƚ\ɫ\‎\‎\‎\‎\ŀ\Ŀ\ᒷ\🄂\⒈\‎\⒓\㏫\㋋\㍤\⒔\㏬\㍥\⒕\㏭\㍦\⒖\㏮\㍧\⒗\㏯\㍨\⒘\㏰\㍩\⒙\㏱\㍪\⒚\㏲\㍫\ǉ\Ĳ\‖\∥\Ⅱ\ǁ\‎\𐆙\⒒\Ⅲ\𐆘\㏪\㋊\㍣\Ю\⒑\㏩\㋉\㍢\ʪ\₶\Ⅳ\Ⅸ\ɮ\ʫ\㏠\㋀\㍙] [o\ం\ಂ\ം\ං\०\੦\૦\௦\౦\೦\൦\๐\໐\၀\‎\۵\ｏ\ℴ\𝐨\𝑜\𝒐\𝓸\𝔬\𝕠\𝖔\𝗈\𝗼\𝘰\𝙤\𝚘\ᴏ\ᴑ\ꬽ\ο\𝛐\𝜊\𝝄\𝝾\𝞸\σ\𝛔\𝜎\𝝈\𝞂\𝞼\ⲟ\о\ჿ\օ\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\ഠ\ဝ\𐓪\𑣈\𑣗\𐐬\‎\ø\ꬾ\ɵ\ꝋ\ө\ѳ\ꮎ\ꮻ\ꭴ\‎\ơ\œ\ɶ\∞\ꝏ\ꚙ\ൟ\တ]
- deleted 4y ago
  [deleted]
- jwilk 4y ago
  These backslashes don't look right.
  - wanderingstan 4y ago
    You're right. The code was correct but the Readme hadn't been updated to reflect the lack of backslashes. Thanks for the find! Commit: https://github.com/wanderingstan/Confusables/commit/c41214285c987aa54c7cf7caa03a995dc5e8fc21 https://github.com/wanderingstan/Confusables/commit/c4121428...
deleted 4y ago
[deleted]

97 points by simonpure 4y ago

ICU has API for confusables: https://unicode-org.github.io/icu-docs/apidoc/released/icu4j/com/ibm/icu/text/SpoofChecker.html https://unicode-org.github.io/icu-docs/apidoc/released/icu4j... https://unicode-org.github.io/icu-docs/apidoc/released/icu4c/uspoof_8h.html https://unicode-org.github.io/icu-docs/apidoc/released/icu4c... PyICU (≥ 1.9) has bindings for this (class icu.SpoofChecker).

jancsika 4y ago

Too bad there isn't a standard lib that will flag any non-ASCII characters to be rendered big like those oversized sprites in the "giant" level of Super Mario 3. So you scroll through source code and hit this weirdly spaced line with a big "h" character and go-- oh-- that's some buggy crap in there. Then maybe have a map for ranges that you want to include in your allowable set, and we're all good to go. :)

jabbany 4y ago
I think the rust compiler kind of does this: https://mobile.twitter.com/hmemcpy/status/1151890968477667328?lang=en https://mobile.twitter.com/hmemcpy/status/115189096847766732...
- jancsika 4y ago
  Ooh, good job, Rust!
- jeroenhd 4y ago
  Only a partial set, though: https://github.com/rust-lang/rust/blob/master/compiler/rustc_parse/src/lexer/unicode_chars.rs https://github.com/rust-lang/rust/blob/master/compiler/rustc...
  - estebank 4y ago
    That's only the lexer check. We have further lints that run after parsing to detect Unicode confusables in identifiers as well, as well as two lints for text flow codepoints that can change what the code looks like in editors (one for comments and one for string literals).
    - rurban 4y ago
      Yeah, but Rust does it properly by following UTS-39, and not blindly checking for just confusable matches.
II2II 4y ago
Better yet, highlight characters out of context. Not everyone writes in languages that are fully representable in ASCII, yet confusable characters are still an issue.

gcau 4y ago

If its of interest, I'm the author of the javascript/npm package 'confusables', which does the same thing (and also in the reverse) Source: https://github.com/gc/confusables https://github.com/gc/confusables Demo: https://confusables.gc.codes/ https://confusables.gc.codes/

danbruc 4y ago

Total raw values: 148,068,974,592,000 Too many raw items to process. Any good guesses what this is about? EDIT: Got it, the number of strings confusable with the input.

hilbert42 4y ago

Potentially this is a big problem. Especially with OCR, transliterations from and across different languages, characters missing their diacriticals, etc. Whilst I was aware of the problem I wasn't aware that it's as big as it is. Those of us who use Latin scripts are reasonably familiar with the common ones such as o, O and 0 [l/c, u/c alpha & zero], but there's some tricky ones even in these Latin scripts that many of us get wrong. As I discovered a while back - but I can't remember how - most of us get the World War Two abbreviation WWII wrong (myself included). The 'II' is not two alpha characters as we almost inevitability use but rather it should be the Unicode characters for Roman numerals. Even then I cannot remember if the correct transliteration for Arabic numeral 2 is supposed to to be Roman numeral '1' used twice/repeated or if Roman numeral '2' actually has its own Roman numerical glyph (I suspect the latter is correct). There are many more instances like this too, the dash, minus sign, en and em dash for instance. Yes, I could look them up but it's a nuisance to do so on-the-fly and that's the whole point/trouble. It seems to me we need much better proofing tools that would flag errors or potential errors. I reckon we've been very poorly served in this regard in that there no simple software tools available of the quality we need. It's not only symbols or characters we need to correct but also typos that don't show up on spelling checkers such as for and fro and the big troublemakers it's and its. Spelling and grammar checkers should automatically highlight or flag such words whether their usage is correct or not.

nerdponx 4y ago
Unicode does have both upper- and lower-case Roman numerals, U+2160 to U+2188. But I doubt that the Romans had any clearer of a sense than you of whether "II" was one letter or two. It probably varied across eras and regions, and most people probably didn't think one way or another about it. It's probably not such a big deal to write II when you meant U+2161. However, I definitely do agree that we are badly lacking in good software tools to help us enrich the set of letters that we can easily use. Imagine if you could type "II" and your text editor could offer to replace it with "Ⅱ". On the other hand, font support is a limiting factor. For most purposes, II should looks similar to Ⅱ. But in our current world, you are at the mercy of whatever fonts that happen to implement Ⅱ, which might be totally different from how II looks.

jonstewart 4y ago

Confusables are used by attackers to make malicious domains and URLs apppear innocuous. If you show such things to users, it’s good to highlight confusables.

raffy 4y ago

Unfortunately, this is only single-character confusables. There are an enormous number of permutations missing (ѐ [450] vs è [E8]). The confusable matching isn't even reflexive (A confuse B, but B doesn't confuse A). I've been developing a normalization library for Ethereum Name Service: https://adraffy.github.io/ens-normalize.js/test/resolver.html#Mab%C3%81%EF%B8%8E%CC%A5%CC%A6%C2%AD%F0%9F%88%9A%EF%B8%8Fa.eth https://adraffy.github.io/ens-normalize.js/test/resolver.htm...

dhosek 4y ago
There are the NKFC and NKFD normalizations which can resolve most of the confusing things so that, e.g., Ⅵ will become VI, Å will become Å, etc. although it doesn’t resolve B and В to the same character.
petesergeant 4y ago
I wonder at what point you need to give up and just OCR a rendering
leni536 4y ago
Honestly, for a brand new name service I would just stick to ASCII to sidestep the whole issue.
- thematrixturtle 4y ago
  Because fuck those people who want to use non-English names, amirite? Seriously, it's 2022, we have better solutions than ASCII by now. And for what it's worthful, even in ASCII some chars like l, I and 1 are quite confusable.
  - cmroanirgo 4y ago
    DNS support is no where near Unicode. At best we get local language support for non-ASCII domains, but everyone else will see it as punycode. Of course, phishing is one good reason why it's this. > Internationalized domain names are stored in the Domain Name System (DNS) as ASCII strings using Punycode transcription. https://en.wikipedia.org/wiki/Internationalized_domain_name https://en.wikipedia.org/wiki/Internationalized_domain_name https://en.wikipedia.org/wiki/Punycode https://en.wikipedia.org/wiki/Punycode

mikotodomo 4y ago

Isn't it dangerous to be posting this? Sorry if this is a stupid question.

shaggie76 4y ago
Spam bots weaponized this long ago I'm afraid.

gumby 4y ago

One useful table that AFAIK isn’t in the Unicode consortium’s tables is “characters that exist primarily or only for round trip compatibility”. An example would be the “Roman numerals” set that are discussed elsewhere in this HN discussion. Others include “ghost characters” and obsolete keitai emojis from the 90s that only appear in obsolete character tables. Mostly these should be normalised to modern / canonical versions.

rurban 4y ago

Note that this table is of extremely bad quality and can NOT be used in production without patches. See https://github.com/rurban/libu8ident/#confusables https://github.com/rurban/libu8ident/#confusables

wanderingstan 4y ago

Years ago I created a Python library for generating regex to match confusables: https://github.com/wanderingstan/Confusables https://github.com/wanderingstan/Confusables For example, "𝓗℮𝐥1೦" matches "Hello". "Hello" gets turned into the following regex of character classes: [H\Ｈ\ℋ\ℌ\ℍ\𝐇\𝐻\𝑯\𝓗\𝕳\𝖧\𝗛\𝘏\𝙃\𝙷\Η\𝚮\𝛨\𝜢\𝝜\𝞖\Ⲏ\Н\Ꮋ\ᕼ\ꓧ\𐋏\Ⱨ\Ң\Ħ\Ӊ\Ӈ] [e\℮\ｅ\ℯ\ⅇ\𝐞\𝑒\𝒆\𝓮\𝔢\𝕖\𝖊\𝖾\𝗲\𝘦\𝙚\𝚎\ꬲ\е\ҽ\ɇ\ҿ] [l\‎\|\∣\⏽\￨1\‎\۱\𐌠\‎\𝟏\𝟙\𝟣\𝟭\𝟷I\Ｉ\Ⅰ\ℐ\ℑ\𝐈\𝐼\𝑰\𝓘\𝕀\𝕴\𝖨\𝗜\𝘐\𝙄\𝙸\Ɩ\ｌ\ⅼ\ℓ\𝐥\𝑙\𝒍\𝓁\𝓵\𝔩\𝕝\𝖑\𝗅\𝗹\𝘭\𝙡\𝚕\ǀ\Ι\𝚰\𝛪\𝜤\𝝞\𝞘\Ⲓ\І\Ӏ\‎\‎\‎\‎\‎\‎\‎\‎\ⵏ\ᛁ\ꓲ\𖼨\𐊊\𐌉\‎\‎\ł\ɭ\Ɨ\ƚ\ɫ\‎\‎\‎\‎\ŀ\Ŀ\ᒷ\🄂\⒈\‎\⒓\㏫\㋋\㍤\⒔\㏬\㍥\⒕\㏭\㍦\⒖\㏮\㍧\⒗\㏯\㍨\⒘\㏰\㍩\⒙\㏱\㍪\⒚\㏲\㍫\ǉ\Ĳ\‖\∥\Ⅱ\ǁ\‎\𐆙\⒒\Ⅲ\𐆘\㏪\㋊\㍣\Ю\⒑\㏩\㋉\㍢\ʪ\₶\Ⅳ\Ⅸ\ɮ\ʫ\㏠\㋀\㍙] [l\‎\|\∣\⏽\￨1\‎\۱\𐌠\‎\𝟏\𝟙\𝟣\𝟭\𝟷I\Ｉ\Ⅰ\ℐ\ℑ\𝐈\𝐼\𝑰\𝓘\𝕀\𝕴\𝖨\𝗜\𝘐\𝙄\𝙸\Ɩ\ｌ\ⅼ\ℓ\𝐥\𝑙\𝒍\𝓁\𝓵\𝔩\𝕝\𝖑\𝗅\𝗹\𝘭\𝙡\𝚕\ǀ\Ι\𝚰\𝛪\𝜤\𝝞\𝞘\Ⲓ\І\Ӏ\‎\‎\‎\‎\‎\‎\‎\‎\ⵏ\ᛁ\ꓲ\𖼨\𐊊\𐌉\‎\‎\ł\ɭ\Ɨ\ƚ\ɫ\‎\‎\‎\‎\ŀ\Ŀ\ᒷ\🄂\⒈\‎\⒓\㏫\㋋\㍤\⒔\㏬\㍥\⒕\㏭\㍦\⒖\㏮\㍧\⒗\㏯\㍨\⒘\㏰\㍩\⒙\㏱\㍪\⒚\㏲\㍫\ǉ\Ĳ\‖\∥\Ⅱ\ǁ\‎\𐆙\⒒\Ⅲ\𐆘\㏪\㋊\㍣\Ю\⒑\㏩\㋉\㍢\ʪ\₶\Ⅳ\Ⅸ\ɮ\ʫ\㏠\㋀\㍙] [o\ం\ಂ\ം\ං\०\੦\૦\௦\౦\೦\൦\๐\໐\၀\‎\۵\ｏ\ℴ\𝐨\𝑜\𝒐\𝓸\𝔬\𝕠\𝖔\𝗈\𝗼\𝘰\𝙤\𝚘\ᴏ\ᴑ\ꬽ\ο\𝛐\𝜊\𝝄\𝝾\𝞸\σ\𝛔\𝜎\𝝈\𝞂\𝞼\ⲟ\о\ჿ\օ\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\‎\ഠ\ဝ\𐓪\𑣈\𑣗\𐐬\‎\ø\ꬾ\ɵ\ꝋ\ө\ѳ\ꮎ\ꮻ\ꭴ\‎\ơ\œ\ɶ\∞\ꝏ\ꚙ\ൟ\တ]

deleted 4y ago
[deleted]
jwilk 4y ago
These backslashes don't look right.
- wanderingstan 4y ago
  You're right. The code was correct but the Readme hadn't been updated to reflect the lack of backslashes. Thanks for the find! Commit: https://github.com/wanderingstan/Confusables/commit/c41214285c987aa54c7cf7caa03a995dc5e8fc21 https://github.com/wanderingstan/Confusables/commit/c4121428...

deleted 4y ago

[deleted]