3 ms·

In the case of Unicode at least, the Unicode consortium maintains a database of "confusable" characters and a tool to detect them: https://util.unicode.org/Unic

In the case of Unicode at least, the Unicode consortium maintains a database of "confusable" characters and a tool to detect them: https://util.unicode.org/UnicodeJsps/confusables.jsp?a=%E2%80%A6&r=None https://util.unicode.org/UnicodeJsps/confusables.jsp?a=%E2%8... You can download the database for use in your own programs, and there is at least one Python package built around it: https://pypi.org/project/confusables/ https://pypi.org/project/confusables/

IanCal 4y ago
If you've got borked encodings around as well, the python package ftfy is wonderful: https://pypi.org/project/ftfy/ https://pypi.org/project/ftfy/ Undoes whatever on earth it is excel does, helps clean up bits of html/etc.
carterschonwald 4y ago
Is there a schema document explaining the format of the dataset? Edit Found it in the associated doc It’s a cute approach ! Data File Format Each line in the data file has the following format: Field 1 is the source, Field 2 is the target, and Field 3 is obsolete, always containing the letters “MA” for backwards compatibility. For example: 0441 ; 0063 ; MA # ( с → c ) CYRILLIC SMALL LETTER ES → LATIN SMALL LETTER C # 2CA5 ; 0063 ; MA # ( ⲥ → c ) COPTIC SMALL LETTER SIMA → LATIN SMALL LETTER C # →ϲ→ Everything after the # is a comment and is purely informative. A asterisk after the comment indicates that the character is not an XID character [UAX31]. The comments provide the character names. Implementations that use the confusable data do not have to recursively apply the mappings, because the transforms are idempotent. That is, skeleton(skeleton(X)) = skeleton(X)