2 ms·
For languages other than English (or even for English with some accented characters), you'll need to do a bit better than just spotting non-ASCII characters. Fi
by yorwba 9y ago
For languages other than English (or even for English with some accented characters), you'll need to do a bit better than just spotting non-ASCII characters. Finding zero-width/not fully normalized characters would be a bit more robust, but still not perfect. IIRC, the zero-width joiner/non-joiner are important for rendering various Indian languages correctly. But I guess it depends on your use case, and how many false positives you can tolerate.
- jancsika 9y agoWhat I'd want for most cases is to "paste as printable ASCII." That covers a large number of use cases (the article's example is printable ASCII ostensibly to reach the largest audience). I'm not sure what to do with more complicated language mixing, but I suppose a script to tell you what languages are being used and where there might be "out-of-place" character codes would help.
- jraph 9y agoSolving this seems pretty similar to how browsers handle phishing related issues with International Domain Names. https://en.wikipedia.org/wiki/IDN_homograph_attack https://en.wikipedia.org/wiki/IDN_homograph_attack