4 ms·
In fact, the Unicode consortium provides a report and extensive list of "confusable" symbols, which you could use alongside Unicode normalization tables to map
by jffry 5y ago
In fact, the Unicode consortium provides a report and extensive list of "confusable" symbols, which you could use alongside Unicode normalization tables to map adversarial back into more ASCII-equivalent text before running it through anti-spam mechanisms that are interested in the content of the message.
https://www.unicode.org/reports/tr39/ https://www.unicode.org/reports/tr39/
- wanderingstan 5y agoI built a Python library for finding strings obfuscated this way. Was critical when moderating our telegram channel before an ICO. https://github.com/wanderingstan/Confusables https://github.com/wanderingstan/Confusables E.g. "𝓗℮𝐥1೦" would match "Hello"
- Alir3z4 5y agoI should have had better eyes while searching, could find this and saved some hours.
- jffry 5y agoI only learned about it myself after spending too long building my own half-baked version. I think it's in pretty opaque language that makes it hard to find even if you know what you want. Maybe somebody else on here will see it and learn about it before they need it, and at least you still have a new tool to reach for in the future.
- loco053100 5y agoMein Handy is voller kacke hacking Apps. Mein Handy wird seit 2004 gehackt ihr seit kleine Maden doch die Maden kommen jetzt zum Anwalt ich lasse mir das nicht mehr bitten. Ich habe genug was ich brauche! Habt ihr kein Leben anscheinend armselig
- airstrike 5y agoYou should post this to Show HN. Also you have a typo on your README ("characgters")
- wanderingstan 5y agoThanks, I've fixed the typo! It was such a simple project, hardly seems worthy of a "Show HN".
- jonplackett 5y agoI've seen crazier things get to #1
- 8note 5y agoTest for the library: would it catch that that typo still refers to characters?
- jart 5y agoThe one built-in to Python will get you most of the way there: >>> import unicodedata >>> unicodedata.normalize('NFKD', '𝓗℮𝐥1೦𝗵𝗲𝗹𝗹𝗼') 'H℮l1೦hello' Obviously it isn't going to remap leetspeak characters like 1 -> l but it covers a lot of cases.
- zerocrates 5y agoObviously you're saying it doesn't cover everything, but a big thing it's not going to catch beyond leetspeak-type situations is the kinds of thing you (used to) see in internationalized domain spoofing: legitimate non-Latin-script letters that just look the same or nearly the same. NFKC/NFKD will handle "this is another form of the Latin letter A" type stuff but not "Cyrillic A looks like Latin A."
- JanisL 5y agoThank you for making this!
- mr_toad 5y agoWhat I don’t understand is who sees spam that looks like an old school ransom note and thinks ‘seems legit’, and then actually buys something?
- jonplackett 5y agoThere should be a 'bad symbols' list you can block like you do expletives. There's surely zero need to support that kind of thing in comments.
- jcims 5y agoOne term used to describe this is homoglyph, as in homoglyph attacks for phishing. Back in 2015 I did some work just using simple bitmap rendering plus OCR to find text that looked like a small selection of known words. It was actually reasonably effective.