3 ms·
I think at some point, a sort of visual-normalization that converts similar looking unicode to a single unique string sequence (ex: convert certain letters from
by haneefmubarak 9y ago
I think at some point, a sort of visual-normalization that converts similar looking unicode to a single unique string sequence (ex: convert certain letters from Cyrillic and other language sets that are also present in Latin to just Latin) is just going to be necessary as a security precaution.
Given the whole "fake news" thing over the past couple of years, I expect that the first step will be taken by one of Google/Twitter/Facebook/etc, but I hope that they (or someone else) releases a library (or worst case, an online API) that allows this sort of normalization for security verification. I get that having it open would make it easier for people to find loopholes by brute-force testing, but these sorts of loopholes could also be patched rather quickly as they came up, providing benefit to everyone (especially from a security perspective).
EDIT: Perhaps this could start out as a series of matches generated using ML classification? I don't know much about ML - does anyone who does think this is a realistic starting point?
- sedev 9y agoUnicode normalization/equivalence is half of what you want and UCAPI is probably the other half. https://en.wikipedia.org/wiki/Unicode_equivalence https://en.wikipedia.org/wiki/Unicode_equivalence https://www.casaba.com/products/UCAPI/ https://www.casaba.com/products/UCAPI/
- haneefmubarak 9y agoThose are interesting, especially the second one. I've read a little here and there about Unicode normalization before, but UCAPI does look like what I really wanted. However, seeing as UCAPI isn't free or even "listed price plans", I get the strong feeling that this will not see much pickup (at least until someone makes a free one).
- BoorishBears 9y agoThere are plenty of free libraries that allow you to detect and compare strings with confusables