3 ms·

This is a pretty bad idea (and English-centric). Stripping diacritics can fundamentally alter a word's meaning. A small example (Portuguese): país ("countr

• diacritics 12y ago
  Or in spanish: Trabajé ("I worked.") Trabaje ("You work." (formal))
• itsybitsycoder 12y ago
  The goal of the exercise is to use the stripped text to check for spam. That doesn't mean that the stripped text is what ends up in the user's inbox. The idea is to see if the text contains things like ＶᎥÄｇԻａ, not whether a given word means "country" or "father". This would only be a problem if some set of characters that looked like "viagra" was actually a valid non-spammy word in some language. There was also an article here a few weeks back about Russian government officials securing fat contracts for their friends in private industry by intentionally replacing one or two letters in the common search terms of their bid requests with Latin lookalike characters. This was impossible to detect while reading the bid request, but also made it impossible for other contractors to find the bid request through the website. As a result only their buddy would submit a bid, and at much higher than market rate. Something similar to this, but converting down to Cyrillic characters instead of ASCII, could be used to check for hinky bid requests on upload.