3 ms·
I did the communications server for a small online gaming site once, and it had a profanity filter on the chat channels. The approach we took mostly avoided the
by tzs 9y ago
I did the communications server for a small online gaming site once, and it had a profanity filter on the chat channels. The approach we took mostly avoided the Scunthorpe problem, and also caught when people tried to sneak profanity past it by splitting it across words (e.g., knowing that "motherfucker" would be filtered, someone might try "motherfu cker", or maybe even "m0th3rfu ck3r").
Here is what we did, as far as I recall. This worked a heck of a lot better than I expected it to.
1. Make a copy of the text and replace all the common non-letters that look somewhat like letters with those letters. E.g., 0 => O, 1 => L, 3 => E, 4 => A, 7 => T, @ => A.
2. Do a spell check on each word. Set a flag on each character that comes from a word that passed the spell check.
3. Scan the sequence of characters, ignoring all punctuation and white space, looking for sequences that match known profanity.
4. For each sequence that matches, mark it as profanity if any of its characters were not flagged in step #2.
As described above this probably would not scale well to high traffice because of the spell check. That could be improved by changing the order. Look for potential profanity first. Only if some is found would you then do the spell check, and you would only need to spell check the words that span the potential profanity.