6 ms·

I'm just agog that people are still doing dumb pattern matching for profanity filters. I just assumed that YEARS AGO people realized how dumb it is, but apparen

I'm just agog that people are still doing dumb pattern matching for profanity filters. I just assumed that YEARS AGO people realized how dumb it is, but apparently: No.

Blikkentrekker 6y ago
A great many places do this and automatically refuse content based on arbitrary “bad words” regardless the context. I remember being denied to post a forum post containing the phrase “tardive dyskinesia”, as it appears that it rejected anything with the string “tard” in it. I'm not sure as to whom they think to be helping with that, but it's entirely possible that their advertisement revenue will actually suffer, if the string “tard” be found on their pages.
Majromax 6y ago
This is Google. It's probably very smart pattern matching for profanity. The neural network may have taken millions of core-hours to learn to be as dumb (here) as a blind keyword search.
- ttt0 6y ago
  We had to give up our privacy to create a highly sophisticated technology that doesn't even work half of the time. I love the future, it was totally worth it.
- smichel17 6y ago
  Well, obviously. If it were a dumb profanity filter then it would be possible to fix it!
deleted 6y ago
[deleted]
boring_twenties 6y ago
Not super relevant or anything but I just can't help but share my favorite profanity filter story, so here you go. I worked at a place that had a profanity filter in two parts. The first part was in C, several pages of if (!strcmp(x, a)) return 0; After all that, it then invokes popen() to ssh to another machine and run a shell script there, which contains several more pages of string comparisons, this time in shell.
- siltpotato 6y ago
  Hold on, what? Okay, return false if they aren't equal, then open another process to repeat this method once again in the shell... I can't guess the reason. Would you know if there is any reason this might have been done?
  - saagarjha 6y ago
    strcmp returns 0 if the strings are equal.
  - boring_twenties 6y ago
    I wouldn't know the real reason for sure, but this seems plausible: 1) They got tired of having to modify C code and wait for the deploy cycle to modify the filter 2) Using, for example, the database would be more work than calling a shell script. On top of that, it might actually be beyond the abilities of the programmer involved. 3) The C code executes on an arbitrary machine. Hence the ssh to a specific machine, so that the shell script would only have to be maintained in one place
- tyingq 6y ago
  Doesn't popen() pass strings to a shell? Sounds dangerous, as you would have to escape semicolons, quotes, etc.
  - ljm 6y ago
    I might be wrong but I think it's about censoring the 'hell' in 'shell'. Because some parts of the world consider words like 'hell' and 'damn' to be profane.
aidos 6y ago
I once had a bug that I traced back to a rule (can’t remember in which part of the stack - though I think it was client controlled IIS) that was striping the “select” from the word “selected” in query string params in an attempt to thwart sql injection. From memory it was naive enough that “sselectelect” was converted nicely in the process.
- simion314 6y ago
  I had similar issues, the software is Mod Security that some hosting companies use and some rules will empty out your POST request field if it contained text like ".... select ...from..." where the 2 keywords were paragraphs apart.
- jimsmart 6y ago
  Similar: Yahoo used to (2002) replace any instance of the character sequence 'eval' (and other 'bad' strings) in their emails, in an attempt to prevent Javascript exploits. Needless to say it created a small amount of havoc! http://news.bbc.co.uk/1/hi/sci/tech/2138014.stm http://news.bbc.co.uk/1/hi/sci/tech/2138014.stm https://en.wiktionary.org/wiki/medireview https://en.wiktionary.org/wiki/medireview
  - distances 6y ago
    I hadn't heard of this and I'm now flabbergasted. Is it even legal for a service provider to secretly change email contents? It's absolutely outlandish to imagine how someone first thought this could be a good idea, and then found someone capable of executing the plan and apparently agree.
drzoltar 6y ago
FWIW, general profanity detection is a highly nontrivial problem. It’s true that such subword profanity filters aren’t that great, but slightly more sophisticated ones (eg whole word matching or n-grams) tend to have relatively good precision. You could train a fancy neural network, but the overall return on precision and recall tends to be not that great (compared to the exponential change in speed and cost). The problem almost always crops up in out-of-distribution sentences (such as “bone” at a paleontology conference).
- wiml 6y ago
  Even humans with full general intelligence and domain knowledge will fail at profanity detection. I think the problem here is not so much that there are false triggers, but that there is no way to deal with the false triggers — no way to appeal to reason or utility.
  - Blikkentrekker 6y ago
    It's a problem with a subjective answer. One man's profanity is not another man's profanity. Of course, the personality trait of desiring censoring “bad words” seems to highly correlate with a belief in objective morality. — the others are wrong about what they find profane!
gaius_baltar 6y ago
They just rebranded it as "AI-powered profanity filter" :)

I'm just agog that people are still doing dumb pattern matching for profanity filters. I just assumed that YEARS AGO people realized how dumb it is, but apparently: No.

Blikkentrekker 6y ago

A great many places do this and automatically refuse content based on arbitrary “bad words” regardless the context. I remember being denied to post a forum post containing the phrase “tardive dyskinesia”, as it appears that it rejected anything with the string “tard” in it. I'm not sure as to whom they think to be helping with that, but it's entirely possible that their advertisement revenue will actually suffer, if the string “tard” be found on their pages.

Majromax 6y ago

This is Google. It's probably very smart pattern matching for profanity. The neural network may have taken millions of core-hours to learn to be as dumb (here) as a blind keyword search.

ttt0 6y ago
We had to give up our privacy to create a highly sophisticated technology that doesn't even work half of the time. I love the future, it was totally worth it.
smichel17 6y ago
Well, obviously. If it were a dumb profanity filter then it would be possible to fix it!

deleted 6y ago

[deleted]

boring_twenties 6y ago

Not super relevant or anything but I just can't help but share my favorite profanity filter story, so here you go. I worked at a place that had a profanity filter in two parts. The first part was in C, several pages of if (!strcmp(x, a)) return 0; After all that, it then invokes popen() to ssh to another machine and run a shell script there, which contains several more pages of string comparisons, this time in shell.

siltpotato 6y ago
Hold on, what? Okay, return false if they aren't equal, then open another process to repeat this method once again in the shell... I can't guess the reason. Would you know if there is any reason this might have been done?
- saagarjha 6y ago
  strcmp returns 0 if the strings are equal.
- boring_twenties 6y ago
  I wouldn't know the real reason for sure, but this seems plausible: 1) They got tired of having to modify C code and wait for the deploy cycle to modify the filter 2) Using, for example, the database would be more work than calling a shell script. On top of that, it might actually be beyond the abilities of the programmer involved. 3) The C code executes on an arbitrary machine. Hence the ssh to a specific machine, so that the shell script would only have to be maintained in one place
tyingq 6y ago
Doesn't popen() pass strings to a shell? Sounds dangerous, as you would have to escape semicolons, quotes, etc.
- ljm 6y ago
  I might be wrong but I think it's about censoring the 'hell' in 'shell'. Because some parts of the world consider words like 'hell' and 'damn' to be profane.

aidos 6y ago

I once had a bug that I traced back to a rule (can’t remember in which part of the stack - though I think it was client controlled IIS) that was striping the “select” from the word “selected” in query string params in an attempt to thwart sql injection. From memory it was naive enough that “sselectelect” was converted nicely in the process.

simion314 6y ago
I had similar issues, the software is Mod Security that some hosting companies use and some rules will empty out your POST request field if it contained text like ".... select ...from..." where the 2 keywords were paragraphs apart.
jimsmart 6y ago
Similar: Yahoo used to (2002) replace any instance of the character sequence 'eval' (and other 'bad' strings) in their emails, in an attempt to prevent Javascript exploits. Needless to say it created a small amount of havoc! http://news.bbc.co.uk/1/hi/sci/tech/2138014.stm http://news.bbc.co.uk/1/hi/sci/tech/2138014.stm https://en.wiktionary.org/wiki/medireview https://en.wiktionary.org/wiki/medireview
- distances 6y ago
  I hadn't heard of this and I'm now flabbergasted. Is it even legal for a service provider to secretly change email contents? It's absolutely outlandish to imagine how someone first thought this could be a good idea, and then found someone capable of executing the plan and apparently agree.

drzoltar 6y ago

FWIW, general profanity detection is a highly nontrivial problem. It’s true that such subword profanity filters aren’t that great, but slightly more sophisticated ones (eg whole word matching or n-grams) tend to have relatively good precision. You could train a fancy neural network, but the overall return on precision and recall tends to be not that great (compared to the exponential change in speed and cost). The problem almost always crops up in out-of-distribution sentences (such as “bone” at a paleontology conference).

wiml 6y ago
Even humans with full general intelligence and domain knowledge will fail at profanity detection. I think the problem here is not so much that there are false triggers, but that there is no way to deal with the false triggers — no way to appeal to reason or utility.
- Blikkentrekker 6y ago
  It's a problem with a subjective answer. One man's profanity is not another man's profanity. Of course, the personality trait of desiring censoring “bad words” seems to highly correlate with a belief in objective morality. — the others are wrong about what they find profane!

gaius_baltar 6y ago

They just rebranded it as "AI-powered profanity filter" :)