4 ms·

Show HN: A free, AI-powered service for automating hate speech moderation

12 points by cipher129 4y ago

Hey everyone! I built ModerateHatespeech, an initiative that working on better understanding + building solutions to combat hate speech in online platform. We have our flagship API, which is completely free and ML-powered, and gives a lot more actionable/better results than pretty much any other similar platform out there. We've been able to leverage our partnerships with many communities to get a lot of good data + feedback, and bring our system to a lot of users (we process ~200k comments a day right now). We've also done a lot of work to better understand biases/potential abuse-cases of our API, which you can read about on our site (trying to avoid too many links getting caught in spam filters). I would definitely love to hear any thoughts/feedback! Here's a link to information about our project/API: https://moderatehatespeech.com/

jawerty 4y ago
How do you define hate speech?
- cipher129 4y ago
  We define hate here: https://moderatehatespeech.com/framework/ https://moderatehatespeech.com/framework/ As: “any unreasonably rude or hateful content, constituting as, "threats, extreme obscenity, insults, and identity-based hate"”
masterof0 4y ago
"Republicans are bigots" { "class": normal "confidence": 0.955 } "Democrats are bigots" { "class": flag "confidence": 0.575 }
- cipher129 4y ago
  Interesting! That’s something we will definitely adjust for in upcoming models, though practically speaking, the confidence is low enough that in mod systems they would be treated the same
  - masterof0 4y ago
    I agree, I wonder what is your data composition? Reddit threads, blogs, HN discussions? You can see why someone could make the case for bias. But I think is pretty good project, and pretty useful for communities that want only members with the same ideology, and need an automated filter to help with undesired comments, etc.
    - masterof0 4y ago
      Just saw the framework section.
af3d 4y ago
Interesting. It seems to be able to detect the subtle nuances of meaning fairly well. Maybe not perfect, but I would give it at least 8 stars on a scale from 1 to 10.
sn0w_crash 4y ago
So it’s just a woke language model?
- masterof0 4y ago
  See my comment, indeed, I still think is useful for lets say a woke Mastodon instance that wants automated non-woke speech removal. Which is fine, the other side can come up with something similar as well.
- warning26 4y ago
  If you look at their examples, it's really more of a trolling classifier.
Sujeto 4y ago
Works pretty well in the few tests I made
itake 4y ago
What is the max qps and latency this supports? What city is it hosted in? Some moderation platforms I've worked with are too slow for messaging, especially if users are in different countries. Do you have plans for image moderation?
- cipher129 4y ago
  > What is the max qps and latency this supports? We generally aim for <400ms in the continental US. The backend is hosted in us-central-1 on GCP. Obviously, we're always trying to expand, though we do rely on grant funding as a non profit so that comes into play too. > Some moderation platforms I've worked with are too slow for messaging, especially if users are in different countries. Typically it works pretty well on the message platforms we're on -- we've got a Chrome extension for Twitter + discord integrations we're testing right now. > Do you have plans for image moderation? We do! Though that's further down the line.
smileybarry 4y ago
It doesn't seem to deal with misgendering hate or transphobia very well (I saw lots of models failing on this so I checked that right away), and I mean obvious ones regardless of your stance, e.g.: > "She is a he" => { "class": normal, "confidence": 1 } > "He will never be a woman" => { "class": normal, "confidence": 0.999 } But it does seem to identify 2nd-person targeted transphobia: > "You will never be a woman" => { "class": flag, "confidence": 0.996 }
- yuvalsteuer 4y ago
  Also misgendering someone isn't considered hatespeech in the US.. but more so in canada. so it depends on where your located geographically, or on your stance.
  - smileybarry 4y ago
    The website linked in this very post doesn’t mean “hate speech according to local laws”. It just means toxicity, and yes, transphobia like “you’ll never be a woman” is clearly it. I obviously don’t mean accidental misgendering or the like.
- sn0w_crash 4y ago
  Misgendering is not against the law in the US and is not considered hate speech by many people. Don’t flame me I’m just pointing it out.
- cipher129 4y ago
  Yeah -- the latter is obviously much easier to identify because it's much more explicit/clear cut. The first are of course, transphobic given the clear intent, but it there is definitely some ambiguity w/o context. Ie, if something just posted, "He will never be a woman" there's definitely many contexts where something like that is not malicious. Ie, if someone else is correcting an accidental mis-gender, they might say, "No, she is actually a he" which isn't malicious. Since the state of our API (and literally every other similar system out there, lol) can't capture the context of the conversation -- what "He" is referring to, the overall topic of the conversations, etc -- we basically have to make the best calls given the available context...which unfortunately does create false negatives in certain cases (and false positives in others). But, context is something we're working on, because its definitely a big issue, and as you can see, plays a very prominent role in informing better moderation decisions!
Komodai 4y ago
DontchaKnowit 4y ago
God can't wait till this is implemented everywhere and the internet turns into fucking Disneyland.

12 points by cipher129 4y ago

jawerty 4y ago

How do you define hate speech?

cipher129 4y ago
We define hate here: https://moderatehatespeech.com/framework/ https://moderatehatespeech.com/framework/ As: “any unreasonably rude or hateful content, constituting as, "threats, extreme obscenity, insults, and identity-based hate"”

masterof0 4y ago

"Republicans are bigots" { "class": normal "confidence": 0.955 } "Democrats are bigots" { "class": flag "confidence": 0.575 }

cipher129 4y ago
Interesting! That’s something we will definitely adjust for in upcoming models, though practically speaking, the confidence is low enough that in mod systems they would be treated the same
- masterof0 4y ago
  I agree, I wonder what is your data composition? Reddit threads, blogs, HN discussions? You can see why someone could make the case for bias. But I think is pretty good project, and pretty useful for communities that want only members with the same ideology, and need an automated filter to help with undesired comments, etc.
  - masterof0 4y ago
    Just saw the framework section.

af3d 4y ago

Interesting. It seems to be able to detect the subtle nuances of meaning fairly well. Maybe not perfect, but I would give it at least 8 stars on a scale from 1 to 10.

sn0w_crash 4y ago

So it’s just a woke language model?

masterof0 4y ago
See my comment, indeed, I still think is useful for lets say a woke Mastodon instance that wants automated non-woke speech removal. Which is fine, the other side can come up with something similar as well.
warning26 4y ago
If you look at their examples, it's really more of a trolling classifier.

Sujeto 4y ago

Works pretty well in the few tests I made

itake 4y ago

What is the max qps and latency this supports? What city is it hosted in? Some moderation platforms I've worked with are too slow for messaging, especially if users are in different countries. Do you have plans for image moderation?

cipher129 4y ago
> What is the max qps and latency this supports? We generally aim for <400ms in the continental US. The backend is hosted in us-central-1 on GCP. Obviously, we're always trying to expand, though we do rely on grant funding as a non profit so that comes into play too. > Some moderation platforms I've worked with are too slow for messaging, especially if users are in different countries. Typically it works pretty well on the message platforms we're on -- we've got a Chrome extension for Twitter + discord integrations we're testing right now. > Do you have plans for image moderation? We do! Though that's further down the line.

smileybarry 4y ago

It doesn't seem to deal with misgendering hate or transphobia very well (I saw lots of models failing on this so I checked that right away), and I mean obvious ones regardless of your stance, e.g.: > "She is a he" => { "class": normal, "confidence": 1 } > "He will never be a woman" => { "class": normal, "confidence": 0.999 } But it does seem to identify 2nd-person targeted transphobia: > "You will never be a woman" => { "class": flag, "confidence": 0.996 }

yuvalsteuer 4y ago
Also misgendering someone isn't considered hatespeech in the US.. but more so in canada. so it depends on where your located geographically, or on your stance.
- smileybarry 4y ago
  The website linked in this very post doesn’t mean “hate speech according to local laws”. It just means toxicity, and yes, transphobia like “you’ll never be a woman” is clearly it. I obviously don’t mean accidental misgendering or the like.
sn0w_crash 4y ago
Misgendering is not against the law in the US and is not considered hate speech by many people. Don’t flame me I’m just pointing it out.
cipher129 4y ago
Yeah -- the latter is obviously much easier to identify because it's much more explicit/clear cut. The first are of course, transphobic given the clear intent, but it there is definitely some ambiguity w/o context. Ie, if something just posted, "He will never be a woman" there's definitely many contexts where something like that is not malicious. Ie, if someone else is correcting an accidental mis-gender, they might say, "No, she is actually a he" which isn't malicious. Since the state of our API (and literally every other similar system out there, lol) can't capture the context of the conversation -- what "He" is referring to, the overall topic of the conversations, etc -- we basically have to make the best calls given the available context...which unfortunately does create false negatives in certain cases (and false positives in others). But, context is something we're working on, because its definitely a big issue, and as you can see, plays a very prominent role in informing better moderation decisions!

Komodai 4y ago

DontchaKnowit 4y ago

God can't wait till this is implemented everywhere and the internet turns into fucking Disneyland.