3 ms·

It's really just moderation in general. It's a long term, never ending issue. I think the big platforms employ thousands of moderators.

coldacid 5y ago
There's a reason there's been so much research into AI moderation, although so far it always seems to either be useless or throwing babies out with the bathwater. Not to mention that without human systems to review what the AI decides (or even with) it tends to appear to users as automatic censorship (and they're usually not wrong about that).
- yeldarb 5y ago
  We had pretty good success on https://paint.wtf https://paint.wtf (AI-judged Pictionary game) using CLIP for content moderation. Feel free to probe it a bit by drawing a dick or something on one of the prompts; it seems to do a pretty good job (if it doesn’t work please hit the “Report” button and let us know). Wrote about it here: https://blog.roboflow.com/zero-shot-content-moderation-openai-new-clip-model/ https://blog.roboflow.com/zero-shot-content-moderation-opena...
- PaulHoule 5y ago
  The difficulty of it is the imbalance. Once I had 10,000 NSFW images in a collection of 1,000,000. You might say that 99% of the images are good and for most classification problems 99% accuracy would be wonderful. But it's not good enough. You might find some simple tricks that eliminate 90% of those NSFW images and now you have 99.9% accuracy. That makes you look like a genius in the machine learning world. But it's not good enough. Even 99.99% accuracy (100 bad images) is not good enough Even if somebody found one NSFW image you could get kicked out of Adsense. One child porn image uploaded to your site won't put you in jail but it is still completely unacceptable.
tjpnz 5y ago
I think a large part of the issue could be solved through taking a more "common sense" approach to product design. Imagine you offer a platform like Facebook and I registered a few hours ago. Is it really smart to allow me to start uploading dozens of images or livestream to potentially thousands of existing users? Surely a better approach would be for my account to have restrictions on what I can do and the number of times/period for which I can do it. Once I become trusted in the eyes of the platform those restrictions can start to be loosened.
dredmorbius 5y ago
A of 10 May 2021 at Facebook, over 15,000, suing for mental health trauma: https://www.nbcnews.com/business/business-news/facebook-content-moderators-say-they-receive-little-support-despite-company-n1266891 https://www.nbcnews.com/business/business-news/facebook-cont...

coldacid 5y ago

There's a reason there's been so much research into AI moderation, although so far it always seems to either be useless or throwing babies out with the bathwater. Not to mention that without human systems to review what the AI decides (or even with) it tends to appear to users as automatic censorship (and they're usually not wrong about that).

yeldarb 5y ago
We had pretty good success on https://paint.wtf https://paint.wtf (AI-judged Pictionary game) using CLIP for content moderation. Feel free to probe it a bit by drawing a dick or something on one of the prompts; it seems to do a pretty good job (if it doesn’t work please hit the “Report” button and let us know). Wrote about it here: https://blog.roboflow.com/zero-shot-content-moderation-openai-new-clip-model/ https://blog.roboflow.com/zero-shot-content-moderation-opena...
PaulHoule 5y ago
The difficulty of it is the imbalance. Once I had 10,000 NSFW images in a collection of 1,000,000. You might say that 99% of the images are good and for most classification problems 99% accuracy would be wonderful. But it's not good enough. You might find some simple tricks that eliminate 90% of those NSFW images and now you have 99.9% accuracy. That makes you look like a genius in the machine learning world. But it's not good enough. Even 99.99% accuracy (100 bad images) is not good enough Even if somebody found one NSFW image you could get kicked out of Adsense. One child porn image uploaded to your site won't put you in jail but it is still completely unacceptable.

tjpnz 5y ago

I think a large part of the issue could be solved through taking a more "common sense" approach to product design. Imagine you offer a platform like Facebook and I registered a few hours ago. Is it really smart to allow me to start uploading dozens of images or livestream to potentially thousands of existing users? Surely a better approach would be for my account to have restrictions on what I can do and the number of times/period for which I can do it. Once I become trusted in the eyes of the platform those restrictions can start to be loosened.

dredmorbius 5y ago

A of 10 May 2021 at Facebook, over 15,000, suing for mental health trauma: https://www.nbcnews.com/business/business-news/facebook-content-moderators-say-they-receive-little-support-despite-company-n1266891 https://www.nbcnews.com/business/business-news/facebook-cont...