4 ms·

For those used to shock sites from 10-20 years ago, this is very different, I'm not going to link to anything but the disturbing content is on a few whole new l

For those used to shock sites from 10-20 years ago, this is very different, I'm not going to link to anything but the disturbing content is on a few whole new levels. I can't even imagine doing this job. This should be 100% automated with only 0.01% error rate for the first-pass filter (e.g. before it reaches humans). They are willing to spare their users the horrors but not their own employees/contractors...

intended 8y ago
Facebook tried the automated system by using employees to train the machine. It failed. They returned to non Facebook employed humans doing the moderation
- rock_hard 8y ago
  I read somewhere that it’s actually a hybrid approach! For some things liken nudity ML based solution work fine. But other things such as hate speech are more nuanced and require human oversight. That said ML is used everywhere possible to flag content to moderators. For some categories the work of human moderators is used to train the ML models. Facebook and other companies like YouTube are well insentivized to do the right thing here and automate as much as possible for all the reasons outlined in OP
- fencepost 8y ago
  One thing I hope they're doing is advanced detection of duplicate inappropriate content - e.g. by splitting a confirmed properly blocked video into frames and identifying matching videos based on frames matching (or even key parts of frames). You'd still have to investigate/review partial matches, but something like that could cut down a lot on duplicate effort and could auto-identify a lot of things that would need manual review.
agentdrtran 8y ago
Machines cannot do this job yet.
ddalex 8y ago
I developed the first version of the screening system for a corporation about 10 years ago, when UGC was still in its infancy. We had the help of a Microsoft tool that would match the visual fingerprint of a photo against a known-images database that prevented human operators from re-seeing an already tagged image in a different cloud, hoping to reduce the psychological scarring of the human workers that screened the content. We also evaluated automatic classification of images, but it's very hard to detect soft core porn from shock images even with today's algos, never mind 10 years ago. One advice from a legal counsel that I received at that time still stands in my memory; she said: "Never look at the images, nevermind how curious you are. It's not just illegal and can land you in prison, but it's permanent. You cannot un-see something you've seen."

intended 8y ago

Facebook tried the automated system by using employees to train the machine. It failed. They returned to non Facebook employed humans doing the moderation

rock_hard 8y ago
I read somewhere that it’s actually a hybrid approach! For some things liken nudity ML based solution work fine. But other things such as hate speech are more nuanced and require human oversight. That said ML is used everywhere possible to flag content to moderators. For some categories the work of human moderators is used to train the ML models. Facebook and other companies like YouTube are well insentivized to do the right thing here and automate as much as possible for all the reasons outlined in OP
fencepost 8y ago
One thing I hope they're doing is advanced detection of duplicate inappropriate content - e.g. by splitting a confirmed properly blocked video into frames and identifying matching videos based on frames matching (or even key parts of frames). You'd still have to investigate/review partial matches, but something like that could cut down a lot on duplicate effort and could auto-identify a lot of things that would need manual review.

agentdrtran 8y ago

Machines cannot do this job yet.

ddalex 8y ago

I developed the first version of the screening system for a corporation about 10 years ago, when UGC was still in its infancy. We had the help of a Microsoft tool that would match the visual fingerprint of a photo against a known-images database that prevented human operators from re-seeing an already tagged image in a different cloud, hoping to reduce the psychological scarring of the human workers that screened the content. We also evaluated automatic classification of images, but it's very hard to detect soft core porn from shock images even with today's algos, never mind 10 years ago. One advice from a legal counsel that I received at that time still stands in my memory; she said: "Never look at the images, nevermind how curious you are. It's not just illegal and can land you in prison, but it's permanent. You cannot un-see something you've seen."