19 ms·

The trauma of Facebook’s content moderators

218 points by donohoe 6y ago

anonu 6y ago
Can we build better AI off of the data these moderators have generated to make their lives easier? Can you crowdsource the moderation task to double check what the ai is flagging?
- Sebb767 6y ago
  > Can you crowdsource the moderation task[...]? I highly doubt that. These reported posts might contain private data or highly shocking content; putting that on a public platform, especially associated with the name of Facebook, is very unlikely to happen.
  - munificent 6y ago
    > putting that on a public platform, especially associated with the name of Facebook Is that not what the person who originally posted the content being moderated is doing?
    - Sebb767 6y ago
      It still would open up a gore/scare platform (think LiveLeak), since only the bad content is reported. Sure, most might be tame, but it wouldn't take long for gore seekers to come.
    - chronid 6y ago
      > Is that not what the person who originally posted the content being moderated is doing? The person that posted the content does not expect a random person from the internet to see a video of his children because it was accidentally flagged by an AI as child porn. :)
- ceejayoz 6y ago
  > Can you crowdsource the moderation task to double check what the ai is flagging? "Facebook sent me child porn" is probably not a headline Facebook wants out there.
- codekansas 6y ago
  Easier said than done https://time.com/5739688/facebook-hate-speech-languages/ https://time.com/5739688/facebook-hate-speech-languages/
- filoleg 6y ago
  >Can you crowdsource the moderation task to double check what the ai is flagging? This would help with false positives, which is a noble goal. But it would not help with false negatives, which is imo a very disastrous consequence in this specific scenario. Something that isn't CP got marked as CP? Sure, a moderator can manually doublecheck that flag and override if needed. Something that is CP didn't get marked as CP? You got a problem on your hands, because there is no way to manually doublecheck that, since that would mean manually checking every single post marked as "no issues here", which gets us back to square one of manual moderation of all posts. Not even mentioning the exposure of FB users to CP that it would bring, which would mentally ruin quite a lot of people and bring very negative press to FB on the level of severity that we haven't seen before.
  - landryraccoon 6y ago
    Speaking as someone who has to deal with supervised learning models, human beings falsely tag positive examples as negative all the time. Merely having "average" human beings tag examples doesn't guarantee perfect recall. Or to restate, sites that rely purely on manual moderation of content that breaks rules still has content that breaks rules leak through all the time.
- uniqueid 6y ago
  > Can we build better AI Nope. That's what the people in charge of our social media companies keep trying, over and over, and it doesn't work at all. AI-moderation is like the Ring is to Golem for them. They can't accept that "cheap and fair" moderation isn't currently possible. > Can you crowdsource the moderation task This is the clear answer, imo, and it isn't obvious to me why these C-level execs avoid it. If I had to make a guess, maybe they think engaging with millions of their users about moderation would open a can of worms.
  - shadowgovt 6y ago
    When the goal is to prevent users from seeing this content, crowd-sourcing moderation defeats the purpose. Users don't want the back-stop to be "a critical mass of community members object;" for content like this, they want to see zero of it, ever, and will choose to use another site that satisfies that hard constraint if FB cannot.
    - uniqueid 6y ago
      > they want to see zero of it, ever Many want that, but very few (probably nobody at all, by this point) expect it. > When the goal is to prevent users from > seeing this content, crowd-sourcing moderation > defeats the purpose. It's not actually all-or-nothing, and if it were, it would defeat it no less frequently than paid moderation already does. People tolerated toxic internet content before social media not because they never encountered any, but because they encountered much less. No system can ever stop 100% of users from seeing 100% of the material they find objectionable. Every user has their own idea of what material is beyond the pale, and even if we invent AI that is competent at blocking content, and capable of tailoring its filters per-user, it will still fail some of the time, because users are people, and people's tastes change over time. Aside from that, the way Youtube, Reddit, Twitter et al, currently operate, a user who reports objectionable content typically waits days, weeks, or years (or forever) for the company to take action. If you give a user a little real agency, it goes a long way to mitigate their displeasure over occasional exposure to unwanted content.
      - 6y ago
- msapaydin 6y ago
  I also made a comment before I saw yours about creating a machine learning based system to automate this process, I think it must be fairly easy given the amount of data already available.
MattGaiser 6y ago
They should recruit people on Reddit for this. A large number of them seem perfectly fine with this type of content. There are dedicated communities to all these things. I do not mean this as a hit against Reddit users. It is just that there are small pockets of people who are psychologically capable of tolerating this or even enjoy it. It seems more ethical to hire people from /r/watchpeopledie.
- dmalvarado 6y ago
  Perhaps. I would argue that there is some difference between seeing someone fall off a cliff and watching someone being tortured to death.
  - throwaway0a5e 6y ago
    Then they should run sidebar ads on the gore sites. The point is that there appears to be a subset of the population that is not traumatized by this stuff. We may as well give them an opportunity to put that difference to use. Edit: and saying things like "well clearly these people are already traumatized if they're seeking this out, it's not good for them, we shouldn't give these jobs to them" seems laughably paternalistic when the alternative is to traumatize more people by making them look at it instead.
    - ceejayoz 6y ago
      How do we know they're not traumatized by it? That they seek it out doesn't mean they're not messed up.
      - filoleg 6y ago
        ^this. I am not arguing one way or another, because I don't see strong evidence making me lean predominantly one way. But imo, it is at least just as possible that they are seeking out that kind of content voluntarily because they are already traumatized in the first place, so this is a coping mechanism. Maybe it helps dealing with that trauma. Maybe it makes the trauma worse. Maybe there was no trauma to begin with, and the person was seeking out that content simply due to morbid curiosity, but then it evolved into something else. Maybe there was no trauma to begin with and no trauma appeared after the exposure. Who knows. But just because there are communities on reddit with people seeking that kind of material and not going on crazy violent sprees doesn't mean that it doesn't negatively affect them. It could be the case, but we should probably look into that first before jumping to "obvious solution" claims. tl;dr: we gotta do more research and weigh pros and cons before simply declaring "we have an obvious solution, let's just hire all those gore content enthusiasts to moderate violent content, because they already do it on their own and don't seem to be going crazy in obvious ways due to it".
boringcuriosity 6y ago
Content moderators have always been in tough situations and they always have problems related to psychology and minds
shadowgovt 6y ago
I can't help but wonder what the numbers look like on this content. One of the things Facebook should have enough data to know is how prevalent this sort of thing is. Is deeply offensive content generated by a handful of users frequently, or many users less often? What's the volume look like?
- filoleg 6y ago
  Agreed. I would definitely love to see some sort of a basic data analysis blog post on this topic from FB, similar to how match.com used to publish data analysis posts on their blog back in the day.
Magodo 6y ago
I don't understand what's done with the contractors in the US after the successful lawsuit, surely all those jobs were just moved offshore? There's no mention of this at all in the article....
econcon 6y ago
How can I become a partime fb content moderator? Is it possible to become one online and work remotely? Or must work from the fb camp? I want to test if it can cause me any trauma
- khazhoux 6y ago
  I suppose the FB job board is the place to start. Did you check there?
- talmr 6y ago
  I'm not sure if you are being sarcastic so forgive me if this is off-base, but there are REAL people who are dealing with trauma from moderating content so you don't need to "test it out" and validate.
praveen9920 6y ago
The issue is as real as western countries dumping garbage in eastern countries just because they can pay them off. The local governments won't acknowledge this as problem because of the money flowing in and a lot of people's livelihood is dependent on this. But we should call it what it is, exploitation.
- blargmaster33 6y ago
  No, western countries are NOT dumping garbage. They are buying the service "waste disposal", they are NOT responsible for the supplier of that service just dumping the garage.
- quadrifoliate 6y ago
  I think in regards to India, the issue is a little more complex, and intertwined with social mores. This one sentence from the article is really important: > According to Dr. K. Jyothirmayi, a Hyderabad-based psychiatrist, stigmas, such as the perceived impact of a mental health diagnosis on one’s marital prospects, often prevent young Indians from seeking treatment. From growing up in India, my view is that meditation and the general state of the mind has been taken seriously for, well, centuries. But move the focus towards anything with 'psych' or 'mental' in the title and people in India will shy away from it even if it is affordable or free. "Seeing a mental health counselor" is a recipe for considerable social stigma in India [1], not an empowering practice like it's perceived as in the United States. The solution to this is hard, and in my opinion not something that Facebook or their Indian subcontractors can accomplish. ----------- [1] I'm talking "All your relatives will refer to you as the crazy person" or "Your girlfriend's parents will call off the engagement" levels of stigma, not just "I don't feel comfortable talking about it in a bar" levels. Like a lot of other things, families with higher levels of income and education are sometimes an exception.
- Nasrudith 6y ago
  The awkward ethical question is would they be better off without "exploitation"? Part of the reason companies are even willing to go through the overhead is because the costs are lower. Real life has always been filled with messed up situations where the punchline is "... and the best/worst part is it qualified as an improvement!".
lowmemcpu 6y ago
I recall a statistic from about 10 years ago that computer forensic investigators in law enforcement burn out after two years due to the trauma of the images they are exposed to.
- throwaway0a5e 6y ago
  There's likely some confounding factors. Pressing "go" on the overpriced software tools and then entering into evidence what you find is the lowest level of work in that field so the churn is going to naturally be very high as people move up or out. The pay also isn't that great.
  - mschuster91 6y ago
    No, that is not the issue. Rather the issue is that even the hardest stuff on Facebook isn't remotely comparable to stuff of actual criminals, and the effort is wildly different: - Facebook: it's violating rules? Delete, next. - Forensic IT on a multi TB disk full with child porn: document every photo, what it shows, extract identifiable faces to cross reference with other content (to check for recurring places and victims), and the process is even more gory for video content. You have to watch every second or the defense can attempt "you didn't watch the video in full where the perp gives the victim an ice cream at the end" or whatever else. The amount of time you spend with documenting a single photo or video is many orders of magnitude worse than FB content mods.
    - henrygrew 6y ago
      This sounds very grevious, it's sad that a human being has to do this work
pantaloony 6y ago
Radical idea: if your service requires subjecting lots and lots of workers to scarring images, the correct solution is for that service not to exist. Facebook doesn’t have to let people post material to their site with very low barriers to entry. Granted they have to if they want to continue to be what they are and to make tons of money... but maybe they shouldn’t continue to be what they are and make tons of money if this kind of abuse is necessarily coupled to those outcomes. “Well this sucks but we can’t keep operating if we don’t do it”. Well... sure but the solution is right there in that statement. Don’t keep operating.
- rosywoozlechan 6y ago
  Any place anyone can add any user content has this problem. Anything with an input box and a file upload would qualify as your not existing solution.
  - pantaloony 6y ago
    1) that seems basically fine, but also 2) non-commercial efforts run by hobbyists and gated from the general public ought to be Ok. If you want to run a PHPBB site and subject yourself to harmful garbage by letting randos write to your server, well, go nuts. [edit] thought experiment: how many of the people making tons of money off Facebook—c-suite, major shareholders—would find some other way to make money if continuing to make Facebook mega bucks meant they had to do this 5 days a week? What would we think of any of them who chose “bring on the trauma, I want those sweet greenbacks” and kept it up for years?
  - jbrennan 6y ago
    I think there’s a difference that depends on scale and privacy. If your network is private (or something more akin to p2p / messaging) this is much less of an issue, especially if you have strong blocking controls. Users moderate themselves. I think the problems that Facebook, Youtube, etc are seeing stem from the fact that the content is much more public, thus in much more need of moderation.
CM30 6y ago
It's probably a bit controversial to say this, but haven't large sites had to deal with these problems for years by the time the likes of Facebook/Twitter/YouTube/Instagram/whatever came around? It's obviously horrible doing this as a job and it's obviously a bit more common on Facebook than say, a large old school internet forum, but... at least in this case people are paid to do this. Reddit mods and old school forum mods have to deal with this stuff for free.
- tmpz22 6y ago
  Right but the difference is the sheer amount of money involved and the politicization that comes with that. If Facebook was worth 100m nobody would care. If facebook only has 10m users nobody would care. It's the scale that completely breaks the legal and political frameworks the our society leans on.
  - wheelie_boy 6y ago
    I'd say another difference is the percentage of your attention is being devoted to this kind of thing. Presumably reddit or phpbb mods aren't spending 40 hr/wk looking at gore.
    - Nasrudith 6y ago
      Wouldn't that suggest then that the "ideal" way to do it is a very part time job then with high volume? Like say 5 hours a week. It would be a scheduling and logistical nightmare of course but it would give more "dispersion time" for trauma.
      - wheelie_boy 6y ago
        I'd be interested to see if the tooling around moderation could also be improved. For example, could the images start heavily blurred, with a circle of clarity that opens when you click. Or something like that, which would harm throughput, but be more humane.
op03 6y ago
Given auto translate works reasonably well these days, how much commonality does Negative Content over different regions and languages have? Anyone know? Is it like 10%? 80%? There must be, by now, a whole lot of data on whats getting flagged region/language wise. Maybe we need a Cloudflare for Content.
- latchkey 6y ago
  Depends on the language. In my experience Vietnamese -> English is horrid and comes out as mostly garbage. Interestingly, the reverse has been better from what my Vietnamese friends tell me.
rosywoozlechan 6y ago
Post gore, rape and torture in places where people don't want to see that and you should lose the right to have internet access. Criminalize this behavior.
- MattGaiser 6y ago
  It is criminal, especially the child porn. But you have to catch them first.
- joncp 6y ago
  That sounds attractive until one starts to delve into the implementation details. How would you implement it?
  - rosywoozlechan 6y ago
    The US has laws with punishments that make it illegal for people to use computers or to use certain online services already. Just do more of this, but broaden the categories to include posting visually horrifying content that's known to cause PTSD to places where people don't expect to see that.
    - bromuro 6y ago
      Sorry, no.
    - WrongThinkerNo5 6y ago
      "Just do more of this, but broaden the categories to include posting…" and here you are already thrusting yourself down the slippery slope. You went from "gore, rape, and torture", all of which inherently not easy to prove in and of themselves, but in just a matter of minutes you've already "broadened the categories" to "visually horrifying" (based on who's opinion?) and which is "known to cause PTSD" (known and determined by whom? Not even to mention that PTSD is a professional construct), but also including having to know where people do and do not expect to see something that is something based on the above mentioned ambiguous determinations. It did not take you long at all to just vault yourself down the full on dictator slope. Next up, let me guess, you want to prevent people's access to the internet if they say things you don't agree with because "word are violence" and then "micro-aggressions are violence", simply because you have the control to abuse. Unfortunately that is already the situation right now, but it does not need to be made worse, because it will only thrust us all down a path of overplaying the unrelenting abuse that I can assure you people will not like the inevitable consequences of one bit. Let me also remind you that the US has the supreme law of the land that supersedes all other laws, the First Amendment to the Constitution that states that no law may abridge the freedom of speech, regardless of whether you like it or not.
rememberlenny 6y ago
Donohoe is back.
irrational 6y ago
I'm surprised there hasn't been a revenge movie where one of these moderators use their access to find out where these people live and start hunting them down like some sort of batman-like person. I could see a movie like this being a means of making the more general population aware of the terrible stuff content moderators have to watch.
- slg 6y ago
  This is totally not what you are talking about, but the closest thing I have seen to that is the Netflix docuseries Don't Fuck With Cats. It hits on a lot of the same topics. The primary differences are that the hunters are Facebook users and not moderators and they are hunting the person to get justice and not violent revenge.
  - Tiksi 6y ago
    There's a show called Darknet https://en.wikipedia.org/wiki/Darknet_(TV_series) https://en.wikipedia.org/wiki/Darknet_(TV_series) that's somewhat along these lines too, but it's very much in the horror genre.
- milquetoastaf 6y ago
  The "Unfriended" horror flicks are like this
- reportgunner 6y ago
  Guns Akimbo is kind of similar but with a twist. I did not like the movie much to be honest.
ed25519FUUU 6y ago
I can't even barely read this article before becoming extremely sad for these people and the individuals (especially children) who were abused in these videos. I hope everyone involve gets the healing they need. It's not the flat-earther and somesuch conspirary communities that worry me. It's the people who share and get enjoyment from this kind of content, and the networks which enable it.
jbrennan 6y ago
In my opinion this is an argument in favour of shutting Facebook (and every other large social network) down. This sort of work is damaging and abusive, and nobody should have to endure it just so we can have social networks. I understand there are some jobs in the world that need to deal with dark stuff (like law enforcement), but social networks just aren’t worth the human cost.
- dangus 6y ago
  That’s a non-solution. What you’re proposing is not just shutting down social networks, it’s shutting down any website that involves user content, anything that allows photo/video upload, comments, or any kind of user interaction. That’s impossible. You point out that public safety jobs are view more “worth it,” and certainly they are, but that logic brings up the question of who judges what job is worth undergoing trauma. In other words, is a subway or freight train driver’s job “worth it,” if they have to see someone commit suicide on the tracks? What about crime scene cleanup companies? Funeral services? Bus drivers? Truck drivers? Nobody’s going to agree on where to draw the line in the sand. A more realistic solution might be to make comprehensive support systems, mental health resources, and treatment a legally mandated, completely free service provided to any employee that works in these kinds of fields. Finally, I think there are most certainly people out there who are not as sensitive and affected by this content who would be candidates for these kinds of roles. Perhaps there’s a way to test for that sensitivity before the real job starts.
  - jbrennan 6y ago
    I’m actually not proposing shutting down any website that allows uploaded content, just large / public sites that require this sort of moderation. Not every site gets this stuff uploaded to it. The more private the network, the less need for this kind of company-led moderation. As far as “worth it” goes, some people have to be exposed to it so long as we have law enforcement (but I’m certainly open to alternatives here). I’m not sure the train operator is a fair comparison, because seeing a suicide is an exceptional circumstance in their job, it’s not the norm. The content moderators, however, are sadly expected to be exposed to traumatizing content as part of their job description — it’s essentially the point of their job. There are plenty of kinds of work we deem as hazardous to people’s health, and thus are either banned or regulated. I’m not sure if there’s a healthy way to expose people in these moderator jobs to the traumatizing content they face. It just doesn’t seem worth the tradeoff to endanger them like this.
steve76 6y ago
wow you should work in the ER room. Code Orange! Breathing mush with eyes !!!!!!!
intopieces 6y ago
Every time you use Facebook, see an advertisement there, or click it, every time you share content on Facebook and get others to engage with that platform, you are contributing to a platform that is directly responsible for human psychological harm, in many different ways. Same for Twitter, and for Reddit, and Instagram... and probably TikTok. I don’t believe the use of these platforms can be considered ethical.
- falcrist 6y ago
  Does this logic also apply to commenting on hacker news? Or is it specifically the platforms you mentioned? What specifically is the reason you consider those particular platforms to be unethical, and what is the solution. It's not enough to say "social media bad". You can say that about anything including the internet as a whole. We need the reasons it's bad and solutions.
  - intopieces 6y ago
    > What specifically is the reason you consider those particular platforms to be unethical, and what is the solution. These platforms are so large that they require hundreds of thousands of moderators in order to remove vile, illegal content, causing trauma for a vast underclass of foreign workers. Here’s a few solutions: Limit the number of people that a person can be friends with to 150, the Dunbar number. Restrict the posting of photos to those which have been identified as having you or your friends in the picture, or no humans at all. Require permission from all human participants to post. Charge a fee for access.
    - falcrist 6y ago
      Why is size the determining factor, though? Is it just that you're worried about the outsourcing of content moderation? Because other than than the problems are the same. Someone will always need to moderate the content.
      - intopieces 6y ago
msapaydin 6y ago
What I am wondering, and this is probably a dumb question, is why this has not been automatized? Can't content moderation be done with modern and strong machine learning based systems? There must be plenty of training data on this, and just like a spam filter which does not require humans in most cases, this should also be automatable. Why is it not?
- amelius 6y ago
  Perhaps because the training of these systems requires human supervisors?
  - msapaydin 6y ago
    well, just as in spam filters, some supervisors would be necessary to automate things in the beginning, but eventually this should be less needed?
    - amelius 6y ago
      Yes, one would hope so. My guess is that we're still in the data-collection phase. It's probably a tough problem because you want the number of false-negatives to be extremely small.
- pixelbath 6y ago
  >this is probably a dumb question Not dumb, but I do think it's not thought-through. You're proposing a simple solution to what is a hugely complex problem, and throwing ML at it just isn't going to work. To get "plenty of training data," a human still has to classify all of that, leading to the problem of viewing that much unpalatable content by a human. You also have to train your network, which requires humans to verify the accuracy of training, hence viewing the content again. If it were as easy as text-based spam filtering, this wouldn't even be a discussion.
  - msapaydin 6y ago
    I am guessing that by now plenty of data has already been generated which could be used to train future systems..
Havoc 6y ago
It also sounds (from other articles) like people are seeing specific clips multiple times. Surely a bit of semi-AI filtering should be able to blacklist the regular stuff
amelius 6y ago
Perhaps it's a solution to use brainwave headsets. So whenever the viewer would get too much negative stimulation, the video would stop and the viewer could take a break.
- droopyEyelids 6y ago
  The headset could also put a mark into their performance review for failing to develop a sufficient coping strategy.
DavidVoid 6y ago
That was good article and I think something really needs to be done to make content moderation more humane. This quote from the article “It gets to a point where you can eat your lunch while watching a video of someone dying. … But at the end of the day, you still have to be human.” reminded me of a similar article published by WIRED six years ago [1]. Eight years after the fact, Jake Swearingen can still recall the video that made him quit. He was 24 years old and between jobs in the Bay Area when he got a gig as a moderator for a then-new startup called VideoEgg. Three days in, a video of an apparent beheading came across his queue. “Oh fuck! I've got a beheading!” he blurted out. A slightly older colleague in a black hoodie casually turned around in his chair. “Oh,” he said, “which one?” At that moment Swearingen decided he did not want to become a connoisseur of beheading videos. “I didn't want to look back and say I became so blasé to watching people have these really horrible things happen to them that I'm ironic or jokey about it.” [1] https://www.wired.com/2014/10/content-moderation/ https://www.wired.com/2014/10/content-moderation/
- kevinskii 6y ago
  It would be great if there were a way to make content moderation more humane, but perhaps this is like saying that it would be great if we could make hospital emergency room work more humane. It is unavoidably traumatic, and ER staff are known to detach and use dark humor as a coping mechanism. Content moderation is a similarly noble and difficult profession.
  - agentdrtran 6y ago
    ER staff are paid a lot more, are sometimes even in a union, and also get the benefit of helping people and seeing the effect of their help.
    - Spooky23 6y ago
      Ever hang out with an ER staffer? Most of their workload is bullshit. People with colds and sore throats, depressing people using ER as primary care, assholes using 911 to score Medicaid cab vouchers. And at any time, any number of people can show up with any kind of personal tragedies from stokes to various traumas. My sister quit after a 13 year old bled out from a gsw, and she walked out of the room and got kicked in the head by an prisoner who had been stabbed after he bit the ears off of three other prisoners, and broke the arm of a guard.
atlgator 6y ago
Is Facebook worth it? Is it worth the trauma?
- paulpauper 6y ago
  if it wasn't worth it, people would not do such jobs and people would not use Facebook, so apparently for a lot of people it is worth it.
S53Vflnr4n 6y ago
This should be the top comment: If you don’t recognize it, “rest of world” is a ridiculous corporate term commonly used in global business operations. It’s a catch-all phrase that means, basically, “everyone else.” And it generally represents billions of people outside of the Western world. We know that their stories matter. The term “rest of world” is a symptom of a larger problem: a Western-centric worldview that leaves innumerable insights, opportunities and complexity out of the conversation.
zitterbewegung 6y ago
This is also relevant: https://www.theguardian.com/technology/2017/jan/11/microsoft-employees-child-abuse-lawsuit-ptsd https://www.theguardian.com/technology/2017/jan/11/microsoft...
ericjang 6y ago
I read somewhere long ago that there have been incidents of FBI employees tasked with reviewing child pornography [1] who saw so much that they became desensitized or even aroused by it. Does anyone know if this is well-documented, or was this an exception rather than a common occurence? [1] https://www.fbi.gov/history/famous-cases/operation-innocent-images https://www.fbi.gov/history/famous-cases/operation-innocent-...
ehnto 6y ago
There are some things I will explicitly avoid in projects I'm working on. One of them is allowing picture or video uploads to users if the service is free. There's no low-risk, cost effective solution for moderating it, and in my country we also don't have safe-harbour type laws, so any content on the platform is your responsibility.
- jcun4128 6y ago
  I had considered using services like Google Cloud Vision or other services for explicit image detection anyway. Most websites it seems when you upload an image it's immediately available, I always wonder if it goes through some basic moderation system or just waits to get reported. I've done tagging jobs on MTurk before it's weird... seeing random people's images.
  - ehnto 6y ago
    I would wager most wait for things to get reported rather than be pre-emptive about it. I wonder if MTurk could ever be fast enough for an approval process on something like Instagram. I think vetting users rather than content is probably the most efficient method of community curation though. If you have an approval process or other ways to validate that users aren't nefarious then you would reduce your workload by a bunch. Most bad images seem to come from burner accounts. Unfortunately the only thing that comes to mind is a social credit as a service style system and maybe that's not where we want to steer this ship...
    - jcun4128 6y ago
      Yeah the tagging jobs I did were weird like "find the baby" haha... And other obvious tagging videos. I can get from the user side about the burner accounts. I feel a physical response when I get denied to post a first post say in a sub and I'm like "Excuse me..." But yeah that's definitely discouraging/limiting to stick around before contributing.
paulpauper 6y ago
how do you become a Facebook moderator?
weresquirrel 6y ago
If you haven’t seen The Cleaners, I highly recommend it. https://thoughtmaybe.com/the-cleaners/ https://thoughtmaybe.com/the-cleaners/
08-15 6y ago
The censors working for Facebook are traumatized by their work. Strange. I feel nothing. Neither pity, nor sympathy, just nothing.
fareesh 6y ago
Some great undercover video of what goes on at Cognizant here, as far as bias in moderating actions is concerned: https://www.politicalite.com/latest/facebook-employee-if-someone-is-wearing-a-maga-hat-i-am-going-to-delete-them-for-terrorism/ https://www.politicalite.com/latest/facebook-employee-if-som... In this video it is shown that regarding beheadings, Facebook made it a point to send their moderation contractors a memo that the image that was frequently reported of President Trump's face with a knife at the neck where he is being beheaded was an exception to the content policy on beheadings/violence/incitement because it was considered to be "art" by some museum somewhere in Portland - which from what I can see on TV - appears to be an extremely volatile and violent place in some parts. There is a reference to a cartoon post of Elmer Fudd shooting a cartoon gun at Beto O'Rourke with the text "I'm here for your guns" at the top - which was not given any similar exception. I don't think Facebook or their partners are serious about this kind of thing at all since they seem ok with advocating violence and gore when it resonates with some personal opinions. Also noteworthy is that the name of this whistleblower garners fewer search results on Google, than the name of another whistleblower Eric Ciaramella who, if you uttered a-la Voldemort, would get you banned instantly. Broadly, content policy at these websites seems to be a lord of the flies, unprincipled hackjob of whatever the political machinery at the organization deems worthy of steering society in the direction of their whims. Given the influence and power of Facebook and other tech platforms, this should worry people. I'm less inclined to be sympathetic to these folks and the difficulty of their jobs, given the apparent willingness to become social constructionists with the power that they wield so irresponsibly.
afaq404alam 6y ago
I hope they are tagging these videos and someone somewhere is working on a neural network to classify the new ones.
shadowgovt 6y ago
This story is an interesting contrast to the story of Twitter dropping users for spreading the QAnon conspiracy. Clearly, there's near-universal agreement that some moderation is fine; only a handful of (downvoted) comments on this thread saying "The easiest way to fix this problem would be for Facebook to stop censoring posts and let users block what they don't want to see." But we clearly don't think that's a good solution in this case. The question of whether QAnon should also be blocked is one of degree, not quality.
maerF0x0 6y ago
I cant help wonder about the economic(choices) side of this around 1) Should this victimization be taken into account when punishing offenders eventually tried and 2) Is the damage done to the content moderators marginally worth keeping the marginally caught content off the platform (recall there already is automation doing the bulk of the work)... I do think we need to think about the side effects of laws where some US citizen demands that Facebook must employ content moderations else be liable for hosting it, it doesnt result in Facebook executives, engineers et al. bearing the burden. It results in some poor person in a 3rd world country bearing the burden. Kind of like the environmental aspect of things I would think the best investment is in total prevention of the heinous acts in the first place. ie, it's much cheaper to prevent CO2 than it is to clean it up after the fact.
zacharycohn 6y ago
About 9 years ago, I found myself at a party consisting mostly of content moderators for The Cheezburger Network. I walked out of there shellshocked.
Thorentis 6y ago
Only a human should be able to decide what other humans can and cannot see. A future where computers are responsible for "deciding" what information we consume about the world is not one I want to live in. Computers are already being used as tools of censorship, we don't need it to be expanded further.
- ece 6y ago
  If computers can help with spam, they can certainly help with other well-defined and well-audited types of illegal and misinformed content. It would be worse to not invest in more automation, even malpractice.
totetsu 6y ago
Is this a case of trying to solve social problems with technical solutions? Maybe a platform like FB that is not run by the community that uses it simply cannot regulate users behaviour like a real social community can.
alecco 6y ago
Funny how they can automatically flag copyright for videos now even when x mirrored but they can't identify these videos. I can't believe there's so much original content for this. Maybe they could share a db of hashes of known bad videos across sites and government agencies.

218 points by donohoe 6y ago

Can we build better AI off of the data these moderators have generated to make their lives easier? Can you crowdsource the moderation task to double check what the ai is flagging?

Sebb767 6y ago
> Can you crowdsource the moderation task[...]? I highly doubt that. These reported posts might contain private data or highly shocking content; putting that on a public platform, especially associated with the name of Facebook, is very unlikely to happen.
- munificent 6y ago
  > putting that on a public platform, especially associated with the name of Facebook Is that not what the person who originally posted the content being moderated is doing?
  - Sebb767 6y ago
    It still would open up a gore/scare platform (think LiveLeak), since only the bad content is reported. Sure, most might be tame, but it wouldn't take long for gore seekers to come.
  - chronid 6y ago
    > Is that not what the person who originally posted the content being moderated is doing? The person that posted the content does not expect a random person from the internet to see a video of his children because it was accidentally flagged by an AI as child porn. :)
ceejayoz 6y ago
> Can you crowdsource the moderation task to double check what the ai is flagging? "Facebook sent me child porn" is probably not a headline Facebook wants out there.
codekansas 6y ago
Easier said than done https://time.com/5739688/facebook-hate-speech-languages/ https://time.com/5739688/facebook-hate-speech-languages/
filoleg 6y ago
>Can you crowdsource the moderation task to double check what the ai is flagging? This would help with false positives, which is a noble goal. But it would not help with false negatives, which is imo a very disastrous consequence in this specific scenario. Something that isn't CP got marked as CP? Sure, a moderator can manually doublecheck that flag and override if needed. Something that is CP didn't get marked as CP? You got a problem on your hands, because there is no way to manually doublecheck that, since that would mean manually checking every single post marked as "no issues here", which gets us back to square one of manual moderation of all posts. Not even mentioning the exposure of FB users to CP that it would bring, which would mentally ruin quite a lot of people and bring very negative press to FB on the level of severity that we haven't seen before.
- landryraccoon 6y ago
  Speaking as someone who has to deal with supervised learning models, human beings falsely tag positive examples as negative all the time. Merely having "average" human beings tag examples doesn't guarantee perfect recall. Or to restate, sites that rely purely on manual moderation of content that breaks rules still has content that breaks rules leak through all the time.
uniqueid 6y ago
> Can we build better AI Nope. That's what the people in charge of our social media companies keep trying, over and over, and it doesn't work at all. AI-moderation is like the Ring is to Golem for them. They can't accept that "cheap and fair" moderation isn't currently possible. > Can you crowdsource the moderation task This is the clear answer, imo, and it isn't obvious to me why these C-level execs avoid it. If I had to make a guess, maybe they think engaging with millions of their users about moderation would open a can of worms.
- shadowgovt 6y ago
  When the goal is to prevent users from seeing this content, crowd-sourcing moderation defeats the purpose. Users don't want the back-stop to be "a critical mass of community members object;" for content like this, they want to see zero of it, ever, and will choose to use another site that satisfies that hard constraint if FB cannot.
  - uniqueid 6y ago
    > they want to see zero of it, ever Many want that, but very few (probably nobody at all, by this point) expect it. > When the goal is to prevent users from > seeing this content, crowd-sourcing moderation > defeats the purpose. It's not actually all-or-nothing, and if it were, it would defeat it no less frequently than paid moderation already does. People tolerated toxic internet content before social media not because they never encountered any, but because they encountered much less. No system can ever stop 100% of users from seeing 100% of the material they find objectionable. Every user has their own idea of what material is beyond the pale, and even if we invent AI that is competent at blocking content, and capable of tailoring its filters per-user, it will still fail some of the time, because users are people, and people's tastes change over time. Aside from that, the way Youtube, Reddit, Twitter et al, currently operate, a user who reports objectionable content typically waits days, weeks, or years (or forever) for the company to take action. If you give a user a little real agency, it goes a long way to mitigate their displeasure over occasional exposure to unwanted content.
    - 6y ago
msapaydin 6y ago
I also made a comment before I saw yours about creating a machine learning based system to automate this process, I think it must be fairly easy given the amount of data already available.

MattGaiser 6y ago

They should recruit people on Reddit for this. A large number of them seem perfectly fine with this type of content. There are dedicated communities to all these things. I do not mean this as a hit against Reddit users. It is just that there are small pockets of people who are psychologically capable of tolerating this or even enjoy it. It seems more ethical to hire people from /r/watchpeopledie.

dmalvarado 6y ago
Perhaps. I would argue that there is some difference between seeing someone fall off a cliff and watching someone being tortured to death.
- throwaway0a5e 6y ago
  Then they should run sidebar ads on the gore sites. The point is that there appears to be a subset of the population that is not traumatized by this stuff. We may as well give them an opportunity to put that difference to use. Edit: and saying things like "well clearly these people are already traumatized if they're seeking this out, it's not good for them, we shouldn't give these jobs to them" seems laughably paternalistic when the alternative is to traumatize more people by making them look at it instead.
  - ceejayoz 6y ago
    How do we know they're not traumatized by it? That they seek it out doesn't mean they're not messed up.
    - filoleg 6y ago
      ^this. I am not arguing one way or another, because I don't see strong evidence making me lean predominantly one way. But imo, it is at least just as possible that they are seeking out that kind of content voluntarily because they are already traumatized in the first place, so this is a coping mechanism. Maybe it helps dealing with that trauma. Maybe it makes the trauma worse. Maybe there was no trauma to begin with, and the person was seeking out that content simply due to morbid curiosity, but then it evolved into something else. Maybe there was no trauma to begin with and no trauma appeared after the exposure. Who knows. But just because there are communities on reddit with people seeking that kind of material and not going on crazy violent sprees doesn't mean that it doesn't negatively affect them. It could be the case, but we should probably look into that first before jumping to "obvious solution" claims. tl;dr: we gotta do more research and weigh pros and cons before simply declaring "we have an obvious solution, let's just hire all those gore content enthusiasts to moderate violent content, because they already do it on their own and don't seem to be going crazy in obvious ways due to it".

boringcuriosity 6y ago

Content moderators have always been in tough situations and they always have problems related to psychology and minds

shadowgovt 6y ago

I can't help but wonder what the numbers look like on this content. One of the things Facebook should have enough data to know is how prevalent this sort of thing is. Is deeply offensive content generated by a handful of users frequently, or many users less often? What's the volume look like?

filoleg 6y ago
Agreed. I would definitely love to see some sort of a basic data analysis blog post on this topic from FB, similar to how match.com used to publish data analysis posts on their blog back in the day.

Magodo 6y ago

I don't understand what's done with the contractors in the US after the successful lawsuit, surely all those jobs were just moved offshore? There's no mention of this at all in the article....

econcon 6y ago

How can I become a partime fb content moderator? Is it possible to become one online and work remotely? Or must work from the fb camp? I want to test if it can cause me any trauma

khazhoux 6y ago
I suppose the FB job board is the place to start. Did you check there?
talmr 6y ago
I'm not sure if you are being sarcastic so forgive me if this is off-base, but there are REAL people who are dealing with trauma from moderating content so you don't need to "test it out" and validate.

praveen9920 6y ago

The issue is as real as western countries dumping garbage in eastern countries just because they can pay them off. The local governments won't acknowledge this as problem because of the money flowing in and a lot of people's livelihood is dependent on this. But we should call it what it is, exploitation.

blargmaster33 6y ago
No, western countries are NOT dumping garbage. They are buying the service "waste disposal", they are NOT responsible for the supplier of that service just dumping the garage.
quadrifoliate 6y ago
I think in regards to India, the issue is a little more complex, and intertwined with social mores. This one sentence from the article is really important: > According to Dr. K. Jyothirmayi, a Hyderabad-based psychiatrist, stigmas, such as the perceived impact of a mental health diagnosis on one’s marital prospects, often prevent young Indians from seeking treatment. From growing up in India, my view is that meditation and the general state of the mind has been taken seriously for, well, centuries. But move the focus towards anything with 'psych' or 'mental' in the title and people in India will shy away from it even if it is affordable or free. "Seeing a mental health counselor" is a recipe for considerable social stigma in India [1], not an empowering practice like it's perceived as in the United States. The solution to this is hard, and in my opinion not something that Facebook or their Indian subcontractors can accomplish. ----------- [1] I'm talking "All your relatives will refer to you as the crazy person" or "Your girlfriend's parents will call off the engagement" levels of stigma, not just "I don't feel comfortable talking about it in a bar" levels. Like a lot of other things, families with higher levels of income and education are sometimes an exception.
Nasrudith 6y ago
The awkward ethical question is would they be better off without "exploitation"? Part of the reason companies are even willing to go through the overhead is because the costs are lower. Real life has always been filled with messed up situations where the punchline is "... and the best/worst part is it qualified as an improvement!".

lowmemcpu 6y ago

I recall a statistic from about 10 years ago that computer forensic investigators in law enforcement burn out after two years due to the trauma of the images they are exposed to.

throwaway0a5e 6y ago
There's likely some confounding factors. Pressing "go" on the overpriced software tools and then entering into evidence what you find is the lowest level of work in that field so the churn is going to naturally be very high as people move up or out. The pay also isn't that great.
- mschuster91 6y ago
  No, that is not the issue. Rather the issue is that even the hardest stuff on Facebook isn't remotely comparable to stuff of actual criminals, and the effort is wildly different: - Facebook: it's violating rules? Delete, next. - Forensic IT on a multi TB disk full with child porn: document every photo, what it shows, extract identifiable faces to cross reference with other content (to check for recurring places and victims), and the process is even more gory for video content. You have to watch every second or the defense can attempt "you didn't watch the video in full where the perp gives the victim an ice cream at the end" or whatever else. The amount of time you spend with documenting a single photo or video is many orders of magnitude worse than FB content mods.
  - henrygrew 6y ago
    This sounds very grevious, it's sad that a human being has to do this work

pantaloony 6y ago

Radical idea: if your service requires subjecting lots and lots of workers to scarring images, the correct solution is for that service not to exist. Facebook doesn’t have to let people post material to their site with very low barriers to entry. Granted they have to if they want to continue to be what they are and to make tons of money... but maybe they shouldn’t continue to be what they are and make tons of money if this kind of abuse is necessarily coupled to those outcomes. “Well this sucks but we can’t keep operating if we don’t do it”. Well... sure but the solution is right there in that statement. Don’t keep operating.

rosywoozlechan 6y ago
Any place anyone can add any user content has this problem. Anything with an input box and a file upload would qualify as your not existing solution.
- pantaloony 6y ago
  1) that seems basically fine, but also 2) non-commercial efforts run by hobbyists and gated from the general public ought to be Ok. If you want to run a PHPBB site and subject yourself to harmful garbage by letting randos write to your server, well, go nuts. [edit] thought experiment: how many of the people making tons of money off Facebook—c-suite, major shareholders—would find some other way to make money if continuing to make Facebook mega bucks meant they had to do this 5 days a week? What would we think of any of them who chose “bring on the trauma, I want those sweet greenbacks” and kept it up for years?
- jbrennan 6y ago
  I think there’s a difference that depends on scale and privacy. If your network is private (or something more akin to p2p / messaging) this is much less of an issue, especially if you have strong blocking controls. Users moderate themselves. I think the problems that Facebook, Youtube, etc are seeing stem from the fact that the content is much more public, thus in much more need of moderation.

CM30 6y ago

It's probably a bit controversial to say this, but haven't large sites had to deal with these problems for years by the time the likes of Facebook/Twitter/YouTube/Instagram/whatever came around? It's obviously horrible doing this as a job and it's obviously a bit more common on Facebook than say, a large old school internet forum, but... at least in this case people are paid to do this. Reddit mods and old school forum mods have to deal with this stuff for free.

tmpz22 6y ago
Right but the difference is the sheer amount of money involved and the politicization that comes with that. If Facebook was worth 100m nobody would care. If facebook only has 10m users nobody would care. It's the scale that completely breaks the legal and political frameworks the our society leans on.
- wheelie_boy 6y ago
  I'd say another difference is the percentage of your attention is being devoted to this kind of thing. Presumably reddit or phpbb mods aren't spending 40 hr/wk looking at gore.
  - Nasrudith 6y ago
    Wouldn't that suggest then that the "ideal" way to do it is a very part time job then with high volume? Like say 5 hours a week. It would be a scheduling and logistical nightmare of course but it would give more "dispersion time" for trauma.
    - wheelie_boy 6y ago
      I'd be interested to see if the tooling around moderation could also be improved. For example, could the images start heavily blurred, with a circle of clarity that opens when you click. Or something like that, which would harm throughput, but be more humane.

op03 6y ago

Given auto translate works reasonably well these days, how much commonality does Negative Content over different regions and languages have? Anyone know? Is it like 10%? 80%? There must be, by now, a whole lot of data on whats getting flagged region/language wise. Maybe we need a Cloudflare for Content.

latchkey 6y ago
Depends on the language. In my experience Vietnamese -> English is horrid and comes out as mostly garbage. Interestingly, the reverse has been better from what my Vietnamese friends tell me.

rosywoozlechan 6y ago

Post gore, rape and torture in places where people don't want to see that and you should lose the right to have internet access. Criminalize this behavior.

MattGaiser 6y ago
It is criminal, especially the child porn. But you have to catch them first.
joncp 6y ago
That sounds attractive until one starts to delve into the implementation details. How would you implement it?
- rosywoozlechan 6y ago
  The US has laws with punishments that make it illegal for people to use computers or to use certain online services already. Just do more of this, but broaden the categories to include posting visually horrifying content that's known to cause PTSD to places where people don't expect to see that.
  - bromuro 6y ago
    Sorry, no.
  - WrongThinkerNo5 6y ago
    "Just do more of this, but broaden the categories to include posting…" and here you are already thrusting yourself down the slippery slope. You went from "gore, rape, and torture", all of which inherently not easy to prove in and of themselves, but in just a matter of minutes you've already "broadened the categories" to "visually horrifying" (based on who's opinion?) and which is "known to cause PTSD" (known and determined by whom? Not even to mention that PTSD is a professional construct), but also including having to know where people do and do not expect to see something that is something based on the above mentioned ambiguous determinations. It did not take you long at all to just vault yourself down the full on dictator slope. Next up, let me guess, you want to prevent people's access to the internet if they say things you don't agree with because "word are violence" and then "micro-aggressions are violence", simply because you have the control to abuse. Unfortunately that is already the situation right now, but it does not need to be made worse, because it will only thrust us all down a path of overplaying the unrelenting abuse that I can assure you people will not like the inevitable consequences of one bit. Let me also remind you that the US has the supreme law of the land that supersedes all other laws, the First Amendment to the Constitution that states that no law may abridge the freedom of speech, regardless of whether you like it or not.

rememberlenny 6y ago

Donohoe is back.

irrational 6y ago

I'm surprised there hasn't been a revenge movie where one of these moderators use their access to find out where these people live and start hunting them down like some sort of batman-like person. I could see a movie like this being a means of making the more general population aware of the terrible stuff content moderators have to watch.

slg 6y ago
This is totally not what you are talking about, but the closest thing I have seen to that is the Netflix docuseries Don't Fuck With Cats. It hits on a lot of the same topics. The primary differences are that the hunters are Facebook users and not moderators and they are hunting the person to get justice and not violent revenge.
- Tiksi 6y ago
  There's a show called Darknet https://en.wikipedia.org/wiki/Darknet_(TV_series) https://en.wikipedia.org/wiki/Darknet_(TV_series) that's somewhat along these lines too, but it's very much in the horror genre.
milquetoastaf 6y ago
The "Unfriended" horror flicks are like this
reportgunner 6y ago
Guns Akimbo is kind of similar but with a twist. I did not like the movie much to be honest.

ed25519FUUU 6y ago

I can't even barely read this article before becoming extremely sad for these people and the individuals (especially children) who were abused in these videos. I hope everyone involve gets the healing they need. It's not the flat-earther and somesuch conspirary communities that worry me. It's the people who share and get enjoyment from this kind of content, and the networks which enable it.

jbrennan 6y ago

In my opinion this is an argument in favour of shutting Facebook (and every other large social network) down. This sort of work is damaging and abusive, and nobody should have to endure it just so we can have social networks. I understand there are some jobs in the world that need to deal with dark stuff (like law enforcement), but social networks just aren’t worth the human cost.

dangus 6y ago
That’s a non-solution. What you’re proposing is not just shutting down social networks, it’s shutting down any website that involves user content, anything that allows photo/video upload, comments, or any kind of user interaction. That’s impossible. You point out that public safety jobs are view more “worth it,” and certainly they are, but that logic brings up the question of who judges what job is worth undergoing trauma. In other words, is a subway or freight train driver’s job “worth it,” if they have to see someone commit suicide on the tracks? What about crime scene cleanup companies? Funeral services? Bus drivers? Truck drivers? Nobody’s going to agree on where to draw the line in the sand. A more realistic solution might be to make comprehensive support systems, mental health resources, and treatment a legally mandated, completely free service provided to any employee that works in these kinds of fields. Finally, I think there are most certainly people out there who are not as sensitive and affected by this content who would be candidates for these kinds of roles. Perhaps there’s a way to test for that sensitivity before the real job starts.
- jbrennan 6y ago
  I’m actually not proposing shutting down any website that allows uploaded content, just large / public sites that require this sort of moderation. Not every site gets this stuff uploaded to it. The more private the network, the less need for this kind of company-led moderation. As far as “worth it” goes, some people have to be exposed to it so long as we have law enforcement (but I’m certainly open to alternatives here). I’m not sure the train operator is a fair comparison, because seeing a suicide is an exceptional circumstance in their job, it’s not the norm. The content moderators, however, are sadly expected to be exposed to traumatizing content as part of their job description — it’s essentially the point of their job. There are plenty of kinds of work we deem as hazardous to people’s health, and thus are either banned or regulated. I’m not sure if there’s a healthy way to expose people in these moderator jobs to the traumatizing content they face. It just doesn’t seem worth the tradeoff to endanger them like this.

steve76 6y ago

wow you should work in the ER room. Code Orange! Breathing mush with eyes !!!!!!!

intopieces 6y ago

Every time you use Facebook, see an advertisement there, or click it, every time you share content on Facebook and get others to engage with that platform, you are contributing to a platform that is directly responsible for human psychological harm, in many different ways. Same for Twitter, and for Reddit, and Instagram... and probably TikTok. I don’t believe the use of these platforms can be considered ethical.

falcrist 6y ago
Does this logic also apply to commenting on hacker news? Or is it specifically the platforms you mentioned? What specifically is the reason you consider those particular platforms to be unethical, and what is the solution. It's not enough to say "social media bad". You can say that about anything including the internet as a whole. We need the reasons it's bad and solutions.
- intopieces 6y ago
  > What specifically is the reason you consider those particular platforms to be unethical, and what is the solution. These platforms are so large that they require hundreds of thousands of moderators in order to remove vile, illegal content, causing trauma for a vast underclass of foreign workers. Here’s a few solutions: Limit the number of people that a person can be friends with to 150, the Dunbar number. Restrict the posting of photos to those which have been identified as having you or your friends in the picture, or no humans at all. Require permission from all human participants to post. Charge a fee for access.
  - falcrist 6y ago
    Why is size the determining factor, though? Is it just that you're worried about the outsourcing of content moderation? Because other than than the problems are the same. Someone will always need to moderate the content.
    - intopieces 6y ago

msapaydin 6y ago

What I am wondering, and this is probably a dumb question, is why this has not been automatized? Can't content moderation be done with modern and strong machine learning based systems? There must be plenty of training data on this, and just like a spam filter which does not require humans in most cases, this should also be automatable. Why is it not?

amelius 6y ago
Perhaps because the training of these systems requires human supervisors?
- msapaydin 6y ago
  well, just as in spam filters, some supervisors would be necessary to automate things in the beginning, but eventually this should be less needed?
  - amelius 6y ago
    Yes, one would hope so. My guess is that we're still in the data-collection phase. It's probably a tough problem because you want the number of false-negatives to be extremely small.
pixelbath 6y ago
>this is probably a dumb question Not dumb, but I do think it's not thought-through. You're proposing a simple solution to what is a hugely complex problem, and throwing ML at it just isn't going to work. To get "plenty of training data," a human still has to classify all of that, leading to the problem of viewing that much unpalatable content by a human. You also have to train your network, which requires humans to verify the accuracy of training, hence viewing the content again. If it were as easy as text-based spam filtering, this wouldn't even be a discussion.
- msapaydin 6y ago
  I am guessing that by now plenty of data has already been generated which could be used to train future systems..

Havoc 6y ago

It also sounds (from other articles) like people are seeing specific clips multiple times. Surely a bit of semi-AI filtering should be able to blacklist the regular stuff

amelius 6y ago

Perhaps it's a solution to use brainwave headsets. So whenever the viewer would get too much negative stimulation, the video would stop and the viewer could take a break.

droopyEyelids 6y ago
The headset could also put a mark into their performance review for failing to develop a sufficient coping strategy.

DavidVoid 6y ago

That was good article and I think something really needs to be done to make content moderation more humane. This quote from the article “It gets to a point where you can eat your lunch while watching a video of someone dying. … But at the end of the day, you still have to be human.” reminded me of a similar article published by WIRED six years ago [1]. Eight years after the fact, Jake Swearingen can still recall the video that made him quit. He was 24 years old and between jobs in the Bay Area when he got a gig as a moderator for a then-new startup called VideoEgg. Three days in, a video of an apparent beheading came across his queue. “Oh fuck! I've got a beheading!” he blurted out. A slightly older colleague in a black hoodie casually turned around in his chair. “Oh,” he said, “which one?” At that moment Swearingen decided he did not want to become a connoisseur of beheading videos. “I didn't want to look back and say I became so blasé to watching people have these really horrible things happen to them that I'm ironic or jokey about it.” [1] https://www.wired.com/2014/10/content-moderation/ https://www.wired.com/2014/10/content-moderation/

kevinskii 6y ago
It would be great if there were a way to make content moderation more humane, but perhaps this is like saying that it would be great if we could make hospital emergency room work more humane. It is unavoidably traumatic, and ER staff are known to detach and use dark humor as a coping mechanism. Content moderation is a similarly noble and difficult profession.
- agentdrtran 6y ago
  ER staff are paid a lot more, are sometimes even in a union, and also get the benefit of helping people and seeing the effect of their help.
  - Spooky23 6y ago
    Ever hang out with an ER staffer? Most of their workload is bullshit. People with colds and sore throats, depressing people using ER as primary care, assholes using 911 to score Medicaid cab vouchers. And at any time, any number of people can show up with any kind of personal tragedies from stokes to various traumas. My sister quit after a 13 year old bled out from a gsw, and she walked out of the room and got kicked in the head by an prisoner who had been stabbed after he bit the ears off of three other prisoners, and broke the arm of a guard.

atlgator 6y ago

Is Facebook worth it? Is it worth the trauma?

paulpauper 6y ago
if it wasn't worth it, people would not do such jobs and people would not use Facebook, so apparently for a lot of people it is worth it.

S53Vflnr4n 6y ago

This should be the top comment: If you don’t recognize it, “rest of world” is a ridiculous corporate term commonly used in global business operations. It’s a catch-all phrase that means, basically, “everyone else.” And it generally represents billions of people outside of the Western world. We know that their stories matter. The term “rest of world” is a symptom of a larger problem: a Western-centric worldview that leaves innumerable insights, opportunities and complexity out of the conversation.

zitterbewegung 6y ago

This is also relevant: https://www.theguardian.com/technology/2017/jan/11/microsoft-employees-child-abuse-lawsuit-ptsd https://www.theguardian.com/technology/2017/jan/11/microsoft...

ericjang 6y ago

I read somewhere long ago that there have been incidents of FBI employees tasked with reviewing child pornography [1] who saw so much that they became desensitized or even aroused by it. Does anyone know if this is well-documented, or was this an exception rather than a common occurence? [1] https://www.fbi.gov/history/famous-cases/operation-innocent-images https://www.fbi.gov/history/famous-cases/operation-innocent-...

ehnto 6y ago

There are some things I will explicitly avoid in projects I'm working on. One of them is allowing picture or video uploads to users if the service is free. There's no low-risk, cost effective solution for moderating it, and in my country we also don't have safe-harbour type laws, so any content on the platform is your responsibility.

jcun4128 6y ago
I had considered using services like Google Cloud Vision or other services for explicit image detection anyway. Most websites it seems when you upload an image it's immediately available, I always wonder if it goes through some basic moderation system or just waits to get reported. I've done tagging jobs on MTurk before it's weird... seeing random people's images.
- ehnto 6y ago
  I would wager most wait for things to get reported rather than be pre-emptive about it. I wonder if MTurk could ever be fast enough for an approval process on something like Instagram. I think vetting users rather than content is probably the most efficient method of community curation though. If you have an approval process or other ways to validate that users aren't nefarious then you would reduce your workload by a bunch. Most bad images seem to come from burner accounts. Unfortunately the only thing that comes to mind is a social credit as a service style system and maybe that's not where we want to steer this ship...
  - jcun4128 6y ago
    Yeah the tagging jobs I did were weird like "find the baby" haha... And other obvious tagging videos. I can get from the user side about the burner accounts. I feel a physical response when I get denied to post a first post say in a sub and I'm like "Excuse me..." But yeah that's definitely discouraging/limiting to stick around before contributing.

paulpauper 6y ago

how do you become a Facebook moderator?

weresquirrel 6y ago

If you haven’t seen The Cleaners, I highly recommend it. https://thoughtmaybe.com/the-cleaners/ https://thoughtmaybe.com/the-cleaners/

08-15 6y ago

The censors working for Facebook are traumatized by their work. Strange. I feel nothing. Neither pity, nor sympathy, just nothing.

fareesh 6y ago

Some great undercover video of what goes on at Cognizant here, as far as bias in moderating actions is concerned: https://www.politicalite.com/latest/facebook-employee-if-someone-is-wearing-a-maga-hat-i-am-going-to-delete-them-for-terrorism/ https://www.politicalite.com/latest/facebook-employee-if-som... In this video it is shown that regarding beheadings, Facebook made it a point to send their moderation contractors a memo that the image that was frequently reported of President Trump's face with a knife at the neck where he is being beheaded was an exception to the content policy on beheadings/violence/incitement because it was considered to be "art" by some museum somewhere in Portland - which from what I can see on TV - appears to be an extremely volatile and violent place in some parts. There is a reference to a cartoon post of Elmer Fudd shooting a cartoon gun at Beto O'Rourke with the text "I'm here for your guns" at the top - which was not given any similar exception. I don't think Facebook or their partners are serious about this kind of thing at all since they seem ok with advocating violence and gore when it resonates with some personal opinions. Also noteworthy is that the name of this whistleblower garners fewer search results on Google, than the name of another whistleblower Eric Ciaramella who, if you uttered a-la Voldemort, would get you banned instantly. Broadly, content policy at these websites seems to be a lord of the flies, unprincipled hackjob of whatever the political machinery at the organization deems worthy of steering society in the direction of their whims. Given the influence and power of Facebook and other tech platforms, this should worry people. I'm less inclined to be sympathetic to these folks and the difficulty of their jobs, given the apparent willingness to become social constructionists with the power that they wield so irresponsibly.

afaq404alam 6y ago

I hope they are tagging these videos and someone somewhere is working on a neural network to classify the new ones.

shadowgovt 6y ago

This story is an interesting contrast to the story of Twitter dropping users for spreading the QAnon conspiracy. Clearly, there's near-universal agreement that some moderation is fine; only a handful of (downvoted) comments on this thread saying "The easiest way to fix this problem would be for Facebook to stop censoring posts and let users block what they don't want to see." But we clearly don't think that's a good solution in this case. The question of whether QAnon should also be blocked is one of degree, not quality.

maerF0x0 6y ago

I cant help wonder about the economic(choices) side of this around 1) Should this victimization be taken into account when punishing offenders eventually tried and 2) Is the damage done to the content moderators marginally worth keeping the marginally caught content off the platform (recall there already is automation doing the bulk of the work)... I do think we need to think about the side effects of laws where some US citizen demands that Facebook must employ content moderations else be liable for hosting it, it doesnt result in Facebook executives, engineers et al. bearing the burden. It results in some poor person in a 3rd world country bearing the burden. Kind of like the environmental aspect of things I would think the best investment is in total prevention of the heinous acts in the first place. ie, it's much cheaper to prevent CO2 than it is to clean it up after the fact.

zacharycohn 6y ago

About 9 years ago, I found myself at a party consisting mostly of content moderators for The Cheezburger Network. I walked out of there shellshocked.

Thorentis 6y ago

Only a human should be able to decide what other humans can and cannot see. A future where computers are responsible for "deciding" what information we consume about the world is not one I want to live in. Computers are already being used as tools of censorship, we don't need it to be expanded further.

ece 6y ago
If computers can help with spam, they can certainly help with other well-defined and well-audited types of illegal and misinformed content. It would be worse to not invest in more automation, even malpractice.

totetsu 6y ago

Is this a case of trying to solve social problems with technical solutions? Maybe a platform like FB that is not run by the community that uses it simply cannot regulate users behaviour like a real social community can.

alecco 6y ago

Funny how they can automatically flag copyright for videos now even when x mirrored but they can't identify these videos. I can't believe there's so much original content for this. Maybe they could share a db of hashes of known bad videos across sites and government agencies.