32 ms·

I'm wondering if there are precedents in other domains. There are other jobs where you do see disturbing things as part of your duty. E.g. doctors, cops, first

• whaleofatw2022 2y ago
  From those I know that worked in the industry, contractor systems are frequently abused to avoid providing the right level of counseling/support to moderators.
• sd9 2y ago
  I don’t have any experience with this, so take this with a pinch of salt. What seems novel about moderation is the frequency that you confront disturbing things. I imagine companies like Meta have such good automated moderation that what remains to be viewed by a human is practically a firehose of almost certainly disturbing shit. And as soon as you’re done with one post, the next is right there. I doubt moderators spend more than 30 seconds on the average image, which is an awful lot of stuff to see in one day. A doctor just isn’t exposed to that sort of imagery at the same rate.
  • prng2021 2y ago
    “I would imagine that companies like Meta have such good automated moderation that what remains to be viewed by a human is practically a firehose of almost certainly disturbing shit.” This doesn’t make sense to me. Their automated content moderation is so good that it’s unable to detect “almost certainly disturbing shit”? What kind of amazing automation only works with subtleties but not certainties?
    • sd9 2y ago
      I assumed that, at the margin, Meta would prioritise reducing false negatives. In other words, they would prefer that as many legitimate posts are published as possible. So the things that are flagged for human review would be on the boundary, but trend more towards disturbing than legitimate, on the grounds that the human in the loop is there to try and publish as many posts as possible, which means sifting through a lot of disturbing stuff that the AI is not sure about. There’s also the question of training the models - the classifiers may need labelled disturbing data. But possibly not these days. However, yes, I expect the absolute most disturbing shit to never be seen by a human. — Again, literally no experience, just a guy on the internet pressing buttons on a keyboard.
      • gruez 2y ago
        >In other words, they would prefer that as many legitimate posts are published as possible. They'd prefer that as many posts are published, but they probably also don't mind some posts being removed if it meant saving a buck. When canada and australia implemented a "link tax", they were happy to ban all news content to avoid paying it.
        • sd9 2y ago
          Yes, Meta are economically incentivised to reduce the number of human reviews (assuming the cost of improving the model is worthwhile). This probably means fewer human reviewers reviewing a firehose, not the same number of human reviewers reviewing content at a slower rate.
  • itake 2y ago
    I’d think the higher density/frequency of disturbing content would cause people to be desensitized. I never seen blood or gore in my life and find seeing it shocking. But I’d imagine gore is a weekly situation for surgeons.
    • sd9 2y ago
      I agree. But that might be comorbid with PTSD. It’s probably not good for you to be _that_ desensitised to this sort of thing. I also feel like there’s something intangible regarding intent that makes moderation different from being a doctor. It’s hard for me to put into words, but doctors see gore because they can hopefully do something to help the individual involved. Moderators see gore but are powerless to help the individual, they can only prevent others from seeing the gore.
      • diggan 2y ago
        It's also the type of gore that matters. Some of the worst stuff I've seen wasn't the worst because of the visuals, but because of the audio. Hearing people begging for their life while being executed surely would feel different to even a surgeon who might be used to digging around in people's bodies.
      • itake 2y ago
        There are many common situations where professionals are helpless, like people that needs to clean up dead bodies after an accident.
        • sangnoir 2y ago
          Imagine if this becomes a specialized, remote job where one tele-operates the brain and blood scrubbing robot all workday long, accident, after accident after accident. I am sure they'd get PTSD too, airey, sometime it's just oil and coolant, but there's still a lot of body-tissue involved.
          • itake 2y ago
  • lm28469 2y ago
    Also a doctor is paid $$$$$ and it mostly is a vocational job Content moderator is a min wage job with bad working hours, no psychological support, and you spend your day looking at rape, child porn, torture and executions.
    • gruez 2y ago
      >Also a doctor is paid $$$$$ >Content moderator is a min wage job So it's purely a monetary dispute? >bad working hours, no psychological support, and you spend your day looking at rape, child porn, torture and executions. Many other jobs have the same issues, though admittedly with less frequency, but where do you draw the line?
      • diggan 2y ago
        > but where do you draw the line? How about grouping the jobs into two categories: A) Causes PTSD and B) Doesn't cause PTSD If a job as a constantly high percentage of people ending up with PTSD, then they aren't equipped well enough to handle it, by the company who employs them.
        • gruez 2y ago
          >How about grouping the jobs into two categories: A) Causes PTSD and B) Doesn't cause PTSD I fail to see how this addresses my previous questions of "it's purely a monetary dispute?" and "where do you draw the line?". If a job "Causes PTSD" (whatever that means), then what? Are you entitled to hazard pay? Does this work out in the end to a higher minimum wage for certain jobs? Moreover, we don't have similar classifications for other hazards, some of which are arguably worse. For instance, dying is probably worse than getting PTSD, but the most dangerous jobs have pay that's well below the national median wage[1][2]. Should workers in those jobs be able to sue for redress as well? [1] https://www.ishn.com/articles/112748-top-25-most-dangerous-jobs-in-the-united-states https://www.ishn.com/articles/112748-top-25-most-dangerous-j... [2] https://www.bls.gov/oes/current/oes_nat.htm https://www.bls.gov/oes/current/oes_nat.htm
          • 2y ago
  • ajb 2y ago
    Also doctors are very frequently able to do something about it. Being powerless is a huge factor in mental illness.
  • tossandthrow 2y ago
    > I imagine companies like Meta have such good automated moderation that what remains to be viewed by a human is practically a firehose of almost certainly disturbing shit. On the contrary I would expect that it would be the edge cases that they were shown - why loop in a content moderator if you an be sure that it is prohibited ont he platform without exposing a content moderator? In this light, it might make sense why they sue: They are there more as a political org so that facebook can say: "We employ 140 moderators in Kenya alone!" while they do indifferent work that facebook already can filter out.
    • Retric 2y ago
      Even if 1% of images are disturbing that’s multiple per hour, let anyone across months. US workman’s comp covers PTSD acquired on the job, and these kinds of jobs are rife with it.
    • aoanevdus 2y ago
      Content moderation also involves reading text, so you’d imagine that there’s a benefit to having people who can label data and provide ground truth in any language you’re moderating. Even with images, you can have different policies in different places or the cultural context can be relevant somehow (eg. some country makes you ban blasphemy). Also, I have heard of outsourcing to Kenya just to save cost. Living is cheaper there so you can hire a desk worker for less. Don’t know where the insistence you’d only hire Kenyans for political reasons comes from.
    • pizza 2y ago
      > They are there more as a political org so that facebook can say: "We employ 140 moderators in Kenya alone!" while they do indifferent work that facebook already can filter out. Why assume they're just token diversity hires who don't do useful work..? Have you ever built an automated content moderation system before? Let me tell you something about them if not: no matter how good your automated moderation tool, it is pretty much always trivial for someone with familiarity with its inputs and outputs to come up with an input it mis-predicts embarrassingly badly. And you know what makes the biggest difference.. is humans specifying the labels.
  • mrweasel 2y ago
    > I imagine companies like Meta have such good automated moderation I imagine that they have a system that is somewhere between shitty and none functional. This is the company that will more often than not flag marketplace posts as "Selling animal", either completely at random or because the pretty obvious "from an animal free home" phrase is used. If they can't get this basic text parsing correct, how can you expect them to correctly flag images with any real sense of accuracy?
• athrowaway3z 2y ago
  I'm not sure your comparisons are close enough to be considered precedents. My guess is even standing at the ambulance drive in of a big hospital, you'll not see as much horrors in a day as these people see in 30 minutes.
  • lazide 2y ago
    Outside of some specific cities, I can guarantee it. Even a busy Emergency Dept on Halloween night had only a small handful of bloody patients/trauma cases, and nothing truly horrific when I did my EMT rotation.
  • s1artibartfast 2y ago
    My friends who are paramedics have seen some horrific scenes. They have also been shot, stabbed, and suffered lifelong injuries. They are obviously not identical scenarios. They have similarities and they also have differences.
• fcmgr 2y ago
  A friend's friend is a paramedic and as far as I remember they can take the rest of a day off after witnessing death on duty and there's an obligatory consulation with a mental healthcare specialist. From reading the article, it seems like those moderators are seeing horrific things almost constantly throughout the day.
  • Ray20 2y ago
    Sounds crazy. Just imagine dying because paramedic responsible for your survival just wanted end his day early.
  • _qua 2y ago
    I've never heard of a policy like that for physicians and doubt it's common for paramedics. I work in an ICU and a typical day involves a death or resuscitation. We would run out of staff with that policy.
    • kranke155 2y ago
      Maybe a different country than yours ?
    • wongarsu 2y ago
      Maybe it's different in the US where ambulances cost money, but here in Germany the typical paramedic will see a wide variety of cases, with the vast majority of patients surviving the encounter. Giving your paramedic a day off after witnessing a death wouldn't break the bank. In the ICU or emergency room it would be a different story.
      • _qua 2y ago
        Ambulances cost money everywhere, it's just a matter of who is paying. Do we think paramedics in Germany are more susceptible to PTSD when patients die than ICU or ER staff, or paramedics anywhere?
        • wongarsu 2y ago
• insane_dreamer 2y ago
  at least in the US, those jobs - doctors, cops, firefighters, first responders - are well compensated (not sure about prison guards), certainly compared to content moderators who are at the bottom of the totem pole in an org like FB
  • deleted 2y ago
    [deleted]
  • caymanjim 2y ago
    What does compensation have to do with it? Is someone who stares at thousands of traumatizing, violent images every day going to be less traumatized if they're getting paid more?
    • t-3 2y ago
      Yes, they will be much more able to deal with the consequences of that trauma than someone who gets a pittance to do the same thing. A low-wage peon won't even be able to afford therapy if they need it.
    • unaindz 2y ago
      At least they can pay for therapy and afford to stop working or find another job
  • deleted 2y ago
    [deleted]
  • BoxFour 2y ago
    Shamefully, first responders are not well compensated - usually it's ~$20 an hour.
    • kevin_thibedeau 2y ago
      I've lived places where the cops make $100k+. It all depends on location.
• croes 2y ago
  Doctors, cops, first responders, prison guards see different horrible things. Content moderators see all of that.
• amelius 2y ago
  Don't forget judges, especially the ones in this case ... And it used to be priests who had to deal with all the nasty confessions.
  • Cumpiler69 2y ago
    Judges get loads of compensation and perks.
• siliconc0w 2y ago
  ER docs definitely get PTSD. Cops too.
• apitman 2y ago
  > surely banning social media or not moderating content aren't options Why not? What good has social media done that can't be accomplished in some other way, when weighed against the clear downsides? That's an honest question, I'm probably missing lots.
  • yodsanklai 2y ago
    Billions of people use them daily (facebook, instagram, X, youtube, tiktok...). Surely we could live without them like we did not long ago, but there's so much interest at play here that I don't see how they could be banned. It's akin to shutting down internet.
  • Nasrudith 2y ago
    Would we really be better served with media returning to being un-interactive and unresponsive? Where just getting on TV was something of note instead of everyone being on the internet. Where there was widespread downright cultish obsession with celebrities. The "We interrupt this news live from Iraq for celebrity getting out of prison news" era. I think not. The gatekeepers of the old media pretty much died for a reason, that they seriously sucked at their job. Open social media and everyone having a camera in their pocket is what allowed us to basically disprove UFO sightings and prove routine police abuse of power.
• smackay 2y ago
  I expect first responders rarely have to deal with the level of depravity mentioned in this Wired article from 2014, https://www.wired.com/2014/10/content-moderation/ https://www.wired.com/2014/10/content-moderation/ You probably DO NOT want to read it. There's a very good reason moderators are employed in far-away countries, where people are unlikely to have the resources to gain redress for the problems they have to deal with as a result.
• Spooky23 2y ago
  In many states, pension systems give police and fire service sworn members a 20 year retirement option. The military has similar arrangements. Doctors and lawyers can’t afford that sort of option, but they tend to embrace alcoholism at higher rates and collect ex-wives. Moderation may be worse in some ways. All day, every day, you see depravity at scale. You see things that shouldn’t be seen. Some of it you can stop, some you cannot due to the nature of the rules. I think banning social media isn’t an answer, but demanding change to the algorithms to reduce the engagement to high risk content is key.
• DocTomoe 2y ago
  Doctors, cops, first responders, prison guards, soldiers etc also just so happen to be the most likely groups of people to develop PTSD.
• evertedsphere 2y ago
  I think part of it is the disconnection from the things you're experiencing. A paramedic or firefighter is there, acting in the world, with a chance to do good and some understanding of how things can go wrong. A content moderator is getting images beamed into their brain that they have no preparation for, of situations that they have no connection to or power over.
  • rrr_oh_man 2y ago
    > A paramedic or firefighter is there, acting in the world, with a chance to do good and some understanding of how things can go wrong. That's bullshit. Ever talked to a paramedic or firefighter?
• sangnoir 2y ago
  As other sibling comments noted: most other jobs don't have the same frequent exposure to disturbing content. The closest are perhaps combat medics in an active warzone, but even they usually get some respite by being rotated.
• gcr 2y ago
  Burnout, PTSD, and high turnover are also hallmarks of suicide hotline operators. The difference? The reputable hotlines care a lot more about their employees' mental health, with mandatory breaks, free counseling, full healthcare benefits (including provisions for preventative mental health care like talk therapy). Another important difference is that suicide hotlines are decoupled from the profit motive. As more and more users sign up to use a social network, it gets more profitable and more and more load needs to be borne by the human moderation team. But suicide and mental health risk is (roughly) constant (or slowly increasing with societal trends, not product trends). There's also less of an incentive to minimize human moderation cost. In large companies, some directors view mod teams as a cost center that takes away from other ventures. In an organization dedicated only to suicide hotline response, a large share of the income (typically fundraising or donations) goes directly into the service itself.
• hnlmorg 2y ago
  Frequency plus lack of post traumatic support. A content moderator for Facebook will invariably see more depravity and more frequently than a doctor or police officer. And likely see far less support provided by their employers to emotionally deal with it too. This results in a circumstance where employees don’t have the time nor the tools to process.
• danielheath 2y ago
  Trauma isn’t just a function of what you’ve experienced, but also of what control you had over the situation and whether you got enough sleep. Being a doctor and helping people through horrific things is unlike helplessly watching them happen. IIRC, PTSD is far more common among people with sleep disorders, and it’s believed that the lack of good sleep is preventing upsetting memories from being processed.
• Nasrudith 2y ago
  I suspect what makes it different is the concentration to just the flagged is what turns it into especially traumatizing. Of course there is probably a bell curve of sorts for "experiences" vs "level of personal trauma". One incident might be enough for someone "weaker" to develop PTSD. Not a slight on the afflicted, just how things are. Casual Facebook viewers may stumble across something disturbing on it, but they certainly don't get PTSD at the rate of the poor moderators. Likewise the professionals usually have their own professional exposure levels to messed up stuff. Meanwhile child pornography investigation departments who have to catalogue the evidence are notorious for suffering poor mental health even with heavy measures taken. There is already the 'blacklist hash' approach to known bad images which can help reduce exposure. So they don't all need to be exposed to say the same cartel brutal execution video, the bot takes care of it. I don't know anything about Facebook's internal practices but I would presume they already are doing this or similar with their tech hiring base. Dilution is likely the answer for how to make it more palitible and less traumatizing. Keep the really bad stuff exposure at similar proportions to what other careers experience. Not having 'report button' and 'checking popular content' as separate tasks and teams would probably help a little bit. I suspect the moderators wouldn't be as traumatized if they just had to click through trending posts all day. A dillution approach would still have to deal with the logistical trade-offs for what could be viable. Increasing the moderation payroll a hundred-fold and making them work at effectively 1% efficiency would make for better moderator experiences, but Facebook would be understandably reluctant to go from 5% revenue content moderation budget to 50% revenue content moderation.
  • BehindBlueEyes 2y ago
    Is it even 5% of revenue? Curious where you got that number because it seems unbelievably high