7 ms·

I don’t have any experience with this, so take this with a pinch of salt. What seems novel about moderation is the frequency that you confront disturbing thi

by sd9 2y ago

I don’t have any experience with this, so take this with a pinch of salt. What seems novel about moderation is the frequency that you confront disturbing things. I imagine companies like Meta have such good automated moderation that what remains to be viewed by a human is practically a firehose of almost certainly disturbing shit. And as soon as you’re done with one post, the next is right there. I doubt moderators spend more than 30 seconds on the average image, which is an awful lot of stuff to see in one day. A doctor just isn’t exposed to that sort of imagery at the same rate.

prng2021 2y ago
“I would imagine that companies like Meta have such good automated moderation that what remains to be viewed by a human is practically a firehose of almost certainly disturbing shit.” This doesn’t make sense to me. Their automated content moderation is so good that it’s unable to detect “almost certainly disturbing shit”? What kind of amazing automation only works with subtleties but not certainties?
- sd9 2y ago
  I assumed that, at the margin, Meta would prioritise reducing false negatives. In other words, they would prefer that as many legitimate posts are published as possible. So the things that are flagged for human review would be on the boundary, but trend more towards disturbing than legitimate, on the grounds that the human in the loop is there to try and publish as many posts as possible, which means sifting through a lot of disturbing stuff that the AI is not sure about. There’s also the question of training the models - the classifiers may need labelled disturbing data. But possibly not these days. However, yes, I expect the absolute most disturbing shit to never be seen by a human. — Again, literally no experience, just a guy on the internet pressing buttons on a keyboard.
  - gruez 2y ago
    >In other words, they would prefer that as many legitimate posts are published as possible. They'd prefer that as many posts are published, but they probably also don't mind some posts being removed if it meant saving a buck. When canada and australia implemented a "link tax", they were happy to ban all news content to avoid paying it.
    - sd9 2y ago
      Yes, Meta are economically incentivised to reduce the number of human reviews (assuming the cost of improving the model is worthwhile). This probably means fewer human reviewers reviewing a firehose, not the same number of human reviewers reviewing content at a slower rate.
itake 2y ago
I’d think the higher density/frequency of disturbing content would cause people to be desensitized. I never seen blood or gore in my life and find seeing it shocking. But I’d imagine gore is a weekly situation for surgeons.
- sd9 2y ago
  I agree. But that might be comorbid with PTSD. It’s probably not good for you to be _that_ desensitised to this sort of thing. I also feel like there’s something intangible regarding intent that makes moderation different from being a doctor. It’s hard for me to put into words, but doctors see gore because they can hopefully do something to help the individual involved. Moderators see gore but are powerless to help the individual, they can only prevent others from seeing the gore.
  - diggan 2y ago
    It's also the type of gore that matters. Some of the worst stuff I've seen wasn't the worst because of the visuals, but because of the audio. Hearing people begging for their life while being executed surely would feel different to even a surgeon who might be used to digging around in people's bodies.
  - itake 2y ago
    There are many common situations where professionals are helpless, like people that needs to clean up dead bodies after an accident.
    - sangnoir 2y ago
      Imagine if this becomes a specialized, remote job where one tele-operates the brain and blood scrubbing robot all workday long, accident, after accident after accident. I am sure they'd get PTSD too, airey, sometime it's just oil and coolant, but there's still a lot of body-tissue involved.
      - itake 2y ago
lm28469 2y ago
Also a doctor is paid $$$$$ and it mostly is a vocational job Content moderator is a min wage job with bad working hours, no psychological support, and you spend your day looking at rape, child porn, torture and executions.
- gruez 2y ago
  >Also a doctor is paid $$$$$ >Content moderator is a min wage job So it's purely a monetary dispute? >bad working hours, no psychological support, and you spend your day looking at rape, child porn, torture and executions. Many other jobs have the same issues, though admittedly with less frequency, but where do you draw the line?
  - diggan 2y ago
    > but where do you draw the line? How about grouping the jobs into two categories: A) Causes PTSD and B) Doesn't cause PTSD If a job as a constantly high percentage of people ending up with PTSD, then they aren't equipped well enough to handle it, by the company who employs them.
    - gruez 2y ago
      >How about grouping the jobs into two categories: A) Causes PTSD and B) Doesn't cause PTSD I fail to see how this addresses my previous questions of "it's purely a monetary dispute?" and "where do you draw the line?". If a job "Causes PTSD" (whatever that means), then what? Are you entitled to hazard pay? Does this work out in the end to a higher minimum wage for certain jobs? Moreover, we don't have similar classifications for other hazards, some of which are arguably worse. For instance, dying is probably worse than getting PTSD, but the most dangerous jobs have pay that's well below the national median wage[1][2]. Should workers in those jobs be able to sue for redress as well? [1] https://www.ishn.com/articles/112748-top-25-most-dangerous-jobs-in-the-united-states https://www.ishn.com/articles/112748-top-25-most-dangerous-j... [2] https://www.bls.gov/oes/current/oes_nat.htm https://www.bls.gov/oes/current/oes_nat.htm
      - 2y ago
ajb 2y ago
Also doctors are very frequently able to do something about it. Being powerless is a huge factor in mental illness.
tossandthrow 2y ago
> I imagine companies like Meta have such good automated moderation that what remains to be viewed by a human is practically a firehose of almost certainly disturbing shit. On the contrary I would expect that it would be the edge cases that they were shown - why loop in a content moderator if you an be sure that it is prohibited ont he platform without exposing a content moderator? In this light, it might make sense why they sue: They are there more as a political org so that facebook can say: "We employ 140 moderators in Kenya alone!" while they do indifferent work that facebook already can filter out.
- Retric 2y ago
  Even if 1% of images are disturbing that’s multiple per hour, let anyone across months. US workman’s comp covers PTSD acquired on the job, and these kinds of jobs are rife with it.
- aoanevdus 2y ago
  Content moderation also involves reading text, so you’d imagine that there’s a benefit to having people who can label data and provide ground truth in any language you’re moderating. Even with images, you can have different policies in different places or the cultural context can be relevant somehow (eg. some country makes you ban blasphemy). Also, I have heard of outsourcing to Kenya just to save cost. Living is cheaper there so you can hire a desk worker for less. Don’t know where the insistence you’d only hire Kenyans for political reasons comes from.
- pizza 2y ago
  > They are there more as a political org so that facebook can say: "We employ 140 moderators in Kenya alone!" while they do indifferent work that facebook already can filter out. Why assume they're just token diversity hires who don't do useful work..? Have you ever built an automated content moderation system before? Let me tell you something about them if not: no matter how good your automated moderation tool, it is pretty much always trivial for someone with familiarity with its inputs and outputs to come up with an input it mis-predicts embarrassingly badly. And you know what makes the biggest difference.. is humans specifying the labels.
mrweasel 2y ago
> I imagine companies like Meta have such good automated moderation I imagine that they have a system that is somewhere between shitty and none functional. This is the company that will more often than not flag marketplace posts as "Selling animal", either completely at random or because the pretty obvious "from an animal free home" phrase is used. If they can't get this basic text parsing correct, how can you expect them to correctly flag images with any real sense of accuracy?

by sd9 2y ago

prng2021 2y ago

“I would imagine that companies like Meta have such good automated moderation that what remains to be viewed by a human is practically a firehose of almost certainly disturbing shit.” This doesn’t make sense to me. Their automated content moderation is so good that it’s unable to detect “almost certainly disturbing shit”? What kind of amazing automation only works with subtleties but not certainties?

sd9 2y ago
I assumed that, at the margin, Meta would prioritise reducing false negatives. In other words, they would prefer that as many legitimate posts are published as possible. So the things that are flagged for human review would be on the boundary, but trend more towards disturbing than legitimate, on the grounds that the human in the loop is there to try and publish as many posts as possible, which means sifting through a lot of disturbing stuff that the AI is not sure about. There’s also the question of training the models - the classifiers may need labelled disturbing data. But possibly not these days. However, yes, I expect the absolute most disturbing shit to never be seen by a human. — Again, literally no experience, just a guy on the internet pressing buttons on a keyboard.
- gruez 2y ago
  >In other words, they would prefer that as many legitimate posts are published as possible. They'd prefer that as many posts are published, but they probably also don't mind some posts being removed if it meant saving a buck. When canada and australia implemented a "link tax", they were happy to ban all news content to avoid paying it.
  - sd9 2y ago
    Yes, Meta are economically incentivised to reduce the number of human reviews (assuming the cost of improving the model is worthwhile). This probably means fewer human reviewers reviewing a firehose, not the same number of human reviewers reviewing content at a slower rate.

itake 2y ago

I’d think the higher density/frequency of disturbing content would cause people to be desensitized. I never seen blood or gore in my life and find seeing it shocking. But I’d imagine gore is a weekly situation for surgeons.

sd9 2y ago
I agree. But that might be comorbid with PTSD. It’s probably not good for you to be _that_ desensitised to this sort of thing. I also feel like there’s something intangible regarding intent that makes moderation different from being a doctor. It’s hard for me to put into words, but doctors see gore because they can hopefully do something to help the individual involved. Moderators see gore but are powerless to help the individual, they can only prevent others from seeing the gore.
- diggan 2y ago
  It's also the type of gore that matters. Some of the worst stuff I've seen wasn't the worst because of the visuals, but because of the audio. Hearing people begging for their life while being executed surely would feel different to even a surgeon who might be used to digging around in people's bodies.
- itake 2y ago
  There are many common situations where professionals are helpless, like people that needs to clean up dead bodies after an accident.
  - sangnoir 2y ago
    Imagine if this becomes a specialized, remote job where one tele-operates the brain and blood scrubbing robot all workday long, accident, after accident after accident. I am sure they'd get PTSD too, airey, sometime it's just oil and coolant, but there's still a lot of body-tissue involved.
    - itake 2y ago

lm28469 2y ago

Also a doctor is paid $$$$$ and it mostly is a vocational job Content moderator is a min wage job with bad working hours, no psychological support, and you spend your day looking at rape, child porn, torture and executions.

gruez 2y ago
>Also a doctor is paid $$$$$ >Content moderator is a min wage job So it's purely a monetary dispute? >bad working hours, no psychological support, and you spend your day looking at rape, child porn, torture and executions. Many other jobs have the same issues, though admittedly with less frequency, but where do you draw the line?
- diggan 2y ago
  > but where do you draw the line? How about grouping the jobs into two categories: A) Causes PTSD and B) Doesn't cause PTSD If a job as a constantly high percentage of people ending up with PTSD, then they aren't equipped well enough to handle it, by the company who employs them.
  - gruez 2y ago
    >How about grouping the jobs into two categories: A) Causes PTSD and B) Doesn't cause PTSD I fail to see how this addresses my previous questions of "it's purely a monetary dispute?" and "where do you draw the line?". If a job "Causes PTSD" (whatever that means), then what? Are you entitled to hazard pay? Does this work out in the end to a higher minimum wage for certain jobs? Moreover, we don't have similar classifications for other hazards, some of which are arguably worse. For instance, dying is probably worse than getting PTSD, but the most dangerous jobs have pay that's well below the national median wage[1][2]. Should workers in those jobs be able to sue for redress as well? [1] https://www.ishn.com/articles/112748-top-25-most-dangerous-jobs-in-the-united-states https://www.ishn.com/articles/112748-top-25-most-dangerous-j... [2] https://www.bls.gov/oes/current/oes_nat.htm https://www.bls.gov/oes/current/oes_nat.htm
    - 2y ago

ajb 2y ago

Also doctors are very frequently able to do something about it. Being powerless is a huge factor in mental illness.

tossandthrow 2y ago

> I imagine companies like Meta have such good automated moderation that what remains to be viewed by a human is practically a firehose of almost certainly disturbing shit. On the contrary I would expect that it would be the edge cases that they were shown - why loop in a content moderator if you an be sure that it is prohibited ont he platform without exposing a content moderator? In this light, it might make sense why they sue: They are there more as a political org so that facebook can say: "We employ 140 moderators in Kenya alone!" while they do indifferent work that facebook already can filter out.

Retric 2y ago
Even if 1% of images are disturbing that’s multiple per hour, let anyone across months. US workman’s comp covers PTSD acquired on the job, and these kinds of jobs are rife with it.
aoanevdus 2y ago
Content moderation also involves reading text, so you’d imagine that there’s a benefit to having people who can label data and provide ground truth in any language you’re moderating. Even with images, you can have different policies in different places or the cultural context can be relevant somehow (eg. some country makes you ban blasphemy). Also, I have heard of outsourcing to Kenya just to save cost. Living is cheaper there so you can hire a desk worker for less. Don’t know where the insistence you’d only hire Kenyans for political reasons comes from.
pizza 2y ago
> They are there more as a political org so that facebook can say: "We employ 140 moderators in Kenya alone!" while they do indifferent work that facebook already can filter out. Why assume they're just token diversity hires who don't do useful work..? Have you ever built an automated content moderation system before? Let me tell you something about them if not: no matter how good your automated moderation tool, it is pretty much always trivial for someone with familiarity with its inputs and outputs to come up with an input it mis-predicts embarrassingly badly. And you know what makes the biggest difference.. is humans specifying the labels.

mrweasel 2y ago

> I imagine companies like Meta have such good automated moderation I imagine that they have a system that is somewhere between shitty and none functional. This is the company that will more often than not flag marketplace posts as "Selling animal", either completely at random or because the pretty obvious "from an animal free home" phrase is used. If they can't get this basic text parsing correct, how can you expect them to correctly flag images with any real sense of accuracy?