5 ms·

This is a very good article and quite an eye opener. As an internet/web "user" for twenty years or so I'm familiar with the existence of "moderation", but had

by jrapdx3 11y ago

This is a very good article and quite an eye opener. As an internet/web "user" for twenty years or so I'm familiar with the existence of "moderation", but had only vague awareness of what moderators endured in that role. Reminiscent of traumatized 911 operators, it would be unsurprising that a subset of moderators develop PTSD-like symptoms related to that experience. Workers' comments quoted in the article lead me to speculate that there are significant mental health epidemiological questions re: current and former moderators. An effective automated approaches will provide a big advantage in narrowing down the "judgement space" that must decided by humans. To the extent that's possible, the key benefit provided is reducing the exposure of moderators to stressful situations as the article describes, and indeed that's a very helpful development. As the article points out the whole domain of moderation practices is a minefield. But now I wonder if there isn't also a risk of automated review making classification errors re: user behavior. Plausibly automated systems can be tuned finely enough to avoid serious errors, and support human oversight to catch and more easily resolve edge cases. Automated moderation systems will need to have such qualities in order to be able to reduce human burden as intended.

justifier 11y ago
south park had a bit where they dealt with this sort of workspace ptsd butters was in charge of filtering cartman's comments so cartman would only see material that would be unable to offend him cartman seemed to flourish with his new perceived support and popularity and butters deteriorated i thought it compelling to look at how they chose to do butters 'this material is offensive! you should deal with it instead of me' cartman was characteristically unconcerned about butter's well being as long as the supportive comments kept coming in i understand the satire loses some weight when you acknowledge that those that hire these moderators are mostly companies with a vested interest in keeping their users blissfully unaware
- staticfish 11y ago
  it's amazing how dead on-the-nail the southpark team do serious, humour ridden topical commentary on serious social and political issues. I think they'll go down as two of the greatest commentators of our generation.
- jrapdx3 11y ago
  Interesting illustration of the issue. As the article shows, users are not merely passive consumers, but an active component in the moderation process because users are first to encounter the offensive material. Sure, users don't want to see that stuff, but believable that it's the companies more than users striving to remain blissful unaware of the horrific crap that moderators have to deal with. As you point out, the issue is the effect on the moderator, if it's as bad and widespread as portrayed it may rise to the level of a public health problem.
  - justifier 11y ago
    yeah the difference is one of volume 10,000 individual users being inadvertently exposed to 1 piece of upsetting material could potentially mean 1 moderator then has to review 10,000 pieces of upsetting material also, the deluge mailroom syndrome, it just keeps coming honestly, computer vision is the only way to do this sort of moderation with any semblance of success and unfortunately i feel all of this human debris is just considered collateral damage until we get it functioning
- ar-jan 11y ago
  Here the episode: http://southpark.cc.com/full-episodes/s19e05-safe-space http://southpark.cc.com/full-episodes/s19e05-safe-space
  - justifier 11y ago
    yup, thanks for pointing out it was season 19 episode 5, safe space i failed to because i refuse to link to that hulu run abomination bah, south park, the show where the creators themselves put their work on the internet to be viewed freely and for free what a unique expression of the creators' personal beliefs a stance that had a profound impact on me, and my peers when hulu bought the rights i felt my skin grow cold, but i held out reservations in the hope that someone at hulu was just a fan and wanted to improve south park studio's backend whilst respecting the implicit wishes of the creators that their work is free and readily consumable by anyone regardless of manufactured scarcity or paywalls now this shit is the new normal: "Safe Space" This episode is currently not available at South Park Studios and There was an error playing this protected content. (Error code: 3365) i lament the loss of a future's exposure to the challenging ideas innate to the show hulu has zero respect for the series or its creators opinions on how open it has always been, and should be fuck you hulu
Karunamon 11y ago
There is probably no way to ask this question that doesn't sound dismissive at best, so please understand that is not the spirit this is intended in, but: it would be unsurprising that a subset of moderators develop PTSD-like symptoms related to that experience. Is that really the case? Most moderation is done on textual-based forums, like the one we're on right now. I'm having a hard time imagining the right combination of words on a screen to generate trauma in the requisite quantities for someone to wind up with a clinical disorder because of it. Personally, I've been on both sides of this coin. I've been the guy kicking trolls off of a decent sized board, I've been the guy getting kicked off because I annoyed the wrong person. At no point did it ever progress beyond internet drama. The meanest, nastiest person I could imagine could type in words along the lines of how they'd like to kill and fornicate with my mother, and my, and most other people's response I wager, would be "That's cool, kid. Bye now. Ban" Maybe there's a case to be made for becoming jaded after a while of dealing with the worst (hang out on the meta Stack Exchange sites for a publicly visible example) - but PTSD symptoms? To me, that both overly glorifies the troll "i can type words so good that I can make other people have legitimate mental breakdowns!" and makes light of the suffering by people who actually have PTSD (who have seen things like people dying). It feels like a lack of perspective, brought on by spending a lot of time online. Unlike real life, disconnecting from online drama is always a button press away.
- jrapdx3 11y ago
  I'm sorry, but I do believe you misunderstood the article and perhaps my comments on it as well. The article clearly describes moderators watching videos flagged as offensive, and these videos included "amateur and profession pornography" and many that contained "child abuse, beatings, and animal cruelty". Furthermore the article discusses moderators having to deal with videos shot during the Iranian revolution in 2009, including the murder of a young woman, "a shaky cell-phone video captured her horrific last moments: in it, blood pours from her eyes, pooling beneath her." The article presents many more examples of the enormously disturbing tasks assigned to moderators. Based on tens of years in clinical practice I'd consider it likely that viewing such material, especially seeing it frequently, can precipitate acute and chronic stress disorders in vulnerable individuals. I understand the point this is only hypothetical in the absence of systematically collected data, call it a clinician's hunch if you want, but still I'd put money on it. Rereading the article confirms for me that it is an excellent piece of journalism, perhaps even exceptional in the current era. Moderators were exposed to far more than words, in a way similar to 911 operators suffering exposure to trauma without being at the scene. Forum moderation has very little in common with the subject of the article. You are severely underestimating the effects of this kind of experience on people in such positions. You should learn more about stress disorders, the complexity of which is bound up with the unique and highly variable attributes of individuals. Some people are much more resilient than others making it misleading to generalize about how people will respond to given situations. Most of all, I strong suggest you (re)read the story and if capable, allow yourself to empathize with the plight of the moderators.
azernik 11y ago
The article addresses this specific point in two places even more bluntly than just through workers' comments: 'Members of the team quickly showed signs of stress — anxiety, drinking, trouble sleeping — and eventually managers brought in a therapist. As moderators described the images they saw each day, the therapist fell silent. The therapist, Mora-Blanco says, was "quite literally scared."' And: 'Beth Medina, who runs a program called SHIFT (Supporting Heroes in Mental Health Foundational Training), which has provided resilience training to Internet Crimes Against Children teams since 2009, details the severe health costs of sustained exposure to toxic images: isolation, relational difficulties, burnout, depression, substance abuse, and anxiety. "There are inherent difficulties doing this kind of work," Chen said, "because the material is so traumatic."'
- jrapdx3 11y ago
  Yes, good point. The observation that the material being moderated is potentially traumatic leads me to conjecture some subset of moderators would become disabled after exposure to the material. Resilience training is a good idea, but may not prevent bad outcomes.

by jrapdx3 11y ago

justifier 11y ago

south park had a bit where they dealt with this sort of workspace ptsd butters was in charge of filtering cartman's comments so cartman would only see material that would be unable to offend him cartman seemed to flourish with his new perceived support and popularity and butters deteriorated i thought it compelling to look at how they chose to do butters 'this material is offensive! you should deal with it instead of me' cartman was characteristically unconcerned about butter's well being as long as the supportive comments kept coming in i understand the satire loses some weight when you acknowledge that those that hire these moderators are mostly companies with a vested interest in keeping their users blissfully unaware

staticfish 11y ago
it's amazing how dead on-the-nail the southpark team do serious, humour ridden topical commentary on serious social and political issues. I think they'll go down as two of the greatest commentators of our generation.
jrapdx3 11y ago
Interesting illustration of the issue. As the article shows, users are not merely passive consumers, but an active component in the moderation process because users are first to encounter the offensive material. Sure, users don't want to see that stuff, but believable that it's the companies more than users striving to remain blissful unaware of the horrific crap that moderators have to deal with. As you point out, the issue is the effect on the moderator, if it's as bad and widespread as portrayed it may rise to the level of a public health problem.
- justifier 11y ago
  yeah the difference is one of volume 10,000 individual users being inadvertently exposed to 1 piece of upsetting material could potentially mean 1 moderator then has to review 10,000 pieces of upsetting material also, the deluge mailroom syndrome, it just keeps coming honestly, computer vision is the only way to do this sort of moderation with any semblance of success and unfortunately i feel all of this human debris is just considered collateral damage until we get it functioning
ar-jan 11y ago
Here the episode: http://southpark.cc.com/full-episodes/s19e05-safe-space http://southpark.cc.com/full-episodes/s19e05-safe-space
- justifier 11y ago
  yup, thanks for pointing out it was season 19 episode 5, safe space i failed to because i refuse to link to that hulu run abomination bah, south park, the show where the creators themselves put their work on the internet to be viewed freely and for free what a unique expression of the creators' personal beliefs a stance that had a profound impact on me, and my peers when hulu bought the rights i felt my skin grow cold, but i held out reservations in the hope that someone at hulu was just a fan and wanted to improve south park studio's backend whilst respecting the implicit wishes of the creators that their work is free and readily consumable by anyone regardless of manufactured scarcity or paywalls now this shit is the new normal: "Safe Space" This episode is currently not available at South Park Studios and There was an error playing this protected content. (Error code: 3365) i lament the loss of a future's exposure to the challenging ideas innate to the show hulu has zero respect for the series or its creators opinions on how open it has always been, and should be fuck you hulu

Karunamon 11y ago

There is probably no way to ask this question that doesn't sound dismissive at best, so please understand that is not the spirit this is intended in, but: it would be unsurprising that a subset of moderators develop PTSD-like symptoms related to that experience. Is that really the case? Most moderation is done on textual-based forums, like the one we're on right now. I'm having a hard time imagining the right combination of words on a screen to generate trauma in the requisite quantities for someone to wind up with a clinical disorder because of it. Personally, I've been on both sides of this coin. I've been the guy kicking trolls off of a decent sized board, I've been the guy getting kicked off because I annoyed the wrong person. At no point did it ever progress beyond internet drama. The meanest, nastiest person I could imagine could type in words along the lines of how they'd like to kill and fornicate with my mother, and my, and most other people's response I wager, would be "That's cool, kid. Bye now. Ban" Maybe there's a case to be made for becoming jaded after a while of dealing with the worst (hang out on the meta Stack Exchange sites for a publicly visible example) - but PTSD symptoms? To me, that both overly glorifies the troll "i can type words so good that I can make other people have legitimate mental breakdowns!" and makes light of the suffering by people who actually have PTSD (who have seen things like people dying). It feels like a lack of perspective, brought on by spending a lot of time online. Unlike real life, disconnecting from online drama is always a button press away.

jrapdx3 11y ago
I'm sorry, but I do believe you misunderstood the article and perhaps my comments on it as well. The article clearly describes moderators watching videos flagged as offensive, and these videos included "amateur and profession pornography" and many that contained "child abuse, beatings, and animal cruelty". Furthermore the article discusses moderators having to deal with videos shot during the Iranian revolution in 2009, including the murder of a young woman, "a shaky cell-phone video captured her horrific last moments: in it, blood pours from her eyes, pooling beneath her." The article presents many more examples of the enormously disturbing tasks assigned to moderators. Based on tens of years in clinical practice I'd consider it likely that viewing such material, especially seeing it frequently, can precipitate acute and chronic stress disorders in vulnerable individuals. I understand the point this is only hypothetical in the absence of systematically collected data, call it a clinician's hunch if you want, but still I'd put money on it. Rereading the article confirms for me that it is an excellent piece of journalism, perhaps even exceptional in the current era. Moderators were exposed to far more than words, in a way similar to 911 operators suffering exposure to trauma without being at the scene. Forum moderation has very little in common with the subject of the article. You are severely underestimating the effects of this kind of experience on people in such positions. You should learn more about stress disorders, the complexity of which is bound up with the unique and highly variable attributes of individuals. Some people are much more resilient than others making it misleading to generalize about how people will respond to given situations. Most of all, I strong suggest you (re)read the story and if capable, allow yourself to empathize with the plight of the moderators.

azernik 11y ago

The article addresses this specific point in two places even more bluntly than just through workers' comments: 'Members of the team quickly showed signs of stress — anxiety, drinking, trouble sleeping — and eventually managers brought in a therapist. As moderators described the images they saw each day, the therapist fell silent. The therapist, Mora-Blanco says, was "quite literally scared."' And: 'Beth Medina, who runs a program called SHIFT (Supporting Heroes in Mental Health Foundational Training), which has provided resilience training to Internet Crimes Against Children teams since 2009, details the severe health costs of sustained exposure to toxic images: isolation, relational difficulties, burnout, depression, substance abuse, and anxiety. "There are inherent difficulties doing this kind of work," Chen said, "because the material is so traumatic."'

jrapdx3 11y ago
Yes, good point. The observation that the material being moderated is potentially traumatic leads me to conjecture some subset of moderators would become disabled after exposure to the material. Resilience training is a good idea, but may not prevent bad outcomes.