7 ms·

Trauma Counselors Pressured to Divulge Confidential Info About Facebook Mods

109 points by jmsflknr 7y ago

a3n 7y ago
If it looks to an outsider like you work for company F, but you actually work for and are managed by company A, then company F has deemed you not worth the trouble of direct responsibility. They don't even see you as expendable, they just don't see you.
sciguy77 7y ago
I saw a heartbreaking YouTube mini-documentary on Facebook moderators.[1] The American workers seem to get the "lighter" flags like animal abuse, which is more than enough to cause trauma with daily exposure. I shudder to think what the offshore moderators go through when dealing with human/child abuse flags. In my view, Facebook clearly wants this to be a temporary evil, so they can use the data from human moderators as a training set for automated ML-based moderation. But I wonder how long people will have to endure this for those models to reach an acceptable efficacy. [1] https://www.youtube.com/watch?v=bDnjiNCtFk4 https://www.youtube.com/watch?v=bDnjiNCtFk4
- munk-a 7y ago
  > In my view, Facebook clearly wants this to be a temporary evil, so they can use the data from human moderators as a training set for automated ML-based moderation. But I wonder how long people will have to endure this for those models to reach an acceptable efficacy. Alternatively I wonder if it's worth it to put any number of human beings through this kind of suffering for something as worthless as social networking.
  - malandrew 7y ago
    Let's assume social networking is worthless per your judgment: Do you honestly think this problem will go away if centralized social networks go away?
    - munk-a 7y ago
      Yes, this is a new thing AFAICT. Magazines that frequently published reader content didn't have to sift through quite as vile content when going through letters - death threats and the like have always been a thing but somehow we've lost the cost of creating vile content in this modern era. I've never heard of anyone getting a visit from the FBI over vague death threats on facebook.
      - celtain 7y ago
        This kind of content came with the internet, not social networks. Traditional forum moderators have had to deal with really vile content for a long time now, along with the operators of almost any other media sharing website. Unless you want to make it illegal for individuals to widely share content without going through an editor, this kind of problem isn't going away any time soon.
        jessaustin 7y ago
        It really depends on how any particular site functions. Some sites function better than others. For example, "this kind of problem" doesn't seem to plague us here on HN.
      - Retric 7y ago
        You might find this interesting as it relates to a 44-month imprisonment and Facebook. https://www.forbes.com/sites/kashmirhill/2014/06/16/supreme-court-facebook-death-threats/#a0288081e617 https://www.forbes.com/sites/kashmirhill/2014/06/16/supreme-... On June 1, 2015, the Supreme Court reversed Elonis's conviction in an 8-1 decision. https://en.m.wikipedia.org/wiki/Elonis_v._United_States#Decision https://en.m.wikipedia.org/wiki/Elonis_v._United_States#Deci...
  - jessaustin 7y ago
    You, too, can walk away from Omelas.
    - michaelmrose 7y ago
      It's a beautiful story. https://www.utilitarianism.com/nu/omelas.pdf https://www.utilitarianism.com/nu/omelas.pdf The Ones Who Walk Away from Omelas By Ursula LeGuin
- benumbed 7y ago
  So, without a doubt, there is some chilling, mind altering content out there. Videos providing evidence of criminal acts of violence that inspire a deep disturbance of emotion lasting days, even among hardened viewers. But the truth is that it's all a pathetic joke. You can't police telephone calls, and the conference lines have to police themselves. This is the price of decentralization and the democratization of content creation. There can no longer persist any such dictatorship of standards and practices. There's only one maxim that applies: Don't talk to strangers. End of story. There are certain manifolds and convolutions that are unhygienic, and if you have a problem with that, don't build them and don't explore them. But, a network is a network is a network. It's a system of passages. Hallways with doors. In some respects, these policing activities and attempts to moderate a mob represent an obvious over-extension beyond the normal scope of operations. It's silly to even think it's possible. But hey, maybe Facebook is catching quite the government hand out, just to experiment with some "DARPA hard" problem solving. Personally, I find it laughable that anyone imagines this problem to be even remotely generalizable enough to fit a statistical model that the current state of "AI" buzzword juicing vaporware purveyors and hucksters might jazz hands a pitch deck with. Imagine: AI staring at clouds, noticing familiar shapes among the drifting cotton balls. How productive. You could waste an entire nuclear power plant's total output on such a thing, and get nothing for it. Attack the right problem at the root. These violent, malignant people torturing the animals and murdering on livestreams. Some will get through, and the internet will simply reflect reality. But make the world a better place. Don't spin your wheels papering over the blood stains. The reality is that these images videos are going to be shared and shared again. The cat's out of the bag. Accept the bald face of the human condition, warts and all.
- tjpnz 7y ago
  I'm guessing that most of the stuff heard about Facebook and ML is basically hype. Even after Christchurch the best they could do to counteract the spread was content fingerprinting. Human moderators are likely their longterm solution.
itronitron 7y ago
Hopefully a small group of licensed psychologists and therapists can set up a small office across from the FB office in Austin and start serving the moderators privately.
- staticautomatic 7y ago
  Yeah I'm sure the therapy will be covered by the amazing mental health benefits these moderators receive through the no-cost health insurance provided by Accenture. If not, they'll have no trouble at all paying out of pocket with that cushy $14.50/hr they're making!
  - perl4ever 7y ago
    There is something (in the US) askew about mental health coverage. I think that regulations were passed to try to get mental health conditions treated more like others, but it seems like there is a shortage of counselors, social workers, and doctors, much more than other kinds of medical professionals. I have this feeling that there's something about the incentive structure under the ACA that is choking off the availability, but I'm not sure what exactly it is. I'm very curious about the economics of running a mental health practice these days. If you can't pay $70-140/hr to talk to a social worker every week or two, they may well offer you the option of online chat for $150/month. But I feel like there is a mystery there, because why should it cost that much? And yet, the people who charge that much seem desperately overworked and barely functional. Where does the money go? Not to doctors, because everything is done by non-MDs these days. Not to insurance, because deductibles are in the thousands.
    - tru3_power 7y ago
      I’ve been wondering this too- especially since I’ve always heard that fields like psychology are “over saturated” and that there are tons of people with psychology degrees that can’t find jobs. Seems like if this is true there is a change for a win/win here (people get jobs and cheaper mental health care)
hos234 7y ago
Can we please start sending a few Facebook execs to jail? No fines. No apologies. Just jail time. They can get out of jail time if they send their kids to work in these centers.
mschuster91 7y ago
For what its worth, I believe that the most egregious offenders (think violence, murder, rape, child porn) should be directly referred to the police with all data Facebook has on them. This at least should slow down the torrent of shit that has to be moderated over the time...
- wincy 7y ago
  I mean in the US pictures of murder aren’t illegal.
  - mschuster91 7y ago
    Showing them to children is, at least in Germany, I do hope there are similar public decency laws in the US. In any case, people sharing murders who are not journalists or citizens trying to get attention for an issue (think gang violence or political oppression) should be investigated for their sanity.
- jeremyjh 7y ago
  Who is to say they are not doing this? For all we know there is a shared database.
  - mannykannot 7y ago
    Deterrents only deter to the extent they are known.
wnmurphy 7y ago
Heard a story a while back on NPR about how Youtube is outsourcing the trauma of moderating terrible content to the Philippines. It was a glimpse into a world I had no idea existed.
- mirimir 7y ago
  Maybe that's just a convenient source of desperately poor Catholic workers. But I wonder, maybe they could recruit people who enjoyed viewing horrible stuff. At least, if they were self-aware enough to know that the stuff they liked was horrible.
  - netsharc 7y ago
    To further that thought, they can just wire up sensors to these people's brains, and flash the images for then to see. If the sensors notice arousal: mark photo as explicit. In the dystopian future, they're going to not have enough psychopaths for this kind of work, so they're going to wire up prisoners in lawless parts of the world and modify the software to detect distress... 20 hours a day, 7 days a week.
    - fiblye 7y ago
      This sounds horrific, but realistic. We already have prisoners making license plates as slave labor. All it takes is a couple congressmen in the pockets of a big social network to make it mandatory for “rehabilitation” to moderate social networks.
    - mirimir 7y ago
      In one of Hannu Rajaniemi's dystopian futures, people sell "API access" to their brains. And ultimately, after uploading, are sold as digital slaves. Used as components of Kubernetes-like clusters.
inflatableDodo 7y ago
>"Facebook, Accenture, and WeCare may try to feign ignorance or implement common liability limiting language in their response. We hope all parties do not succumb to these common and repeated trends, and instead do what is right instead of what you are legally allowed to get away with." I admire their optimism.
fourthark 7y ago
moderators could previously count on 45 minutes every week with a counselor, or two hours a day for those viewing images of child sexual abuse, with a minimum quota of one visit per quarter. Today, moderators find themselves barred from even this scant mental health care unless “their productivity was high enough for that day,”
viburnum 7y ago
I kind of think if you need hundreds of people looking at horrible content all day, then maybe your business is messed up and shouldn’t exist. Humans can survive without Facebook. It’s not like they’re hospice workers or criminal investigators. The only goal is to make Facebook shareholders rich.
- AlexandrB 7y ago
  I knew someone who did content moderation for BlackBerry (née RIM) back when BBM was huge. Any platform that enables communication can be used for illegal and horrible content. What’s changed is the fidelity. The worst I heard of on BBM at the time was images such as illicit/child pornography and human trafficking (via text). The ability to share high res video means the horror today is more visceral for those that have to watch it. Having said that, there’s no excuse for how Facebook and their subcontractors treat the people doing this job. They can easily afford to do better.
fromthestart 7y ago
Well here's an interesting thought. Let's say you train a generative network on illegal imagery. Is the combination of network and weights illegal to posses? What about images it generates?
- bzbz 7y ago
  Depends on your nation’s gun laws. Is a gun legal? What about shooting someone?

109 points by jmsflknr 7y ago

a3n 7y ago

If it looks to an outsider like you work for company F, but you actually work for and are managed by company A, then company F has deemed you not worth the trouble of direct responsibility. They don't even see you as expendable, they just don't see you.

sciguy77 7y ago

I saw a heartbreaking YouTube mini-documentary on Facebook moderators.[1] The American workers seem to get the "lighter" flags like animal abuse, which is more than enough to cause trauma with daily exposure. I shudder to think what the offshore moderators go through when dealing with human/child abuse flags. In my view, Facebook clearly wants this to be a temporary evil, so they can use the data from human moderators as a training set for automated ML-based moderation. But I wonder how long people will have to endure this for those models to reach an acceptable efficacy. [1] https://www.youtube.com/watch?v=bDnjiNCtFk4 https://www.youtube.com/watch?v=bDnjiNCtFk4

munk-a 7y ago
> In my view, Facebook clearly wants this to be a temporary evil, so they can use the data from human moderators as a training set for automated ML-based moderation. But I wonder how long people will have to endure this for those models to reach an acceptable efficacy. Alternatively I wonder if it's worth it to put any number of human beings through this kind of suffering for something as worthless as social networking.
- malandrew 7y ago
  Let's assume social networking is worthless per your judgment: Do you honestly think this problem will go away if centralized social networks go away?
  - munk-a 7y ago
    Yes, this is a new thing AFAICT. Magazines that frequently published reader content didn't have to sift through quite as vile content when going through letters - death threats and the like have always been a thing but somehow we've lost the cost of creating vile content in this modern era. I've never heard of anyone getting a visit from the FBI over vague death threats on facebook.
    - celtain 7y ago
      This kind of content came with the internet, not social networks. Traditional forum moderators have had to deal with really vile content for a long time now, along with the operators of almost any other media sharing website. Unless you want to make it illegal for individuals to widely share content without going through an editor, this kind of problem isn't going away any time soon.
      - jessaustin 7y ago
        It really depends on how any particular site functions. Some sites function better than others. For example, "this kind of problem" doesn't seem to plague us here on HN.
    - Retric 7y ago
      You might find this interesting as it relates to a 44-month imprisonment and Facebook. https://www.forbes.com/sites/kashmirhill/2014/06/16/supreme-court-facebook-death-threats/#a0288081e617 https://www.forbes.com/sites/kashmirhill/2014/06/16/supreme-... On June 1, 2015, the Supreme Court reversed Elonis's conviction in an 8-1 decision. https://en.m.wikipedia.org/wiki/Elonis_v._United_States#Decision https://en.m.wikipedia.org/wiki/Elonis_v._United_States#Deci...
- jessaustin 7y ago
  You, too, can walk away from Omelas.
  - michaelmrose 7y ago
    It's a beautiful story. https://www.utilitarianism.com/nu/omelas.pdf https://www.utilitarianism.com/nu/omelas.pdf The Ones Who Walk Away from Omelas By Ursula LeGuin
benumbed 7y ago
So, without a doubt, there is some chilling, mind altering content out there. Videos providing evidence of criminal acts of violence that inspire a deep disturbance of emotion lasting days, even among hardened viewers. But the truth is that it's all a pathetic joke. You can't police telephone calls, and the conference lines have to police themselves. This is the price of decentralization and the democratization of content creation. There can no longer persist any such dictatorship of standards and practices. There's only one maxim that applies: Don't talk to strangers. End of story. There are certain manifolds and convolutions that are unhygienic, and if you have a problem with that, don't build them and don't explore them. But, a network is a network is a network. It's a system of passages. Hallways with doors. In some respects, these policing activities and attempts to moderate a mob represent an obvious over-extension beyond the normal scope of operations. It's silly to even think it's possible. But hey, maybe Facebook is catching quite the government hand out, just to experiment with some "DARPA hard" problem solving. Personally, I find it laughable that anyone imagines this problem to be even remotely generalizable enough to fit a statistical model that the current state of "AI" buzzword juicing vaporware purveyors and hucksters might jazz hands a pitch deck with. Imagine: AI staring at clouds, noticing familiar shapes among the drifting cotton balls. How productive. You could waste an entire nuclear power plant's total output on such a thing, and get nothing for it. Attack the right problem at the root. These violent, malignant people torturing the animals and murdering on livestreams. Some will get through, and the internet will simply reflect reality. But make the world a better place. Don't spin your wheels papering over the blood stains. The reality is that these images videos are going to be shared and shared again. The cat's out of the bag. Accept the bald face of the human condition, warts and all.
tjpnz 7y ago
I'm guessing that most of the stuff heard about Facebook and ML is basically hype. Even after Christchurch the best they could do to counteract the spread was content fingerprinting. Human moderators are likely their longterm solution.

itronitron 7y ago

Hopefully a small group of licensed psychologists and therapists can set up a small office across from the FB office in Austin and start serving the moderators privately.

staticautomatic 7y ago
Yeah I'm sure the therapy will be covered by the amazing mental health benefits these moderators receive through the no-cost health insurance provided by Accenture. If not, they'll have no trouble at all paying out of pocket with that cushy $14.50/hr they're making!
- perl4ever 7y ago
  There is something (in the US) askew about mental health coverage. I think that regulations were passed to try to get mental health conditions treated more like others, but it seems like there is a shortage of counselors, social workers, and doctors, much more than other kinds of medical professionals. I have this feeling that there's something about the incentive structure under the ACA that is choking off the availability, but I'm not sure what exactly it is. I'm very curious about the economics of running a mental health practice these days. If you can't pay $70-140/hr to talk to a social worker every week or two, they may well offer you the option of online chat for $150/month. But I feel like there is a mystery there, because why should it cost that much? And yet, the people who charge that much seem desperately overworked and barely functional. Where does the money go? Not to doctors, because everything is done by non-MDs these days. Not to insurance, because deductibles are in the thousands.
  - tru3_power 7y ago
    I’ve been wondering this too- especially since I’ve always heard that fields like psychology are “over saturated” and that there are tons of people with psychology degrees that can’t find jobs. Seems like if this is true there is a change for a win/win here (people get jobs and cheaper mental health care)

hos234 7y ago

Can we please start sending a few Facebook execs to jail? No fines. No apologies. Just jail time. They can get out of jail time if they send their kids to work in these centers.

mschuster91 7y ago

For what its worth, I believe that the most egregious offenders (think violence, murder, rape, child porn) should be directly referred to the police with all data Facebook has on them. This at least should slow down the torrent of shit that has to be moderated over the time...

wincy 7y ago
I mean in the US pictures of murder aren’t illegal.
- mschuster91 7y ago
  Showing them to children is, at least in Germany, I do hope there are similar public decency laws in the US. In any case, people sharing murders who are not journalists or citizens trying to get attention for an issue (think gang violence or political oppression) should be investigated for their sanity.
jeremyjh 7y ago
Who is to say they are not doing this? For all we know there is a shared database.
- mannykannot 7y ago
  Deterrents only deter to the extent they are known.

wnmurphy 7y ago

Heard a story a while back on NPR about how Youtube is outsourcing the trauma of moderating terrible content to the Philippines. It was a glimpse into a world I had no idea existed.

mirimir 7y ago
Maybe that's just a convenient source of desperately poor Catholic workers. But I wonder, maybe they could recruit people who enjoyed viewing horrible stuff. At least, if they were self-aware enough to know that the stuff they liked was horrible.
- netsharc 7y ago
  To further that thought, they can just wire up sensors to these people's brains, and flash the images for then to see. If the sensors notice arousal: mark photo as explicit. In the dystopian future, they're going to not have enough psychopaths for this kind of work, so they're going to wire up prisoners in lawless parts of the world and modify the software to detect distress... 20 hours a day, 7 days a week.
  - fiblye 7y ago
    This sounds horrific, but realistic. We already have prisoners making license plates as slave labor. All it takes is a couple congressmen in the pockets of a big social network to make it mandatory for “rehabilitation” to moderate social networks.
  - mirimir 7y ago
    In one of Hannu Rajaniemi's dystopian futures, people sell "API access" to their brains. And ultimately, after uploading, are sold as digital slaves. Used as components of Kubernetes-like clusters.

inflatableDodo 7y ago

>"Facebook, Accenture, and WeCare may try to feign ignorance or implement common liability limiting language in their response. We hope all parties do not succumb to these common and repeated trends, and instead do what is right instead of what you are legally allowed to get away with." I admire their optimism.

fourthark 7y ago

moderators could previously count on 45 minutes every week with a counselor, or two hours a day for those viewing images of child sexual abuse, with a minimum quota of one visit per quarter. Today, moderators find themselves barred from even this scant mental health care unless “their productivity was high enough for that day,”

viburnum 7y ago

I kind of think if you need hundreds of people looking at horrible content all day, then maybe your business is messed up and shouldn’t exist. Humans can survive without Facebook. It’s not like they’re hospice workers or criminal investigators. The only goal is to make Facebook shareholders rich.

AlexandrB 7y ago
I knew someone who did content moderation for BlackBerry (née RIM) back when BBM was huge. Any platform that enables communication can be used for illegal and horrible content. What’s changed is the fidelity. The worst I heard of on BBM at the time was images such as illicit/child pornography and human trafficking (via text). The ability to share high res video means the horror today is more visceral for those that have to watch it. Having said that, there’s no excuse for how Facebook and their subcontractors treat the people doing this job. They can easily afford to do better.

fromthestart 7y ago

Well here's an interesting thought. Let's say you train a generative network on illegal imagery. Is the combination of network and weights illegal to posses? What about images it generates?

bzbz 7y ago
Depends on your nation’s gun laws. Is a gun legal? What about shooting someone?