22 ms·

Facebook moderators in America

304 points by ajay-d 8y ago

duxup 8y ago
I remember when "user generated content" was a common phrase. It was great, users generate the content for the site and you just have to provide a platform for it. It seemed like a positive thing. Not it seems clear that if you open yourself to host something for your users you also need to be prepared to accept that you're going to be hosting the worst of humanity, exposing others to it, including people who work for you. It seems like such an extreme contrast from what it seemed like user content could be and what it is.
- Bahamut 8y ago
  This isn't particularly surprising to anyone who has been online for a while - sometimes you'd see gruesome stuff even on forums, and then you also had the culture on sites like 4chan. Moderation isn't something that one could really escape from even about 20 years ago, just now we're more cognizant of the effects of poor moderation and users are more aware of how people behave when moderators/admins are pushed to the boundaries of the set rules.
  - duxup 8y ago
    It's not a surprise now. It is a big contrast as far as I think what folks thought in the early days of user content.
  - tenpies 8y ago
    I think the big difference is how oversensitive society is now. Before, your moderators just had to deal with outright illegal content, maybe a flame war or two (or just move the discussion to the flame war section). If you ran into anything as a user, it was understood that you should report it and that the moderators would get to it with varying degrees of speed. There was no expectation of a perfectly safe bubble. Now, if even the most milquetoast comment which could possible be offensive if I assume the absolute worst about the poster, isn't moderated post-hate, you'll be reported to the media who will almost certainly archive it and use it to fill the next 1 - 24 hours of their outrage media cycle. I honestly see a future where people file lawsuits against web sites for not moderating enough, a failure to keep safe or sort of negligence. "Your web site gave me PTSD or enabled cyber-bulling". What's worst, they'll win.
    - komali2 8y ago
      I think this argument has been being made by people for two thousand years (this generation is too soft! Uphill both ways in the snow!) but I've yet to see a causation demonstrated.
      - mmsimanga 8y ago
        Just watch an old movie from about the 90s and you will be surprised how politically incorrect some of those movies were. They are just a reflection of what was acceptable in those days.
        Pfhreak 8y ago
        And in 20 years we'll watch media from this decade and be shocked at what we put in our media. Society advances, culture changes. Viewing the past through the lens of the present is always going to be a different exercise than viewing the present through the lens of the present.
        reaperducer 8y ago
        Society advances, culture changes. I agree with the second part of that. Not necessarily the first. Viewing the past through the lens of the present is always going to be a different exercise than viewing the present through the lens of the present. Unfortunately, we live in a culture of intolerance. A few months ago there was a race car driver who lost his sponsorship because of something his father said before the driver was even born.
        komali2 8y ago
        >we live in a culture of intolerance This is quite the theory - do you have more backing other than the single instance you've just quoted? And what is "intolerance" and why is this "bad"?
        8y ago
  - 013a 8y ago
    The the scale of the platform is the differentiator. If you're moderating a few hundred or thousand people on a forum, especially a site with a constrained vision and purpose, you're simply not encountering content and behavior like this on a regular basis, if ever. Now, look at facebook. Billions of users. No constrained vision or purpose to being there; if you're human, you're a customer, and human behavior can be pretty gross at the fringes. And maybe most importantly; they can't just Ban whatever people or content they want, due to both a profit motive and public outrage due to their preeminent position. This isn't a problem that has already existed, because the problem is with the Scale, not with the Behavior.
    - deleted 8y ago
      [deleted]
    - jachee 8y ago
      > if you're human, you're a customer No. If you're human, you're a product. Can't have the product spoiled, now, can we?
  - spurgu 8y ago
    Yeah the problem today compared to BBS's/forums 10-20 years ago is that there are "smart" algorithms that will escalate the spread of fake news and shit content, compared to having everything equally exposed to everyone.
- deleted 8y ago
  [deleted]
- rootusrootus 8y ago
  This surprises nobody who has any experience with hosting. Even as far back as when I ran my own BBS, it was immediately clear that some people only wanted to leverage the content hosting for nasty things. If you were lucky, it was just perfectly legal pornography.
mruts 8y ago
Maybe Facebook just let all videos be posted, except when it’s illegal (the video, not the act). Pretty much the only thing that would fall in that category would be child porn. I’m not sure I really understand why big companies feel it’s their job to censor the world. Just let the floodgates open: allow violence, allow porn, allow whatever else. Who cares? And maybe this might actually good for people to see. I feel like we are leaving in this outrage culture that wants to censor everything. It’s not healthy for individuals or society.
- duxup 8y ago
  Including allowing the very first thing mentioned in the article?
  - ahakki 8y ago
    Depends if very first thing mentioned, a graphic video of a murder, is illegal in the relevant jurisdiction. The legal status of snuff films is pretty complicated.
  - thepangolino 8y ago
    Well, why not? It won't put it under your nose unless you actively search for it.
    - duxup 8y ago
      "It won't put it under your nose unless you actively search for it." That's not how social networks work.
  - cf141q5325 8y ago
    The alternative is paying people to moderate it all day every day. With the mental damage this causes to people who are desperate for a job. Just blocking it yourself seems like a better solution. You dont have to be fair in what you block yourself and (surprise surprise) you dont need to watch it in the first place.
deleted 8y ago
[deleted]
ohiovr 8y ago
This job sounds more horrific than a crime scene cleaner. 4500 foot soldiers to clean up a billion users. Murder rape assault fraud. Burn it all with brimstone.
- canttestthis 8y ago
  Are you saying that there aren't enough people cleaning up or that cleaning up should be done differently?
  - ohiovr 8y ago
    I honestly don’t know how moderators work and i actually don’t use facebook. But 1,750,000,000 / 4500 is about around 380,000 users per moderator. I think they need 5 armys of moderators instead of just a brigade trying to find the worst. They need to step in with private messages or public posts to keep the discussion from going rancid fast. I see it as being logisticaly impossible unfortunately but to keep discussions from going to that depth of evil they need to step up moderatoon pressure a couple orders of magnitude. I’ve said things here that moderators told me it isn’t allowed (actually said things here that were hurtful and unnecessary blanket statements) and I gripe to myself and limp back. If this pressure isn’t there yc would turn into kuro5hin in short order. I’ve seen discussion groups fall into the toxic slime pit more than a few times never to return. I think at least facebook groups can self moderate i have been told. I really doubt good behavior can be coded into a forum. Now that pit is billions of people not thousands. I have little hope it’s going to ever get better. Is this the destiny of all internet discussions? I worry about this a lot.
WisNorCan 8y ago
I would be impressed if Zuck and Sheryl decided to spend a day per year doing this job. I suspect that as a billionaire and the CXO of one of the largest companies of the world, you can get disconnected from the details. It would be valuable for them to see a different reality of their platform.
- Shish2k 8y ago
  What do you think / hope that the outcome of this experience would be?
  - deleted 8y ago
    [deleted]
  - untog 8y ago
    That they would gain knowledge of what the job is like, and so realise that their employees doing the work need a lot more support than they currently have.
    - wink 8y ago
      If it were Facebook employees that might even be the case. As the article states it's conveniently outsourced so they can just label it as a dollar expense without a guilty conscience.
    - intended 8y ago
      They probably know. And the secret is that there is no current solution. There is no automation to stop the sheer amount of dumb, sick, sad, horrifying, malignant and monstrous stuff being put up every minute from around the world. Frankly it could probably break social media the same way cancer could kill the tobacco industry. And this is simple moderation - ignoring newsworthiness, censorship, speech, propaganda and other issues this throws up. The best they probably hope is that the users forget about it. I mean seriously - what are publicly owned firms supposed to do?
      - 8y ago
endofcapital 8y ago
If I was running a marketing team or PR firm working on this problem I'd keep pushing the nonsense of AI magically solving everything someday. The actual nuts and bolts of content is pretty horrific, and they are obviously going to want to hide this reality. At least pay these people more than a dishwasher, please.
dsfyu404ed 8y ago
Meh. Sounds like a run of the mill shitty job. You can destroy your body laying bricks and unloading trucks or you can destroy your mental health doing something like this. Of course they're not paid well enough to compensate for the long term damage. Nobody ever is. There's all sorts of shit you might have to put up with in a shitty job, the work, the environment the customers. This job is shitty but some people are bothered less by this particular kind of shitty than they are by by angry customers yelling at you or laying bricks in the Arizona weather. Sure you might have to see some really screwed up shit among the sea of nip-slips and racial slurs but you get to do it in an air conditioned office without killing your body like the guys in the Amazon warehouse. Some people prefer that. Different people have different degrees to which they'll tolerate the various ways shitty jobs are shitty. At least one person the author interviewed said "well, it sucks less than Walmart". Back at the point in my life when I was doing shitty jobs I would have preferred brick laying or custodial work but I wouldn't have turned down this job if I needed it. Of course it's terrible for you and turnover is high but all the equally shitty jobs are this way. There really is no winning in the "minimum wage or thereabouts" income bracket. Edit: Since apparently this opinion is unpopular can anyone tell me why this particular implementation of shitty job is worse than all the other implementations?
- untog 8y ago
  Disregarding for a second that you're dismissing this entirely without having done so much of one minute of the job yourself... A lot of manual labour jobs are protected by unions, which establish safety standards, limits to shift length and healthcare to help employees when they need it. Sounds like these employees could do with the same.
  - dsfyu404ed 8y ago
    >A lot of manual labour jobs are protected by unions, which establish safety standards, limits to shift length and healthcare to help employees when they need it. Sounds like these employees could do with the same. I'm not comparing to those jobs. The union dishwasher in some public university cafeteria gets time off, healthcare, etc, etc. that makes his or her job much less shitty than the infinite-term temp that works beside them. I'm comparing to the temp. You can make a job better/worse by adding/subtracting pay, benefits or working conditions. If I could get tech pay and benefits for construction work I'd be doing construction. If we're gonna compare this job to others then we should compare to other jobs that are "equally shitty" not jobs that are similar but actually compensated more highly if you roll the benefits into total comp.
TorKlingberg 8y ago
This part is scary: The moderators told me it’s a place where the conspiracy videos and memes that they see each day gradually lead them to embrace fringe views. One auditor walks the floor promoting the idea that the Earth is flat. A former employee told me he has begun to question certain aspects of the Holocaust. Another former employee, who told me he has mapped every escape route out of his house and sleeps with a gun at his side, said: “I no longer believe 9/11 was a terrorist attack.” I have a hope that people will eventually develop immunity to conspiracy theories after falling for enough of them and eventually seeing them disproved. But with things like pizzagate/Qanon it seems some people just keep going deeper.
- monetus 8y ago
  When it comes to neuroplasticity and change, all it takes is time. If you spend more in the shoes of those flagged for moderation than their opposites, maybe its inevitable that you attain a form of Stockholm syndrome.
  - megous 8y ago
    It's a bit cultlike. Also add isolation from family/friends, and tight control of what you can and can't do, and having no trusted person (therapist there works for the comapny). It doesn't surprise me that it can lead to similar outcomes, as being in a cult.
- rgoulter 8y ago
  To extend the metaphor "develop immunity" a bit, I'd be curious about the mappings to: - vaccines, which build immunity by providing an inert or weak threat that the body builds resistance to. - herd immunity: restricts the spread of diseases if a sufficiently high proportion of the population is immune to the disease.
- Diederich 8y ago
  > seeing them disproved Well, many of these things essentially have a starting state as 'disproved', ya?
b1r6 8y ago
I would never be able to to do that job. Not because of the exposure to "bad things" like the article pushes, but because I've always felt totally fine (even bored) looking at that stuff, and I don't agree with censoring it. It's just "meh", another day of reality on Earth; the Internet is just a mirror held up next to it. Will we ever collectively realize this? It seems younger crowds are more normalized to this stuff because they grew up on the Internet. But add in the hypersensitivity in today's public social network sphere, and we're all freaking out over anything even potentially flammable.
- pjc50 8y ago
  I think you're on the far side of the "desensitization" process. It's not entirely normal to have a flat emotional response to everything, including images of human suffering.
  - b1r6 8y ago
    Certainly a good point. I'd add that when it comes to suffering, I certainly feel discontent with that happening: In the article, the example of a stabbing video is given. I would strongly feel the need for retribution or justice for that victim. Same for all content that shows someone being hurt. I guess what I mean is that it's wrong to try and purge all this stuff as if it just doesn't exist. This is real content, real people, being hurt in reality. I'd feel disgusting trying to censor it, not disgusted by the content.
    - opT52ifk 8y ago
      Is not publicizing (in a system like Facebook, designed to proactively promote and spread attention-grabbing content to any audience it can) the same thing as censorship? In other words - I sympathize with the hard-ACLU/EFF stance on free speech. But there's a difference between government censorship and societal moderation - the latter has always existed, and the scale and automation of modern platforms in publicizing content that normally wouldn't spread so far is what's new. Should people be aware of bad things that happen in the world? Absolutely. Is broadening the audience for disturbing videos the right way to raise awareness? Maybe not. And if folks really feel such content needs to be published, they still have more options and reach than they did in years past even if "mainstream" places like Facebook moderate them. Granted this last point is getting a bit trickier, as people go after registrars and web hosts themselves for political reasons occasionally (i.e. if this was Cloudflare we were talking about I'd be in full agreement with you - then again Cloudflare doesn't run a recommendation service that automatically causes new unexpected content to appear in front of billions of people).
aeling 8y ago
If I think this is unacceptable[1] and I'm skeptical of the nebulous "AI will do this job in the future" claims, are there conclusions to be drawn other than "UGC isn't sustainable on a global, public platform"? That is, are there serious alternative options, or anybody working on ideas in this space? I think it's readily apparent that "just show everything" doesn't work if you want to attract a mainstream audience, but I'm reluctant to just give up on the global public platform that FB was originally idealized as. [1] I think I'd still find it unacceptable if the moderators were being paid 6 figures, had extensive 1:1 counseling, or any other perks - selling mental health for money is something I'm happy saying a utopian society wouldn't include.
- 0x54D5 8y ago
  The public platform needs to be decentralized. We need a social protocol where the data lives in the protocol and not in some corporate server where it is subject to their whims. We currently live in a dystopia where your Twitter or Facebook could be banned at their whim leaving you a digital outcast.
  - gipp 8y ago
    That's pretty much the opposite problem of what the GP is asking about.
  - ghaff 8y ago
    So utopia is a completely decentralized uncensorable network where basically anything goes? Not wholly achievable probably because of DNS etc. but you could come close. Not an unsupportable position but you probably amp up the underlying issues Facebook is trying to address by 10x.
  - BinaryIdiot 8y ago
    I’m not sure I follow. Almost any time there is an issue with a large social network folks on HN say decentralization needs to happen to fix it. But how do you “fix it” while still having the same or better user experience? It’s a very difficult problem to solve and I don’t think saying “make it decentralized! Make it live on a protocol!” is useful without the extensive “how” that everyone seems to ignore.
_bxg1 8y ago
I get a certain amount of secrecy for their own protection, but this: > They are pressured not to discuss the emotional toll that their job takes on them, even with loved ones, leading to increased feelings of isolation and anxiety. Just doesn't make any sense. Not only does it not protect the moderators, I can't even figure out what the cynical corporate interest would be.
- duado 8y ago
  The cynical corporate interest is that they don’t really want more people knowing about this whole underworld.
  - Tsubasachan 8y ago
    I think most people prefer not to think too much about this underworld. Like when somebody throws themselves in front of a train one of the first things the police does is put up barriers. And how do you even talk with your loved ones about this kind of work?
- em3rgent0rdr 8y ago
  Indeed, I'd think the cynical corporate interest would be to watch out for well-being of their workers and to prevent PR backlash from workers getting mental issues.
  - maxxxxx 8y ago
    "watch out for well-being of their workers" Usually the first corporate response is to sweep things under the rug and hope they will go away. Only if that doesn't work they will take real steps.
  - egypturnash 8y ago
    That's the behavior of an enlightened psychopath who has realized how useful good will is. Facebook-the-company is an unenlightened psychopath who just wants money and power and doesn't care what anyone thinks about it.
- 8y ago
statictype 8y ago
Interesting that the moderators were provided by Cognizant - essentially a software outsourcing company.
product50 8y ago
At this point Casey Newton has so much angst against Facebook, that even if you show him (hypothetically) that everyone at FB is earning $100k and enjoying their jobs, he will still point out that the plumber who came into fix the bathroom the other day is being paid $40/hr and that is a travesty. /rantagainstbiasness
_bxg1 8y ago
The internet is one great big libertarian experiment. It illustrates all of the creativity, and all of the depravity, that result when you just put millions of people together and tell them to do whatever they want without consequences.
hnaccy 8y ago
If seeing it makes you cry it seems like you should be looking for a new job. There is a portion of population that would be generally unfazed by the content, it would be best for everyone if they worked these jobs.
- untog 8y ago
  You really think the people being paid $28k a year to watch videos of literal murder haven't thought that maybe they'd rather have a different job?
  - Nadya 8y ago
    Tons of people watch these kinds of videos by choice over on /r/watchpeopledie (Warning: Very NSFW Subreddit if the name of it doesn't make that immediately obvious. Contains many videos of accidental deaths and murders, including those of children.) I'm sure some of them are jobless and would love to be paid $28k a year to do something they already do as a bit of a hobby.
    - untog 8y ago
      How many people who browse /r/watchpeopledie browse the sub for a length of time equivalent to an entire work day, Monday to Friday? And how many of those users, who browse videos of people dying all day every day, for pleasure, are qualified to judge what is offensive and what is not? These people should be referred to mental health professionals, not assess what is suitable for the world. You're also entirely missing my original point. The people currently doing the job don't like it. They don't want to be doing it. And yet they are. Why do you suppose that is? Perhaps... because there are no other jobs available? Saying "you should be looking for a new job" to these people is like telling a homeless man he really ought to be looking for a rental property in his budget range.
    - bendotero 8y ago
      That's sick. What you just described is someone who is unhealthy, not a person who needs a full time job to further indulge in something deviant.
crankylinuxuser 8y ago
Yep. And this is what you get when collectively "We" demand moderation. Its the other side of the fence... Usenet was the unmoderated wasteland, and Facebook is a the moderated hell-landscape of "ok not ok".
- intopieces 8y ago
  Usenet still exists and is moderated in a similar way against exploitative content in partnership with The Internet Watch Foundation (https://www.iwf.org.uk https://www.iwf.org.uk), depending on your provider. Giganews, the largest Usenet provider and owner of other affiliated providers, is a member of the IWF: https://www.iwf.org.uk/member/giganews-inc https://www.iwf.org.uk/member/giganews-inc.
  - crankylinuxuser 8y ago
    But due to its decentralized system, subscribing to a block list from IWF is just that. Its the equivalent of a spamblock list.
    - intopieces 8y ago
      Not really. Usenet providers voluntarily subscribe to the list for you, leaving you without the choice.
Mr_Shiba 8y ago
This is a great opportunity to encourage everyone to delete your facebook & Instagram.
- tjkrusinski 8y ago
  Also stop using Google, Twitter, YouTube, etc etc because they all work similarly.
  - Mr_Shiba 8y ago
    Agreed, duckduckgo ftw.
jrochkind1 8y ago
> Every moderator I spoke with took great pride in their work, and talked about the job with profound seriousness. They wished only that Facebook employees would think of them as peers, and to treat them with something resembling equality. > “If we weren’t there doing that job, Facebook would be so ugly,” Li says. “We’re seeing all that stuff on their behalf. And hell yeah, we make some wrong calls. But people don’t know that there’s actually human beings behind those seats.” Some respect and acknowledgement, and more than $15/hour, seems like it would be a good start.
- jrochkind1 8y ago
  Also: High-quality no-copay mental health services, that are also contractually guaranteed for, say, a year after severance (for any termination reason). It also occurs to me that even if this work arguably needs to be done, it doesn't need to be done by anyone 8 hours/day or 40/week. (Let alone in conditions where your job security depends on maximizing your "efficiency"). Anyone needing to do this should do it as a part of a job with a balanced set of work duties, with this being something they do only some of their work time. These things would hurt Facebook's bottom line of course. These people are being chewed up and spit out for Facebook's profit and reputation.
  - zelon88 8y ago
    I agree that this could be a part of a bigger position instead of being the bread and butter, but we are talking about sub $30k/year jobs that nobody wants to do. We could just as easily make the argument that it's not safe for garbage men to hang off the back of the garbage truck, but at the end of the day a) nobody cares and b) I've got my own job that nobody wants to do that I'd love for someone to come along and make less shitty.
    - jrochkind1 8y ago
      > but we are talking about sub $30k/year jobs that nobody wants to do. Yes, that's the problem. There is no law of nature that says we need to pay people with the jobs that are worst for their health the worst and give them no respect or acknowledgement. If there are other jobs that is true for, yes, those too. I wouldn't tell you what to care about, but I would say that we all would benefit by taking care of those of us whose jobs harm them. Your job should be less shitty too, it doesn't take away from that to point out other people who's jobs are harming them and aren't getting what they need. It is reasonable to begin focus on the jobs that are the _worst_, mentally or physically. Maybe yours is one of those too. It's also reasonable in the HN venue to focus on the jobs that are the worst that make possible the economy that many of us reading this benefit from by being paid a lot more than $15/hour to contribute to, and probably aren't as harmful to our health as those for many of us. If your situation is especially dire, then, yes, you too.
scoutt 8y ago
I always wondered why they don't have a system where the videos or images are shown first filtered, like blurred, and then at the discretion of the operator, they can take "several levels of blurring" off until they see the original image. That way, for some images/videos/ you can tell if it's violent/inappropriate right away, and the operator may see "less" of whatever is disturbing for them. I've also wondered how they deal with long videos that may seem correct for the first minutes, but the real inappropriate content is interlaced for a few frames in-between or later in the video.
- perbu 8y ago
  It's low-value work. Facebook doesn't value these people enough to employ them so they likely don't have access to properly engineered tools.
  - dymk 8y ago
    What makes you think that? How can you say that with such confidence? Are you letting your pre-existing negative feelings color your assumptions about what people actually think at the company? I used to work there, and I built tools for these workers. Every one of the other engineers that I knew had nothing but respect for them, and everybody recognized that it was important work.
    - agentdrtran 8y ago
      So important their bathroom breaks are timed, they get inadequate counseling, and they get paid $28k a year!
      - tjkrusinski 8y ago
        You have to remember that those are policies put in place by the contracting company, not Facebook or the engineers that make the tools that these people use.
        8y ago
aboutruby 8y ago
For those used to shock sites from 10-20 years ago, this is very different, I'm not going to link to anything but the disturbing content is on a few whole new levels. I can't even imagine doing this job. This should be 100% automated with only 0.01% error rate for the first-pass filter (e.g. before it reaches humans). They are willing to spare their users the horrors but not their own employees/contractors...
- intended 8y ago
  Facebook tried the automated system by using employees to train the machine. It failed. They returned to non Facebook employed humans doing the moderation
  - rock_hard 8y ago
    I read somewhere that it’s actually a hybrid approach! For some things liken nudity ML based solution work fine. But other things such as hate speech are more nuanced and require human oversight. That said ML is used everywhere possible to flag content to moderators. For some categories the work of human moderators is used to train the ML models. Facebook and other companies like YouTube are well insentivized to do the right thing here and automate as much as possible for all the reasons outlined in OP
  - fencepost 8y ago
    One thing I hope they're doing is advanced detection of duplicate inappropriate content - e.g. by splitting a confirmed properly blocked video into frames and identifying matching videos based on frames matching (or even key parts of frames). You'd still have to investigate/review partial matches, but something like that could cut down a lot on duplicate effort and could auto-identify a lot of things that would need manual review.
- agentdrtran 8y ago
  Machines cannot do this job yet.
- ddalex 8y ago
  I developed the first version of the screening system for a corporation about 10 years ago, when UGC was still in its infancy. We had the help of a Microsoft tool that would match the visual fingerprint of a photo against a known-images database that prevented human operators from re-seeing an already tagged image in a different cloud, hoping to reduce the psychological scarring of the human workers that screened the content. We also evaluated automatic classification of images, but it's very hard to detect soft core porn from shock images even with today's algos, never mind 10 years ago. One advice from a legal counsel that I received at that time still stands in my memory; she said: "Never look at the images, nevermind how curious you are. It's not just illegal and can land you in prison, but it's permanent. You cannot un-see something you've seen."
intended 8y ago
This sounds about right >dark humor and camaraderie Yes. No one can know what happens if you start moderating, except unusually inquisitive users or other mods. > believing conspiracy theories Not all ideas are equal - and not all brains/people are equipped to deal with them equally. Your mod team will soon get polarized or even subverted if faced with the kind of crazy work these facebooks mods have to do. If you are actual community mod, life is slightly better, since there’s more context.
rhema 8y ago
Is it just me, or do should we expect that some small population will be naturally equipped to deal with this job? We don't expect everyone to want to be a soldier or medical doctor. These people deal with situations that would make most people throw up. If you can't handle an hour of browsing 4chan, maybe you should know that content moderation is not a good fit for you.
- intended 8y ago
  The pay is poor, and the population willign to do this may not be in the same region as the jobs, may have better jobs and so on. The workload is massive. There’s not enough people, and the firms don’t want to pay as much as they need to.
- sharkweek 8y ago
  I'm not sure if that solves the core issue of long term mental effects, even on the "hardest" people. Stepdad is a retired firefighter, has kind of that old-timey tough guy thing going on (although deep down he's a real softy) - he rarely likes talking about the hardest parts of the job, but the few times it has come up he mentions seeing some things that haunt him. I remember one time him mentioning seeing something so grotesque when he pulled up to the scene that he started laughing at the absurdity. Sort of in a way that he could do nothing but throw his hands up and say "well fuck this." He jokes about some of that stuff, but I can sense pain beneath the humor.
  - darkpuma 8y ago
    Consider that there are people who've only seen a small fraction of what your stepfather has, but can't handle it nearly as well.
- tomcam 8y ago
  It seems to me (and I’m serious about this) a classic sociopath would in fact be willing to do this kind of thing for money if I understand correctly, with no ill effects. They apparently have conscience and are something like 1% of the population. But I think they tend to be smart, and can probably get higher-paying jobs.
intended 8y ago
Remember that Reddit quarantines /watchpeopledie These mods may randomly see gang videos (go look for them if you want to see how tough you are) on any day. Viewing them would make many people I know retch, worse than if they saw a gory horror movie. And then some ideas, if presented without an antidote (creationism, flat earth, conspiracies) will subvert the people watching it. And then comes the worst part - the damage to society itself. If violent images are removed, depending on what people in Facebook and their friends, consider safe- then they are also deciding what is newsworthy. And Facebook mods are not in a position to decide if an image of burning child, will stop a war or awaken the conscience of a people. All they know is that it breaks “rule 4- death: graphic wounding, no gore.”
- casefields 8y ago
  The only one that got to me was Funkytown. I've got to imagine if they increased pay and hunted through those users, they could find people that can withstand sifting through this stuff.
wespiser_2018 8y ago
I wonder how far we are from automating this with image recognition. The interesting thing is policy aspect to the learning problem, does an post/image/video violate a collective set of rules? They are certainly building a huge enough labelled dataset of images, decisions, and policies. I wouldn't be surprised if this went automated in 5-10 years!
throwaway-1283 8y ago
I feel like selling moderation tools would be a huge biz for any co with UGC...
- tjkrusinski 8y ago
  Yes, it already is.
walrus01 8y ago
A huge amount of content moderation has been offshored to the Philippines, where salaries are considerably less than $15/hour. Combination of access to good internet connections, modern office space, an English speaking workforce, and low salary levels. I have not seen this documentary, but compared to the verge article, I think a major difference is that the filmmakers did not get cooperation from Facebook or the third-party content moderation company. In fact their photos were posted and employees were told to stay away from them. https://www.vice.com/en_us/article/ywe7gb/the-companies-cleaning-the-deepest-darkest-parts-of-social-media https://www.vice.com/en_us/article/ywe7gb/the-companies-clea...
jrochkind1 8y ago
See also this upcoming book by Dr. Sarah T. Roberts, the product of several years of research. https://www.amazon.com/Behind-Screen-Content-Moderation-Shadows/dp/0300235887 https://www.amazon.com/Behind-Screen-Content-Moderation-Shad... Here's a presentation Dr. Roberts' gave at conference last week on the topic too: https://www.youtube.com/watch?v=7mdMtukvtxc#t=15m50s https://www.youtube.com/watch?v=7mdMtukvtxc#t=15m50s (Until recently at least, most of this content moderation work was done by contractors overseas, largely in the Phillipines. Can you imagine what the working conditions are like there?)
- 2sk21 8y ago
  The presentation you linked to by Sarah Roberts was very interesting to say the least. Highly recommended. She adds a lot of information about the condition of content moderators in other countries.
iambateman 8y ago
Question: is there a legal prerogative for Facebook to proactively moderate content in the US? Don't get me wrong: the article is incredibly hard to read. When humans become untethered from social mores, they are capable of shocking and evil behavior. But...can't people mostly police this themselves? If I had a "friend" posting video of a dog being stabbed, I wouldn't be their friend for much longer. It seems that the result would be relatively small cesspools of highly antisocial behavior. Cesspools which will continue to exist regardless of any moderating effort. Alternative to moderation: introduce "public content ratings" for the content users post publicly. Machines automatically assess content and rate the user based on the maturity level of their public discussion. I would be "teen-mature" because I sometimes say "shit" online and perhaps another user would be "highly mature - disturbing violence and sexual content". Their ratings would weight groups. Then, it's up to me to decide who I associate with online. Caveat: nothing is a panacea. We're all trying to figure out how to handle the fact that we gave every maniac a megaphone to the masses.
- Despegar 8y ago
  If HN can't do without moderators why do you think Facebook could?
  - iambateman 8y ago
    I don't think Facebook can do without mods entirely. Maybe they don't have enough mods or strict enough rules. I honestly don't know. But Facebook has a more user-curated thread, whereas HN is public. In a sense, I am my own mod on FB. FWIW...I definitely don't use FB on a regular basis. Haven't since Oct 2017.
- rchaud 8y ago
  > If I had a "friend" posting video of a dog being stabbed, I wouldn't be their friend for much longer. It's less about it being for you, and more about that it would be hard to sell ad space when it sits next to content of that type. Your friend might not post those materials, but what about Pages and Groups, which is where I'd say most FB users spend their time these days.
- 8y ago
NoblePublius 8y ago
The only thing remarkable here is that these people are not hired by Facebook directly. Someone needs to ask Zuck why he thinks these hires aren’t worthy of catered lunch and @fb.com emails.
- mmagin 8y ago
  Then there would be the risk that the rest of fb's employees might be exposed to the reality of this.
daenz 8y ago
>If we weren’t there doing that job, Facebook would be so ugly It would be 4chan, minus the anonymity.
- faissaloo 8y ago
  That would be many times worse than all the chans combined, imagine a chan exclusively used by middle aged mothers where anyone you talk to is doxable at the click of a button.
lazugod 8y ago
> When I ask about the risks of contractors developing PTSD, a counselor I’ll call Logan tells me about a different psychological phenomenon: “post-traumatic growth,” an effect whereby some trauma victims emerge from the experience feeling stronger than before. Wow. Wow. Is this really a thing, excusing forced trauma by claiming it makes people better?
- darkpuma 8y ago
  You seem outraged at the suggestion that this could happen. Post-traumatic growth is an actual phenomena: https://en.wikipedia.org/wiki/Posttraumatic_growth https://en.wikipedia.org/wiki/Posttraumatic_growth Of course that's no excuse to downplay the occupational hazard of PTSD. But it's a real thing; facebook didn't make it up.
- dsfyu404ed 8y ago
  People respond in varying manners to traumatic stress. Post traumatic growth is simply the other extreme. For example, some people take traumatic stress at really poorly (PTSD). Some people develop a world outlook to the tune of "well no matter what happens life is looking up from here" (post traumatic growth). The counselor is looking at the glass half full. You kind of need to be a glass half full person to persist in that line of work.
  - watwut 8y ago
    PTSD is not an issue of attitude while your description of post traumatic growth is nothing but attitude. The two are incomparable. The actual benefit of what you describe as growth is quite low to offset clinical PTSD which affects life quite a lot.
    - darkpuma 8y ago
      PTSD isn't universal. Not everybody that experiences trauma will develop PTSD. So it's not a matter of "you have PTSD but at least your outlook on life has improved."* Rather, sometimes with some people, trauma won't induce PTSD but will cause post-traumatic growth. Research suggests there is a correlation between PTSD and post-traumatic growth, but it's not a hard and fast rule. Furthermore there are many factors in play. The nature of the trauma and the predisposition of the person experiencing the trauma both seem to play a big role in whether or not post-traumatic growth is likely to occur. People with social support networks or spirituality are more likely to experience post-traumatic growth. Perhaps for related reasons, trauma that is systematic or collective (such as being a prisoner of war) is more likely to induce post-traumatic growth than trauma which is personal or individual (e.g. sexual assault.) (Furthermore, no matter how distasteful the possibility may seem, it is possible that post-traumatic growth can take people past whatever their baseline was prior to the inducement of PTSD.)
raverbashing 8y ago
Oh so "in the future" she will be able to watch it without sound or to pause the video? Seriously, who thought blocking this is a good idea? Do you really need to see an entire video to see it is against the rules? It will also save (a lot of) time "Oh but then they can just do a shallow evaluation of the video" yeah I don't think that is the case with most of the videos.
paul_milovanov 8y ago
I think that's called "censors"
faissaloo 8y ago
The monolithic social network model just does not scale, this is exactly what we need federation for.
- agentdrtran 8y ago
  How does federation solve the problem of underpaid moderators?
  - faissaloo 8y ago
    Underpaid moderators are a side effect, not the main issue. If the network were federated each node wouldn't grow to such an unmanagable size, at least not without other nodes severing ties.
tareqak 8y ago
It looks like Facebook has responded to this article here: https://newsroom.fb.com/news/2019/02/commitment-to-content-reviewers/ https://newsroom.fb.com/news/2019/02/commitment-to-content-r... .
- smsm42 8y ago
  As I see it, TLDR is: we needed a lot of people, cheap, and mega-contractor-shops is the only way to get the headcount, cheap. We also wrote nice contracts that require them to treat workers well, so if they don't, it's clearly not our fault. Also, we hired somebody who will be responsible for dealing with such things. So everything is going fine, except rare exceptions which we take very seriously. Thank you for providing feedback, it was very valuable for us.
- gcb0 8y ago
  what a joke this response is. Article mentions A, B and C. with pictures and first hand anecdotes. FB press release issue a statement that they have contracts that do not allow for A, B and C. They don't even try anymore.
homemadejam 8y ago
Wow, that was eye-opening. Maybe a suggestion would be to have certain filters. Anyone below the age of 18 wouldn't be able to view posts by groups/people that have been flagged as "NSFW" posters? I know Tumblr implemented a similar idea a while ago.
unicornann 8y ago
These contractors work from different agencies. Sometimes they become full-time if they don't end up quitting their job. To make things worse, toxic culture like work place harassment is something they also experience
elchupanebre 8y ago
What's going to happen next: FB will quietly layoff all US moderators after moving the operations offshore.

304 points by ajay-d 8y ago

I remember when "user generated content" was a common phrase. It was great, users generate the content for the site and you just have to provide a platform for it. It seemed like a positive thing. Not it seems clear that if you open yourself to host something for your users you also need to be prepared to accept that you're going to be hosting the worst of humanity, exposing others to it, including people who work for you. It seems like such an extreme contrast from what it seemed like user content could be and what it is.

Bahamut 8y ago
This isn't particularly surprising to anyone who has been online for a while - sometimes you'd see gruesome stuff even on forums, and then you also had the culture on sites like 4chan. Moderation isn't something that one could really escape from even about 20 years ago, just now we're more cognizant of the effects of poor moderation and users are more aware of how people behave when moderators/admins are pushed to the boundaries of the set rules.
- duxup 8y ago
  It's not a surprise now. It is a big contrast as far as I think what folks thought in the early days of user content.
- tenpies 8y ago
  I think the big difference is how oversensitive society is now. Before, your moderators just had to deal with outright illegal content, maybe a flame war or two (or just move the discussion to the flame war section). If you ran into anything as a user, it was understood that you should report it and that the moderators would get to it with varying degrees of speed. There was no expectation of a perfectly safe bubble. Now, if even the most milquetoast comment which could possible be offensive if I assume the absolute worst about the poster, isn't moderated post-hate, you'll be reported to the media who will almost certainly archive it and use it to fill the next 1 - 24 hours of their outrage media cycle. I honestly see a future where people file lawsuits against web sites for not moderating enough, a failure to keep safe or sort of negligence. "Your web site gave me PTSD or enabled cyber-bulling". What's worst, they'll win.
  - komali2 8y ago
    I think this argument has been being made by people for two thousand years (this generation is too soft! Uphill both ways in the snow!) but I've yet to see a causation demonstrated.
    - mmsimanga 8y ago
      Just watch an old movie from about the 90s and you will be surprised how politically incorrect some of those movies were. They are just a reflection of what was acceptable in those days.
      - Pfhreak 8y ago
        And in 20 years we'll watch media from this decade and be shocked at what we put in our media. Society advances, culture changes. Viewing the past through the lens of the present is always going to be a different exercise than viewing the present through the lens of the present.
        reaperducer 8y ago
        Society advances, culture changes. I agree with the second part of that. Not necessarily the first. Viewing the past through the lens of the present is always going to be a different exercise than viewing the present through the lens of the present. Unfortunately, we live in a culture of intolerance. A few months ago there was a race car driver who lost his sponsorship because of something his father said before the driver was even born.
        komali2 8y ago
        >we live in a culture of intolerance This is quite the theory - do you have more backing other than the single instance you've just quoted? And what is "intolerance" and why is this "bad"?
        8y ago
- 013a 8y ago
  The the scale of the platform is the differentiator. If you're moderating a few hundred or thousand people on a forum, especially a site with a constrained vision and purpose, you're simply not encountering content and behavior like this on a regular basis, if ever. Now, look at facebook. Billions of users. No constrained vision or purpose to being there; if you're human, you're a customer, and human behavior can be pretty gross at the fringes. And maybe most importantly; they can't just Ban whatever people or content they want, due to both a profit motive and public outrage due to their preeminent position. This isn't a problem that has already existed, because the problem is with the Scale, not with the Behavior.
  - deleted 8y ago
    [deleted]
  - jachee 8y ago
    > if you're human, you're a customer No. If you're human, you're a product. Can't have the product spoiled, now, can we?
- spurgu 8y ago
  Yeah the problem today compared to BBS's/forums 10-20 years ago is that there are "smart" algorithms that will escalate the spread of fake news and shit content, compared to having everything equally exposed to everyone.
deleted 8y ago
[deleted]
rootusrootus 8y ago
This surprises nobody who has any experience with hosting. Even as far back as when I ran my own BBS, it was immediately clear that some people only wanted to leverage the content hosting for nasty things. If you were lucky, it was just perfectly legal pornography.

mruts 8y ago

Maybe Facebook just let all videos be posted, except when it’s illegal (the video, not the act). Pretty much the only thing that would fall in that category would be child porn. I’m not sure I really understand why big companies feel it’s their job to censor the world. Just let the floodgates open: allow violence, allow porn, allow whatever else. Who cares? And maybe this might actually good for people to see. I feel like we are leaving in this outrage culture that wants to censor everything. It’s not healthy for individuals or society.

duxup 8y ago
Including allowing the very first thing mentioned in the article?
- ahakki 8y ago
  Depends if very first thing mentioned, a graphic video of a murder, is illegal in the relevant jurisdiction. The legal status of snuff films is pretty complicated.
- thepangolino 8y ago
  Well, why not? It won't put it under your nose unless you actively search for it.
  - duxup 8y ago
    "It won't put it under your nose unless you actively search for it." That's not how social networks work.
- cf141q5325 8y ago
  The alternative is paying people to moderate it all day every day. With the mental damage this causes to people who are desperate for a job. Just blocking it yourself seems like a better solution. You dont have to be fair in what you block yourself and (surprise surprise) you dont need to watch it in the first place.

deleted 8y ago

[deleted]

ohiovr 8y ago

This job sounds more horrific than a crime scene cleaner. 4500 foot soldiers to clean up a billion users. Murder rape assault fraud. Burn it all with brimstone.

canttestthis 8y ago
Are you saying that there aren't enough people cleaning up or that cleaning up should be done differently?
- ohiovr 8y ago
  I honestly don’t know how moderators work and i actually don’t use facebook. But 1,750,000,000 / 4500 is about around 380,000 users per moderator. I think they need 5 armys of moderators instead of just a brigade trying to find the worst. They need to step in with private messages or public posts to keep the discussion from going rancid fast. I see it as being logisticaly impossible unfortunately but to keep discussions from going to that depth of evil they need to step up moderatoon pressure a couple orders of magnitude. I’ve said things here that moderators told me it isn’t allowed (actually said things here that were hurtful and unnecessary blanket statements) and I gripe to myself and limp back. If this pressure isn’t there yc would turn into kuro5hin in short order. I’ve seen discussion groups fall into the toxic slime pit more than a few times never to return. I think at least facebook groups can self moderate i have been told. I really doubt good behavior can be coded into a forum. Now that pit is billions of people not thousands. I have little hope it’s going to ever get better. Is this the destiny of all internet discussions? I worry about this a lot.

WisNorCan 8y ago

I would be impressed if Zuck and Sheryl decided to spend a day per year doing this job. I suspect that as a billionaire and the CXO of one of the largest companies of the world, you can get disconnected from the details. It would be valuable for them to see a different reality of their platform.

Shish2k 8y ago
What do you think / hope that the outcome of this experience would be?
- deleted 8y ago
  [deleted]
- untog 8y ago
  That they would gain knowledge of what the job is like, and so realise that their employees doing the work need a lot more support than they currently have.
  - wink 8y ago
    If it were Facebook employees that might even be the case. As the article states it's conveniently outsourced so they can just label it as a dollar expense without a guilty conscience.
  - intended 8y ago
    They probably know. And the secret is that there is no current solution. There is no automation to stop the sheer amount of dumb, sick, sad, horrifying, malignant and monstrous stuff being put up every minute from around the world. Frankly it could probably break social media the same way cancer could kill the tobacco industry. And this is simple moderation - ignoring newsworthiness, censorship, speech, propaganda and other issues this throws up. The best they probably hope is that the users forget about it. I mean seriously - what are publicly owned firms supposed to do?
    - 8y ago

endofcapital 8y ago

If I was running a marketing team or PR firm working on this problem I'd keep pushing the nonsense of AI magically solving everything someday. The actual nuts and bolts of content is pretty horrific, and they are obviously going to want to hide this reality. At least pay these people more than a dishwasher, please.

dsfyu404ed 8y ago

Meh. Sounds like a run of the mill shitty job. You can destroy your body laying bricks and unloading trucks or you can destroy your mental health doing something like this. Of course they're not paid well enough to compensate for the long term damage. Nobody ever is. There's all sorts of shit you might have to put up with in a shitty job, the work, the environment the customers. This job is shitty but some people are bothered less by this particular kind of shitty than they are by by angry customers yelling at you or laying bricks in the Arizona weather. Sure you might have to see some really screwed up shit among the sea of nip-slips and racial slurs but you get to do it in an air conditioned office without killing your body like the guys in the Amazon warehouse. Some people prefer that. Different people have different degrees to which they'll tolerate the various ways shitty jobs are shitty. At least one person the author interviewed said "well, it sucks less than Walmart". Back at the point in my life when I was doing shitty jobs I would have preferred brick laying or custodial work but I wouldn't have turned down this job if I needed it. Of course it's terrible for you and turnover is high but all the equally shitty jobs are this way. There really is no winning in the "minimum wage or thereabouts" income bracket. Edit: Since apparently this opinion is unpopular can anyone tell me why this particular implementation of shitty job is worse than all the other implementations?

untog 8y ago
Disregarding for a second that you're dismissing this entirely without having done so much of one minute of the job yourself... A lot of manual labour jobs are protected by unions, which establish safety standards, limits to shift length and healthcare to help employees when they need it. Sounds like these employees could do with the same.
- dsfyu404ed 8y ago
  >A lot of manual labour jobs are protected by unions, which establish safety standards, limits to shift length and healthcare to help employees when they need it. Sounds like these employees could do with the same. I'm not comparing to those jobs. The union dishwasher in some public university cafeteria gets time off, healthcare, etc, etc. that makes his or her job much less shitty than the infinite-term temp that works beside them. I'm comparing to the temp. You can make a job better/worse by adding/subtracting pay, benefits or working conditions. If I could get tech pay and benefits for construction work I'd be doing construction. If we're gonna compare this job to others then we should compare to other jobs that are "equally shitty" not jobs that are similar but actually compensated more highly if you roll the benefits into total comp.

TorKlingberg 8y ago

This part is scary: The moderators told me it’s a place where the conspiracy videos and memes that they see each day gradually lead them to embrace fringe views. One auditor walks the floor promoting the idea that the Earth is flat. A former employee told me he has begun to question certain aspects of the Holocaust. Another former employee, who told me he has mapped every escape route out of his house and sleeps with a gun at his side, said: “I no longer believe 9/11 was a terrorist attack.” I have a hope that people will eventually develop immunity to conspiracy theories after falling for enough of them and eventually seeing them disproved. But with things like pizzagate/Qanon it seems some people just keep going deeper.

monetus 8y ago
When it comes to neuroplasticity and change, all it takes is time. If you spend more in the shoes of those flagged for moderation than their opposites, maybe its inevitable that you attain a form of Stockholm syndrome.
- megous 8y ago
  It's a bit cultlike. Also add isolation from family/friends, and tight control of what you can and can't do, and having no trusted person (therapist there works for the comapny). It doesn't surprise me that it can lead to similar outcomes, as being in a cult.
rgoulter 8y ago
To extend the metaphor "develop immunity" a bit, I'd be curious about the mappings to: - vaccines, which build immunity by providing an inert or weak threat that the body builds resistance to. - herd immunity: restricts the spread of diseases if a sufficiently high proportion of the population is immune to the disease.
Diederich 8y ago
> seeing them disproved Well, many of these things essentially have a starting state as 'disproved', ya?

b1r6 8y ago

I would never be able to to do that job. Not because of the exposure to "bad things" like the article pushes, but because I've always felt totally fine (even bored) looking at that stuff, and I don't agree with censoring it. It's just "meh", another day of reality on Earth; the Internet is just a mirror held up next to it. Will we ever collectively realize this? It seems younger crowds are more normalized to this stuff because they grew up on the Internet. But add in the hypersensitivity in today's public social network sphere, and we're all freaking out over anything even potentially flammable.

pjc50 8y ago
I think you're on the far side of the "desensitization" process. It's not entirely normal to have a flat emotional response to everything, including images of human suffering.
- b1r6 8y ago
  Certainly a good point. I'd add that when it comes to suffering, I certainly feel discontent with that happening: In the article, the example of a stabbing video is given. I would strongly feel the need for retribution or justice for that victim. Same for all content that shows someone being hurt. I guess what I mean is that it's wrong to try and purge all this stuff as if it just doesn't exist. This is real content, real people, being hurt in reality. I'd feel disgusting trying to censor it, not disgusted by the content.
  - opT52ifk 8y ago
    Is not publicizing (in a system like Facebook, designed to proactively promote and spread attention-grabbing content to any audience it can) the same thing as censorship? In other words - I sympathize with the hard-ACLU/EFF stance on free speech. But there's a difference between government censorship and societal moderation - the latter has always existed, and the scale and automation of modern platforms in publicizing content that normally wouldn't spread so far is what's new. Should people be aware of bad things that happen in the world? Absolutely. Is broadening the audience for disturbing videos the right way to raise awareness? Maybe not. And if folks really feel such content needs to be published, they still have more options and reach than they did in years past even if "mainstream" places like Facebook moderate them. Granted this last point is getting a bit trickier, as people go after registrars and web hosts themselves for political reasons occasionally (i.e. if this was Cloudflare we were talking about I'd be in full agreement with you - then again Cloudflare doesn't run a recommendation service that automatically causes new unexpected content to appear in front of billions of people).

aeling 8y ago

If I think this is unacceptable[1] and I'm skeptical of the nebulous "AI will do this job in the future" claims, are there conclusions to be drawn other than "UGC isn't sustainable on a global, public platform"? That is, are there serious alternative options, or anybody working on ideas in this space? I think it's readily apparent that "just show everything" doesn't work if you want to attract a mainstream audience, but I'm reluctant to just give up on the global public platform that FB was originally idealized as. [1] I think I'd still find it unacceptable if the moderators were being paid 6 figures, had extensive 1:1 counseling, or any other perks - selling mental health for money is something I'm happy saying a utopian society wouldn't include.

0x54D5 8y ago
The public platform needs to be decentralized. We need a social protocol where the data lives in the protocol and not in some corporate server where it is subject to their whims. We currently live in a dystopia where your Twitter or Facebook could be banned at their whim leaving you a digital outcast.
- gipp 8y ago
  That's pretty much the opposite problem of what the GP is asking about.
- ghaff 8y ago
  So utopia is a completely decentralized uncensorable network where basically anything goes? Not wholly achievable probably because of DNS etc. but you could come close. Not an unsupportable position but you probably amp up the underlying issues Facebook is trying to address by 10x.
- BinaryIdiot 8y ago
  I’m not sure I follow. Almost any time there is an issue with a large social network folks on HN say decentralization needs to happen to fix it. But how do you “fix it” while still having the same or better user experience? It’s a very difficult problem to solve and I don’t think saying “make it decentralized! Make it live on a protocol!” is useful without the extensive “how” that everyone seems to ignore.

_bxg1 8y ago

I get a certain amount of secrecy for their own protection, but this: > They are pressured not to discuss the emotional toll that their job takes on them, even with loved ones, leading to increased feelings of isolation and anxiety. Just doesn't make any sense. Not only does it not protect the moderators, I can't even figure out what the cynical corporate interest would be.

duado 8y ago
The cynical corporate interest is that they don’t really want more people knowing about this whole underworld.
- Tsubasachan 8y ago
  I think most people prefer not to think too much about this underworld. Like when somebody throws themselves in front of a train one of the first things the police does is put up barriers. And how do you even talk with your loved ones about this kind of work?
em3rgent0rdr 8y ago
Indeed, I'd think the cynical corporate interest would be to watch out for well-being of their workers and to prevent PR backlash from workers getting mental issues.
- maxxxxx 8y ago
  "watch out for well-being of their workers" Usually the first corporate response is to sweep things under the rug and hope they will go away. Only if that doesn't work they will take real steps.
- egypturnash 8y ago
  That's the behavior of an enlightened psychopath who has realized how useful good will is. Facebook-the-company is an unenlightened psychopath who just wants money and power and doesn't care what anyone thinks about it.
8y ago

statictype 8y ago

Interesting that the moderators were provided by Cognizant - essentially a software outsourcing company.

product50 8y ago

At this point Casey Newton has so much angst against Facebook, that even if you show him (hypothetically) that everyone at FB is earning $100k and enjoying their jobs, he will still point out that the plumber who came into fix the bathroom the other day is being paid $40/hr and that is a travesty. /rantagainstbiasness

_bxg1 8y ago

The internet is one great big libertarian experiment. It illustrates all of the creativity, and all of the depravity, that result when you just put millions of people together and tell them to do whatever they want without consequences.

hnaccy 8y ago

If seeing it makes you cry it seems like you should be looking for a new job. There is a portion of population that would be generally unfazed by the content, it would be best for everyone if they worked these jobs.

untog 8y ago
You really think the people being paid $28k a year to watch videos of literal murder haven't thought that maybe they'd rather have a different job?
- Nadya 8y ago
  Tons of people watch these kinds of videos by choice over on /r/watchpeopledie (Warning: Very NSFW Subreddit if the name of it doesn't make that immediately obvious. Contains many videos of accidental deaths and murders, including those of children.) I'm sure some of them are jobless and would love to be paid $28k a year to do something they already do as a bit of a hobby.
  - untog 8y ago
    How many people who browse /r/watchpeopledie browse the sub for a length of time equivalent to an entire work day, Monday to Friday? And how many of those users, who browse videos of people dying all day every day, for pleasure, are qualified to judge what is offensive and what is not? These people should be referred to mental health professionals, not assess what is suitable for the world. You're also entirely missing my original point. The people currently doing the job don't like it. They don't want to be doing it. And yet they are. Why do you suppose that is? Perhaps... because there are no other jobs available? Saying "you should be looking for a new job" to these people is like telling a homeless man he really ought to be looking for a rental property in his budget range.
  - bendotero 8y ago
    That's sick. What you just described is someone who is unhealthy, not a person who needs a full time job to further indulge in something deviant.

crankylinuxuser 8y ago

Yep. And this is what you get when collectively "We" demand moderation. Its the other side of the fence... Usenet was the unmoderated wasteland, and Facebook is a the moderated hell-landscape of "ok not ok".

intopieces 8y ago
Usenet still exists and is moderated in a similar way against exploitative content in partnership with The Internet Watch Foundation (https://www.iwf.org.uk https://www.iwf.org.uk), depending on your provider. Giganews, the largest Usenet provider and owner of other affiliated providers, is a member of the IWF: https://www.iwf.org.uk/member/giganews-inc https://www.iwf.org.uk/member/giganews-inc.
- crankylinuxuser 8y ago
  But due to its decentralized system, subscribing to a block list from IWF is just that. Its the equivalent of a spamblock list.
  - intopieces 8y ago
    Not really. Usenet providers voluntarily subscribe to the list for you, leaving you without the choice.

Mr_Shiba 8y ago

This is a great opportunity to encourage everyone to delete your facebook & Instagram.

tjkrusinski 8y ago
Also stop using Google, Twitter, YouTube, etc etc because they all work similarly.
- Mr_Shiba 8y ago
  Agreed, duckduckgo ftw.

jrochkind1 8y ago

> Every moderator I spoke with took great pride in their work, and talked about the job with profound seriousness. They wished only that Facebook employees would think of them as peers, and to treat them with something resembling equality. > “If we weren’t there doing that job, Facebook would be so ugly,” Li says. “We’re seeing all that stuff on their behalf. And hell yeah, we make some wrong calls. But people don’t know that there’s actually human beings behind those seats.” Some respect and acknowledgement, and more than $15/hour, seems like it would be a good start.

jrochkind1 8y ago
Also: High-quality no-copay mental health services, that are also contractually guaranteed for, say, a year after severance (for any termination reason). It also occurs to me that even if this work arguably needs to be done, it doesn't need to be done by anyone 8 hours/day or 40/week. (Let alone in conditions where your job security depends on maximizing your "efficiency"). Anyone needing to do this should do it as a part of a job with a balanced set of work duties, with this being something they do only some of their work time. These things would hurt Facebook's bottom line of course. These people are being chewed up and spit out for Facebook's profit and reputation.
- zelon88 8y ago
  I agree that this could be a part of a bigger position instead of being the bread and butter, but we are talking about sub $30k/year jobs that nobody wants to do. We could just as easily make the argument that it's not safe for garbage men to hang off the back of the garbage truck, but at the end of the day a) nobody cares and b) I've got my own job that nobody wants to do that I'd love for someone to come along and make less shitty.
  - jrochkind1 8y ago
    > but we are talking about sub $30k/year jobs that nobody wants to do. Yes, that's the problem. There is no law of nature that says we need to pay people with the jobs that are worst for their health the worst and give them no respect or acknowledgement. If there are other jobs that is true for, yes, those too. I wouldn't tell you what to care about, but I would say that we all would benefit by taking care of those of us whose jobs harm them. Your job should be less shitty too, it doesn't take away from that to point out other people who's jobs are harming them and aren't getting what they need. It is reasonable to begin focus on the jobs that are the _worst_, mentally or physically. Maybe yours is one of those too. It's also reasonable in the HN venue to focus on the jobs that are the worst that make possible the economy that many of us reading this benefit from by being paid a lot more than $15/hour to contribute to, and probably aren't as harmful to our health as those for many of us. If your situation is especially dire, then, yes, you too.

scoutt 8y ago

I always wondered why they don't have a system where the videos or images are shown first filtered, like blurred, and then at the discretion of the operator, they can take "several levels of blurring" off until they see the original image. That way, for some images/videos/ you can tell if it's violent/inappropriate right away, and the operator may see "less" of whatever is disturbing for them. I've also wondered how they deal with long videos that may seem correct for the first minutes, but the real inappropriate content is interlaced for a few frames in-between or later in the video.

perbu 8y ago
It's low-value work. Facebook doesn't value these people enough to employ them so they likely don't have access to properly engineered tools.
- dymk 8y ago
  What makes you think that? How can you say that with such confidence? Are you letting your pre-existing negative feelings color your assumptions about what people actually think at the company? I used to work there, and I built tools for these workers. Every one of the other engineers that I knew had nothing but respect for them, and everybody recognized that it was important work.
  - agentdrtran 8y ago
    So important their bathroom breaks are timed, they get inadequate counseling, and they get paid $28k a year!
    - tjkrusinski 8y ago
      You have to remember that those are policies put in place by the contracting company, not Facebook or the engineers that make the tools that these people use.
      - 8y ago

aboutruby 8y ago

For those used to shock sites from 10-20 years ago, this is very different, I'm not going to link to anything but the disturbing content is on a few whole new levels. I can't even imagine doing this job. This should be 100% automated with only 0.01% error rate for the first-pass filter (e.g. before it reaches humans). They are willing to spare their users the horrors but not their own employees/contractors...

intended 8y ago
Facebook tried the automated system by using employees to train the machine. It failed. They returned to non Facebook employed humans doing the moderation
- rock_hard 8y ago
  I read somewhere that it’s actually a hybrid approach! For some things liken nudity ML based solution work fine. But other things such as hate speech are more nuanced and require human oversight. That said ML is used everywhere possible to flag content to moderators. For some categories the work of human moderators is used to train the ML models. Facebook and other companies like YouTube are well insentivized to do the right thing here and automate as much as possible for all the reasons outlined in OP
- fencepost 8y ago
  One thing I hope they're doing is advanced detection of duplicate inappropriate content - e.g. by splitting a confirmed properly blocked video into frames and identifying matching videos based on frames matching (or even key parts of frames). You'd still have to investigate/review partial matches, but something like that could cut down a lot on duplicate effort and could auto-identify a lot of things that would need manual review.
agentdrtran 8y ago
Machines cannot do this job yet.
ddalex 8y ago
I developed the first version of the screening system for a corporation about 10 years ago, when UGC was still in its infancy. We had the help of a Microsoft tool that would match the visual fingerprint of a photo against a known-images database that prevented human operators from re-seeing an already tagged image in a different cloud, hoping to reduce the psychological scarring of the human workers that screened the content. We also evaluated automatic classification of images, but it's very hard to detect soft core porn from shock images even with today's algos, never mind 10 years ago. One advice from a legal counsel that I received at that time still stands in my memory; she said: "Never look at the images, nevermind how curious you are. It's not just illegal and can land you in prison, but it's permanent. You cannot un-see something you've seen."

intended 8y ago

This sounds about right >dark humor and camaraderie Yes. No one can know what happens if you start moderating, except unusually inquisitive users or other mods. > believing conspiracy theories Not all ideas are equal - and not all brains/people are equipped to deal with them equally. Your mod team will soon get polarized or even subverted if faced with the kind of crazy work these facebooks mods have to do. If you are actual community mod, life is slightly better, since there’s more context.

rhema 8y ago

Is it just me, or do should we expect that some small population will be naturally equipped to deal with this job? We don't expect everyone to want to be a soldier or medical doctor. These people deal with situations that would make most people throw up. If you can't handle an hour of browsing 4chan, maybe you should know that content moderation is not a good fit for you.

intended 8y ago
The pay is poor, and the population willign to do this may not be in the same region as the jobs, may have better jobs and so on. The workload is massive. There’s not enough people, and the firms don’t want to pay as much as they need to.
sharkweek 8y ago
I'm not sure if that solves the core issue of long term mental effects, even on the "hardest" people. Stepdad is a retired firefighter, has kind of that old-timey tough guy thing going on (although deep down he's a real softy) - he rarely likes talking about the hardest parts of the job, but the few times it has come up he mentions seeing some things that haunt him. I remember one time him mentioning seeing something so grotesque when he pulled up to the scene that he started laughing at the absurdity. Sort of in a way that he could do nothing but throw his hands up and say "well fuck this." He jokes about some of that stuff, but I can sense pain beneath the humor.
- darkpuma 8y ago
  Consider that there are people who've only seen a small fraction of what your stepfather has, but can't handle it nearly as well.
tomcam 8y ago
It seems to me (and I’m serious about this) a classic sociopath would in fact be willing to do this kind of thing for money if I understand correctly, with no ill effects. They apparently have conscience and are something like 1% of the population. But I think they tend to be smart, and can probably get higher-paying jobs.

intended 8y ago

Remember that Reddit quarantines /watchpeopledie These mods may randomly see gang videos (go look for them if you want to see how tough you are) on any day. Viewing them would make many people I know retch, worse than if they saw a gory horror movie. And then some ideas, if presented without an antidote (creationism, flat earth, conspiracies) will subvert the people watching it. And then comes the worst part - the damage to society itself. If violent images are removed, depending on what people in Facebook and their friends, consider safe- then they are also deciding what is newsworthy. And Facebook mods are not in a position to decide if an image of burning child, will stop a war or awaken the conscience of a people. All they know is that it breaks “rule 4- death: graphic wounding, no gore.”

casefields 8y ago
The only one that got to me was Funkytown. I've got to imagine if they increased pay and hunted through those users, they could find people that can withstand sifting through this stuff.

wespiser_2018 8y ago

I wonder how far we are from automating this with image recognition. The interesting thing is policy aspect to the learning problem, does an post/image/video violate a collective set of rules? They are certainly building a huge enough labelled dataset of images, decisions, and policies. I wouldn't be surprised if this went automated in 5-10 years!

throwaway-1283 8y ago

I feel like selling moderation tools would be a huge biz for any co with UGC...

tjkrusinski 8y ago
Yes, it already is.

walrus01 8y ago

A huge amount of content moderation has been offshored to the Philippines, where salaries are considerably less than $15/hour. Combination of access to good internet connections, modern office space, an English speaking workforce, and low salary levels. I have not seen this documentary, but compared to the verge article, I think a major difference is that the filmmakers did not get cooperation from Facebook or the third-party content moderation company. In fact their photos were posted and employees were told to stay away from them. https://www.vice.com/en_us/article/ywe7gb/the-companies-cleaning-the-deepest-darkest-parts-of-social-media https://www.vice.com/en_us/article/ywe7gb/the-companies-clea...

jrochkind1 8y ago

See also this upcoming book by Dr. Sarah T. Roberts, the product of several years of research. https://www.amazon.com/Behind-Screen-Content-Moderation-Shadows/dp/0300235887 https://www.amazon.com/Behind-Screen-Content-Moderation-Shad... Here's a presentation Dr. Roberts' gave at conference last week on the topic too: https://www.youtube.com/watch?v=7mdMtukvtxc#t=15m50s https://www.youtube.com/watch?v=7mdMtukvtxc#t=15m50s (Until recently at least, most of this content moderation work was done by contractors overseas, largely in the Phillipines. Can you imagine what the working conditions are like there?)

2sk21 8y ago
The presentation you linked to by Sarah Roberts was very interesting to say the least. Highly recommended. She adds a lot of information about the condition of content moderators in other countries.

iambateman 8y ago

Question: is there a legal prerogative for Facebook to proactively moderate content in the US? Don't get me wrong: the article is incredibly hard to read. When humans become untethered from social mores, they are capable of shocking and evil behavior. But...can't people mostly police this themselves? If I had a "friend" posting video of a dog being stabbed, I wouldn't be their friend for much longer. It seems that the result would be relatively small cesspools of highly antisocial behavior. Cesspools which will continue to exist regardless of any moderating effort. Alternative to moderation: introduce "public content ratings" for the content users post publicly. Machines automatically assess content and rate the user based on the maturity level of their public discussion. I would be "teen-mature" because I sometimes say "shit" online and perhaps another user would be "highly mature - disturbing violence and sexual content". Their ratings would weight groups. Then, it's up to me to decide who I associate with online. Caveat: nothing is a panacea. We're all trying to figure out how to handle the fact that we gave every maniac a megaphone to the masses.

Despegar 8y ago
If HN can't do without moderators why do you think Facebook could?
- iambateman 8y ago
  I don't think Facebook can do without mods entirely. Maybe they don't have enough mods or strict enough rules. I honestly don't know. But Facebook has a more user-curated thread, whereas HN is public. In a sense, I am my own mod on FB. FWIW...I definitely don't use FB on a regular basis. Haven't since Oct 2017.
rchaud 8y ago
> If I had a "friend" posting video of a dog being stabbed, I wouldn't be their friend for much longer. It's less about it being for you, and more about that it would be hard to sell ad space when it sits next to content of that type. Your friend might not post those materials, but what about Pages and Groups, which is where I'd say most FB users spend their time these days.
8y ago

NoblePublius 8y ago

The only thing remarkable here is that these people are not hired by Facebook directly. Someone needs to ask Zuck why he thinks these hires aren’t worthy of catered lunch and @fb.com emails.

mmagin 8y ago
Then there would be the risk that the rest of fb's employees might be exposed to the reality of this.

daenz 8y ago

>If we weren’t there doing that job, Facebook would be so ugly It would be 4chan, minus the anonymity.

faissaloo 8y ago
That would be many times worse than all the chans combined, imagine a chan exclusively used by middle aged mothers where anyone you talk to is doxable at the click of a button.

lazugod 8y ago

> When I ask about the risks of contractors developing PTSD, a counselor I’ll call Logan tells me about a different psychological phenomenon: “post-traumatic growth,” an effect whereby some trauma victims emerge from the experience feeling stronger than before. Wow. Wow. Is this really a thing, excusing forced trauma by claiming it makes people better?

darkpuma 8y ago
You seem outraged at the suggestion that this could happen. Post-traumatic growth is an actual phenomena: https://en.wikipedia.org/wiki/Posttraumatic_growth https://en.wikipedia.org/wiki/Posttraumatic_growth Of course that's no excuse to downplay the occupational hazard of PTSD. But it's a real thing; facebook didn't make it up.
dsfyu404ed 8y ago
People respond in varying manners to traumatic stress. Post traumatic growth is simply the other extreme. For example, some people take traumatic stress at really poorly (PTSD). Some people develop a world outlook to the tune of "well no matter what happens life is looking up from here" (post traumatic growth). The counselor is looking at the glass half full. You kind of need to be a glass half full person to persist in that line of work.
- watwut 8y ago
  PTSD is not an issue of attitude while your description of post traumatic growth is nothing but attitude. The two are incomparable. The actual benefit of what you describe as growth is quite low to offset clinical PTSD which affects life quite a lot.
  - darkpuma 8y ago
    PTSD isn't universal. Not everybody that experiences trauma will develop PTSD. So it's not a matter of "you have PTSD but at least your outlook on life has improved."* Rather, sometimes with some people, trauma won't induce PTSD but will cause post-traumatic growth. Research suggests there is a correlation between PTSD and post-traumatic growth, but it's not a hard and fast rule. Furthermore there are many factors in play. The nature of the trauma and the predisposition of the person experiencing the trauma both seem to play a big role in whether or not post-traumatic growth is likely to occur. People with social support networks or spirituality are more likely to experience post-traumatic growth. Perhaps for related reasons, trauma that is systematic or collective (such as being a prisoner of war) is more likely to induce post-traumatic growth than trauma which is personal or individual (e.g. sexual assault.) (Furthermore, no matter how distasteful the possibility may seem, it is possible that post-traumatic growth can take people past whatever their baseline was prior to the inducement of PTSD.)

raverbashing 8y ago

Oh so "in the future" she will be able to watch it without sound or to pause the video? Seriously, who thought blocking this is a good idea? Do you really need to see an entire video to see it is against the rules? It will also save (a lot of) time "Oh but then they can just do a shallow evaluation of the video" yeah I don't think that is the case with most of the videos.

paul_milovanov 8y ago

I think that's called "censors"

faissaloo 8y ago

The monolithic social network model just does not scale, this is exactly what we need federation for.

agentdrtran 8y ago
How does federation solve the problem of underpaid moderators?
- faissaloo 8y ago
  Underpaid moderators are a side effect, not the main issue. If the network were federated each node wouldn't grow to such an unmanagable size, at least not without other nodes severing ties.

tareqak 8y ago

It looks like Facebook has responded to this article here: https://newsroom.fb.com/news/2019/02/commitment-to-content-reviewers/ https://newsroom.fb.com/news/2019/02/commitment-to-content-r... .

smsm42 8y ago
As I see it, TLDR is: we needed a lot of people, cheap, and mega-contractor-shops is the only way to get the headcount, cheap. We also wrote nice contracts that require them to treat workers well, so if they don't, it's clearly not our fault. Also, we hired somebody who will be responsible for dealing with such things. So everything is going fine, except rare exceptions which we take very seriously. Thank you for providing feedback, it was very valuable for us.
gcb0 8y ago
what a joke this response is. Article mentions A, B and C. with pictures and first hand anecdotes. FB press release issue a statement that they have contracts that do not allow for A, B and C. They don't even try anymore.

homemadejam 8y ago

Wow, that was eye-opening. Maybe a suggestion would be to have certain filters. Anyone below the age of 18 wouldn't be able to view posts by groups/people that have been flagged as "NSFW" posters? I know Tumblr implemented a similar idea a while ago.

unicornann 8y ago

These contractors work from different agencies. Sometimes they become full-time if they don't end up quitting their job. To make things worse, toxic culture like work place harassment is something they also experience

elchupanebre 8y ago

What's going to happen next: FB will quietly layoff all US moderators after moving the operations offshore.