28 ms·

This was a valuable article to read. Facebook is enormously valuable. They made something like $15B in net income in the last four quarters. Content moderator

This was a valuable article to read. Facebook is enormously valuable. They made something like $15B in net income in the last four quarters. Content moderators are a necessary condition for that profit. If kiddie porn, gore and animal cruelty flooded the network, it would cease to be a destination visited by people that advertisers will pay to reach. And yet, there are two sets of entry-level knowledge workers at Facebook: engineers ($150k/year, benefits, upward career trajectory) and content moderators ($30k/year, no benefits, likely going to acquire mental illnesses). I understand the arguments about supply and demand of labour, but I'd have more respect for Facebook if they demonstrated awareness of this issue. The article talks about moderators re-evaluating the same piece of distressing content that they've already flagged. Why? I suspect because the moderator is cheap, and so Facebook isn't putting in the effort to ensure that every judgment needs to be made the minimum number of times. More so than salary, I suspect Facebook considers the moderator cheap in terms of reputation risk. By outsourcing to contractors located offsite from main campus, engineers aren't thinking daily about the absolutely horrible stuff moderators are seeing, and so the one group doesn't impact Facebook's ability to hire engineers. This is a guess - can anyone at Facebook speak to whether engineers are aware of the working conditions of moderators, and agitate to improve their lot?

baud147258 7y ago
> cheap in terms of reputation risk Also in terms of the reputation of Facebook: any issue with moderation failures and Facebook can easily says that it's the contractor's fault
RegBarclay 7y ago
I thought the same thing -- that once content is moderated, it should be trivial to block exactly the same content from being posted and moderated again. What a waste.
- Nasrudith 7y ago
  The problem is actual use and context sensitivity. Since if they let historic signifiant images get flat banned they also look like big bad idiots trying to memory hole history for blocking say death camp images while still having to ban asshats who post it and say "Not enough." Neither the text nor the image themselves would be ban worthy but in combination it is boot-worthy. Extrapolating has Scunthorpe problem issues.
- LinuxBender 7y ago
  The content may get modified quite a bit if censored though. A good example was the ChristChurch video. 4chan members modified it so many times, there are probably 4k-5k different iterations of it. NZ made a big deal about blocking it and arresting people for viewing it, making the demand for it significantly higher. There would have to be some really efficient and highly optimized ML to group all the variations together with a common label. AFAIK this does not exist for videos yet. The only partial mitigation that comes to mind would be if a small circle of your friends had to "approve" your video before people outside your circle could see it. That would put them on the hook for sharing bad things too. That might even reduce the psychological load on the moderators.
pawelmurias 7y ago
It's knowledge workers vs unskilled labour.
- McWobbleston 7y ago
  It's also people not being paid nearly the value they produce for an organization. Everything is always a race to the bottom for us
- eswat 7y ago
  I was thinking about this more but you’re right. I doubt there’s any acquired knowledge needed to know what’s repugnant to the human mind.
  - cldellow 7y ago
    I don't have first-hand experience with the job, but I've read a few articles about these sort of teams, as they're widely deployed by all the large Internet companies. In my view, it's knowledge work. Sure, it's knowledge work that almost anyone can be trained to do, but there's a skill set required. For example, moderators need to make judgment calls that distinguish between offensive uses and artistic/journalistic/parody uses. There are many cultures and in-groups on Earth, which makes this harder than it might seem at first blush. Not all nudity is obscenity, not all hate speech is blatant.
    - qudat 7y ago
      > For example, moderators need to make judgment calls that distinguish between offensive uses and artistic/journalistic/parody uses. There are many cultures and in-groups on Earth, which makes this harder than it might seem at first blush. Not all nudity is obscenity, not all hate speech is blatant. In these cases I'm not sure it really matters. Over-moderation is going to win overall and I don't think moderators are going to get into that much trouble for being slightly over-zealous. I could be wrong but I feel like after a week of being on the job you'd get the hang of it.
      - 7y ago
anon029102 7y ago
> Can anyone at Facebook speak to whether engineers are aware of the working conditions of moderators, and agitate to improve their lot? They are passively aware but mostly don't care. The engineers who express frustration are in the vast minority. Most engineers are just super happy about their own situations, very focused on PSCs, and try to keep to their own stuff. Any internal frustrations are channelled towards appropriate feedback channels where the anger is watered down.
- johnkpaul 7y ago
  PSCs?
  - tristanperry 7y ago
    From a quick Google, "Performance Summary Cycle" - presumably regular reviews and bonuses.
  - deleted 7y ago
    [deleted]
- plankers 7y ago
  Nobody's making waves when everybody is focused on their own struggle. Gamifying for a docile workforce.
Miner49er 7y ago
Facebook won't do anything to help these moderaters unless the cost of the negative PR exceeds the cost of doing something. Engineers at FB could help achieve this by causing agitation, for sure. Maybe the best solution is for the moderaters to unionize.
- plankers 7y ago
  If you can think of a practical way to unionize workers on a 1-2 year contract with a high turnover rate, I'd love to hear it. The deck is stacked against employee organization. If I were conspiracy-minded, I might even say this was by design.
  - Barrin92 7y ago
    >If you can think of a practical way to unionize workers on a 1-2 year contract with a high turnover rate, I'd love to hear it. Adopt the Danish model, make it mandatory for companies with workforces over a certain size to at least vote on unionisation. Will shift the default from not being unionised to being unionised except for cases were the workforce is actively convinced its detrimental.
    - plankers 7y ago
      I love that idea! But I have a hard time seeing a law like that being passed in the U.S., even in liberal California. Side thought, things like this help me understand why the Danes are so darn happy.
- shujito 7y ago
  ...which it's unlikely to happen because of the NDA's they have to sign. If there's something wrong with this reasoning, please elaborate.
  - vageli 7y ago
    > ...which it's unlikely to happen because of the NDA's they have to sign. > If there's something wrong with this reasoning, please elaborate. I would be utterly surprised if an NDA could be used to prevent an employee from exercising their right to organize (in the US at least). Just because you sign something does not make it legally enforceable.
Shivetya 7y ago
anyone who has done moderation for any active forum can tell you how nightmarish it can be. the more popular the platform the more problems you can have. anecdotal, friend who streams on twitch also mods for a few streamers and they even have issues when a stream is in subscriber mode only. simply because anonymity and distance from those being affected empower people to do bad things edit : I am really surprised there is/are not companies which would spring up to provide these services seeing how most of these activities are required by law. however before bemoaning what they are paid, just go look at your local 911 operators who are government employees. just because we think it should be paid more doesn't mean others do.
- kevingadd 7y ago
  Anonymity is definitely empowerment (especially because Twitch has no effective ban system, so bad actors will create dozens of accounts and keep coming back), but even people acting under their real names will do horrible things. We've seen this with real names on FB/Google+. It seems like there are no easy solutions for this and it's really frustrating: Societal norms are just completely shredded right now.
  - nvr219 7y ago
    > but even people acting under their real names will do horrible things. We've seen this with real names on FB/Google+. And on twitter. Their most famous user uses his real name.
Cthulhu_ 7y ago
Re: salaries, don't forget that there's also salaries working on the content moderation problem - taking the huge datasets that the content moderators are reviewing and applying machine learning and content ID and I have no clue what else more to automatically filter out a lot of the content, ideally with such a high certainty rate that the moderators don't see it in the first place. I want to believe the content moderators only end up seeing the 1% of content that is dubious, or brand new content (not published and hashed earlier). I don't understand in that regard why the one video mentioned in the article (about the iguana) wasn't marked as "ok" in the systems, or marked as "on hold for law enforcement" so the moderators didn't have to see it again.
ralusek 7y ago
I don't understand how this isn't a problem perfectly solved by the market. If it's a shit job that isn't worth the pay, people won't do it. If it's an okay job that isn't really worth the pay, they will have high turnover. High turnover has a cost, maybe it's worth it to them. If the turnover is so high or the job is so undesirable, maybe they'll axe the role. If the role is so important to the business that it can't be axed, but people aren't taking the jobs, they'll have to pay more for it. If they can't pay more for it because it cuts too much into profitability, but it's necessary for the company to operate, then they are no longer a viable company in the market. Or they'll have to innovate or adapt. All possibilities are fully encapsulated in very simple dynamics.
- sandworm101 7y ago
  >> If it's a shit job that isn't worth the pay, people won't do it. That assumes perfect knowledge. As mentioned in the article, these moderators often do not understand what they are getting into until long after they have committed to the job. Once you have quit a previous job, moved, setup your new life, only then do you realize your new "brand consultant" job means watching kittens die all day. Jobs like this also open persons to realities that we normal keep tucked away deep in our brains. We all know that cops don't investigate most reported crimes, but that doesn't come up very often for us. A facebook moderator sees the crimes, sees the lack of response by the police or anyone else, every minute. Repeatedly, every minute. Nobody can appreciate what this does to ones' mental state until you have been there.
  - debacle 7y ago
    Nobody moves for $30k a year.
    - skyyler 7y ago
      Are you serious? I know people that have moved for less. Making absolute statements like that doesn't do much for anyone.
    - sandworm101 7y ago
jfindley 7y ago
Given the description of the videos in the article, I'm not sure that merely paying the moderators more would solve it. I think for most people that I'd want to be moderating content (e.g. not sociopaths) making them richer isn't going to do anything to deal with the real reason this is worse than a call centre - the content itself. I'd rather see facebook put more effort into protecting moderators from the content they're viewing. I realise this is a non-trivial problem, but here's a few ideas of things they could do which may help: * Once a video is confirmed bad, fingerprint it and block future uploads. * Provide colour-inversion buttons to help reduce visual impact * Rotate people between areas, so they aren't looking at the same class of bad content constantly * Use ML training to detect which content is likely to be the most severe and for this content, reduce minimum viewing interval to ~ 5 seconds, and ensure that a single individual doesn't get too many of these. * Flag accounts who post bad content too often, and shadow-ban them so their posts are not visible by anyone else (this could go in stages, starting off by blocking re-shares of their posts or something)
- quadcore 7y ago
  Provide colour-inversion buttons to help reduce visual impact I was thinking also about blurring the picture/video.
  - afasadaf 7y ago
    They could have levels, and use moderator consensus to remove the pictures 1. Blurred+Mosaic 2. Contour lines only 3. Windowed/Striped (only showing like 10-25% of the photo) If 2/3 of the moderators above agree that it's a bad photo, it moves onto stage 4: 4. Color filters - Break the photo down into k-means color clusters, moderators choose which colors to see. They can see 100% of the colors, or one, or two, or however many they need. If #4 agrees, the photo goes into the round folder. No need for anyone to ever see the whole photo. It'll be pretty obvious that something was violating the TOS without anyone being able identify anything specific about the photo. We're not talking about 'beyond the shadow of a doubt' here, we're talking about 'yeah, that's probably not ok.'
- zanmat0 7y ago
p1esk 7y ago
How about training a classifier to detect and remove potentially bad stuff, then let the users who uploaded it argue that the classifier made a mistake? Only in those cases have human moderators look at it?
- Sharlin 7y ago
  They already have classifiers, of course. It's not like the moderators go through everything posted to FB–you'd need millions of people to do that. What the mods actually do, presumably, is review content reported by users.
  - p1esk 7y ago
    Then perhaps they need better classifiers? I don’t remember seeing any papers from FB about that, so it does not seem like a priority for them. Or just make the existing bad classifiers more aggressive and shift the moderator’s effort to reviewing good content.
sandworm101 7y ago
>> If kiddie porn, gore and animal cruelty flooded the network, it would cease to be a destination visited by people that advertisers will pay to reach. Or the opposite. People flock to the internet to see this stuff, the material that isn't on TV or even dedicated commercial websites. They get to view it in the privacy of their homes, a secret between them an their trusted "friends" on facebook. Facebook knows that if it really cracks down, if cops bust down doors over this stuff, that people will go elsewhere to share horrible material. So facebook does a poor job of moderation, one just good enough to avoid new regulation but not so good as to actually make a difference. Advertisers care about eyeballs. They may claim to not want to be associated with X or Y content, but in reality they would rather be in front of an audience than not.
- lapnitnelav 7y ago
  Advertisers do care a lot about where their ads are displayed. There's a whole section of adtech dedicated to brand safety. I believe it's over the top[1] but it's something marketing folks do care about. Also the good old pop up ads were created for that reason, aeons ago. [[1] https://www.campaignlive.co.uk/article/advertisers-embrace-news-not-shy-away-brand-safety-paranoia/1458646 https://www.campaignlive.co.uk/article/advertisers-embrace-n...
geggam 7y ago
150k a year in Silicon Valley is similar to 30k a year in the midwest... Where are the moderators living ?
- kixiQu 7y ago
  This is very much an answered-in-the-article question.
- unicornfinder 7y ago
  Poland
scarface74 7y ago
$30k/year, no benefits, likely going to acquire mental illnesses Why do you think they don’t get benefits? Facebook contracts out to another company. The company they contract with still is probably providing some benefits
- cldellow 7y ago
  That's true, the article says Cognizant offers "a 24/7 hotline, full healthcare benefits, and other wellness programs". But at the same time, it quotes the on-site counsellor as having said "I don’t really know how to help you guys" when an employee went to him for help. This leads me to believe that there is a qualitative difference in the calibre of benefits Cognizant provides versus those that Facebook provides. Further, since the job has high turnover and can be expected to put a high toll on an employee's wellbeing, it's also important to me to think about what happens to their benefits if/when they quit. In the US model of employer-linked healthcare, this tends to result in losing your benefits when you need them most. By contrast, other organizations that do similar moderation jobs seem to offer psychological health care treatment even after the employee leaves the company. See https://www.theguardian.com/news/2017/may/25/facebook-moderator-underpaid-overburdened-extreme-content https://www.theguardian.com/news/2017/may/25/facebook-modera... for more info.
Bakary 7y ago
>This is a guess - can anyone at Facebook speak to whether engineers are aware of the working conditions of moderators, and agitate to improve their lot? If you go work at Facebook as a engineer chances are you aren't too bothered by ethical questions
wrongdonf 7y ago
It says right in the article that the content moderators have full medical benefits
- cldellow 7y ago
  Sorry, yes, I could have expanded on that. You're right, they provide health benefits, in theory. My experience with the US health care system is that there is a wide gulf in health benefits. When I worked for Microsoft in the late 2000s(as an employee, not as "dash trash", as contractors were called), I paid some very small premium like $10-20/month for a plan with 0 deductible, 0 copay, and pretty much every doctor was in network and covered at 100%. When I dislocated my shoulder, some orthopedic surgeon who used to consult for an NBA team looked at it for me. I understand that that was considered quite good insurance and that most people in the USA don't get that level of care, much less at that price. If you read Glassdoor reviews of Cognizant, people complain about the insurance offerings available to employees. https://www.homebaseiowa.gov/sites/homebaseiowa.gov/files/document_uploads/2017%20Cognizant%20US%20Benefits%20Overview.pdf https://www.homebaseiowa.gov/sites/homebaseiowa.gov/files/do... looks like it might be what they offered 2 years ago. It doesn't look very good to me. If I was earning $15/hour, I think I'd pretty much buy the cheapest plan and only use it if I was actively bleeding to death. I'll also link to https://www.theguardian.com/news/2017/may/25/facebook-moderator-underpaid-overburdened-extreme-content https://www.theguardian.com/news/2017/may/25/facebook-modera... again - it describes how other companies that moderate distressing content approach this. It includes the company paying for psychological treatment, including after the employee leaves the company.
  - pravda 7y ago
    >I understand that that was considered quite good insurance and that most people in the USA don't get that level of care, much less at that price. AFAIK, Microsoft 'self-insures', in the sense that they simply pay the medical bills for their employees.
meerita 7y ago
"If kiddie porn, gore and animal cruelty flooded the network, it would cease to be a destination visited by people that advertisers will pay to reach." I bet this can be done with ML and IA instead human power. Yet, FB uses their human cattle to censor conservative voices.
- mythrowaway1124 7y ago
  Good luck! There's at least a billion dollars in it for you if you figure out how to do this.
- kartan 7y ago
  > I bet this can be done with ML and IA instead human power. IA and ML work in bulk. But they misclassify things that a human easily can discern. Think Tesla VS Google. Google wants a self-driving car, and that is not working yet. Tesla wants a car that helps you to drive and that is good enough. It's order of magnitude easier to be almost always correct that to be correct all the time. And you only need one kiddie porn video in your Facebook feed to start to worry about Facebook.
  - meerita 7y ago
    It is matter of time this will happen. Even with self-driving cars. Using people to do this is just killing people. Think all the mental effort that has to be done to censor. Humans aren't that perfect and cannot oversee the entire FB daily activity.
nonwifehaver3 7y ago
How can kiddie porn, gore and animal cruelty "flood the network" for people using Facebook? Don't you normally just see things from your friends, ads, and groups that you are a part of? If they post that, either call the cops, block them on Facebook, don't hang out with them irl, or leave the group as appropriate. Problem solved? I haven't used Facebook in 5+ years so maybe this is no longer accurate.
- lazyasciiart 7y ago
  As you may have guessed from the downvotes, this is not even close to accurate. It should have been possible for you to realize this was inaccurate simply from how trivial the problem looked to you in comparison to how much effort is being spent on it.
  - nonwifehaver3 7y ago
    I genuinely don't understand how an average Facebook user would encounter videos like those described in the article. Is it some recommender system gone awry?
    - lazyasciiart 7y ago
      As far as I know, mostly because people post them as comments on public or widely shared items, along with "get free sunglasses" and every other kind of spam. There are tons of these public posts from pages that do news, memes, politics, trolling, regular advertising, whatever.
hashkb 7y ago
> Facebook is enormously valuable. They made something like $15B in net income in the last four quarters. Their profit is not proof they are valuable, because they are willfully abdicating responsibility for the damage they cause, and, further, involving themselves in government debates over their own regulation. Advertising businesses are inherently suspect, and when one is addictive and potentially enabling election fraud etc, we need to be extra careful in declaring its value. Anyone participating in the debate should have to disclose if they are a daily user or if their company advertises on any social network. (I'm clean) Edit: my opinion of course is that FB, Insta, etc (any advertising product with push notifications and/or infinite scroll) ought to be illegal. In 50 or 100 years we'll all recognize that social networks are like internet cigarettes.
- mehrdadn 7y ago
  > my opinion of course is that FB, Insta, etc (any advertising product with push notifications and/or infinite scroll) ought to be illegal. In 50 or 100 years we'll all recognize that social networks are like internet cigarettes. Well cigarettes are still not illegal, so we have some ways to go...
  - allthecybers 7y ago
    This was an earlier comment that disappeared. Frankly I ask myself consistently "what net value does Facebook add to the world?" other than being a 100% optional vice.
ummwhat 7y ago
I've talked about this somewhere else. What you're seeing is just a manifestation of what I call "the fundamental problem of user created content." Said problem is that warehousing and distributing content scales insanely well but curation does not. Until we have strong AI, curation is a manual process. Moderators just aren't efficient enough at processing content for their output to pay for a full time salary. You can cut costs by making end users into moderators (the Reddit model) but results may vary. This problem applies to other forms of sites and content as well. The app store gets hundreds of submissions in the game category per day. Hosting and distributing that content is easy. Only a tiny fraction of those games are going to be played and rated enough times to show up in a recommendation engine. The bulk of the incoming content stream isn't being matched to interested people at all (sorting by interesting is the same curation issue as filtering by offensive). Literally every content platform is going to have some problem similar to Facebook. We can blame Facebook, but the reality is no one has a good solution. Not even me.
- hashkb 7y ago
  > We can blame Facebook, but the reality is no one has a good solution. We can take a step backwards and eliminate Facebook, Insta, etc from our societies and enact legislation that will govern the second generation of social networks.
  - Jedi72 7y ago
    What legislation? Specifically? What do you want to make law, that they have to hire moderators (which they already do?) or do you want to make it illegal to post "bad things" on the internet?
    - hashkb 7y ago
      Let me start by saying that I am not running for elected office because it's a really hard job; even though I might believe I would do it with more integrity and rationality than the next person, I'm also smart enough to realize that everyone thinks that way and I'd probably end up within a standard deviation of average corruption. What I'm saying, seriously, is that we need a society-wide blameless post-mortem on social networks. It needs to be a slow, careful discussion, where all the stakeholders have their voices heard, and we decide what is good for us all. I don't know that we need them at all, I'm not sure they provide ANY value to ANYONE, but as a non-user I'd of course be open to persuasion by current victims. Erm, users. In the meantime, it should be illegal to operate a social network. Nobody needs to go to prison yet, even though I think it's horrifyingly clear at this point that the ethics are out the window. One obvious one: advertising and public conversation must always be separated; the same way no public schoolteacher may read scripture in class, it should be illegal for an internet service that hosts public conversation (e.g. Twitter) to allow sponsored content. We should also establish guidelines for addiction. After cigarettes, drugs, sugar, etc; we should as a society be prepared to understand the various forms that addictive products can take and regulate them aggressively before they become serious problems. UX patterns like infinite scroll, pull-to-refresh, and push notifications are particularly suspect. I also think we should close the apparent loophole in COPPA that allows parents to post photos of their children on social networks. > that they have to hire moderators (which they already do?) or do you want to make it illegal to post "bad things" on the internet? I tend to come down on the "free speech" side of these issues as much as possible; I would prefer unregulated public fora that (perhaps by requiring identity verification) encouraged good-faith participation in substantial discussions. I think if you are just fooling around, maybe you should head down to the bar and get drunk with your friends and do your shit talking there.
m463 7y ago
Stuff like this should be brought in-house. You can't outsource responsibility. Engineering and AI/pattern matching should be in the loop as a key ingredient so that a terrible image/video won't have to be viewed by human eyes at all, or worst case just once.
- laminar_flow 7y ago
  I would agree. Having to watch some of this stuff is a great incentive to make the AI better.
deegles 7y ago
I think all companies with user generated content should require every employee to moderate content for some amount of time. 1 hour per quarter would be plenty.
krick 7y ago
> I understand the arguments about supply and demand of labour, but I'd have more respect for Facebook if they demonstrated awareness of this issue. It is a very politically correct statement, which, as it usually happens with politically correct statements, really says nothing. I mean, what exactly do you propose they should do about it? I am the last person to defend Facebook, but, seriously, they are running a business. That is, essentially, finding a way to do/own/produce some stuff while spending less on it than you receive for it. And cost management is all about supply and demand. After all, it's not like Facebook wants to pay engineers $150k/year and for them to be happy. It needs good engineers and the only way to get them is to keep them happy, or else they stop coming. Engineers are not really special, they are not even exactly more necessary than cleaners, and loaders, and content moderators. The only difference is, demand for them is higher than supply. And that would be tragic in a sense, if everyone would be randomly assigned a role and if you draw an "engineer" you are lucky, but if you draw a "content moderator" you are not. Or if we were talking about some factory in some giant village in Vietnam, where no matter how smart you are and no matter what you do, you'll be working much harder job in conditions far, far worse than described for $500/year and you really don't have any choice. But we are not. No, this is Florida, $30K/year and these are people choosing the job of content moderator over a job of an engineer, either because they don't want to or cannot do otherwise. They while many of them may have stories that can be seen as tragic (because who hasn't, really?), they are no more victims than anyone else born into living a life on the Earth. And thus, hardly entitled to anything. Let me be clear: I really hate Facebook and I find companies with a modus operandi like Congizant's distasteful to say the least. And I can see how Cognizant can be held accountable for giving somebody a false hope, luring into doing the job while promising something else. But what exactly do you want for Facebook to do? They want a service as cheap as possible, they get asked a given price, they agree. That's it. It won't save their reputation after an article like this (because, well, public relations...), but it seems just fair.
GoodJokes 7y ago
Are you telling companies exploit labor? Are you telling me meritocracy is a joke? Are you telling me a ton of software devs get paid WAY too much and aren’t that “special.” I think we have hit the nail on the headddddd
nswest23 7y ago
for the record the article states that cognizant, the contractor that FB deals w/for content moderation, does provide health benefits: > Cognizant also offers a 24/7 hotline, full healthcare benefits, and other wellness programs.

baud147258 7y ago

> cheap in terms of reputation risk Also in terms of the reputation of Facebook: any issue with moderation failures and Facebook can easily says that it's the contractor's fault

RegBarclay 7y ago

I thought the same thing -- that once content is moderated, it should be trivial to block exactly the same content from being posted and moderated again. What a waste.

Nasrudith 7y ago
The problem is actual use and context sensitivity. Since if they let historic signifiant images get flat banned they also look like big bad idiots trying to memory hole history for blocking say death camp images while still having to ban asshats who post it and say "Not enough." Neither the text nor the image themselves would be ban worthy but in combination it is boot-worthy. Extrapolating has Scunthorpe problem issues.
LinuxBender 7y ago
The content may get modified quite a bit if censored though. A good example was the ChristChurch video. 4chan members modified it so many times, there are probably 4k-5k different iterations of it. NZ made a big deal about blocking it and arresting people for viewing it, making the demand for it significantly higher. There would have to be some really efficient and highly optimized ML to group all the variations together with a common label. AFAIK this does not exist for videos yet. The only partial mitigation that comes to mind would be if a small circle of your friends had to "approve" your video before people outside your circle could see it. That would put them on the hook for sharing bad things too. That might even reduce the psychological load on the moderators.

pawelmurias 7y ago

It's knowledge workers vs unskilled labour.

McWobbleston 7y ago
It's also people not being paid nearly the value they produce for an organization. Everything is always a race to the bottom for us
eswat 7y ago
I was thinking about this more but you’re right. I doubt there’s any acquired knowledge needed to know what’s repugnant to the human mind.
- cldellow 7y ago
  I don't have first-hand experience with the job, but I've read a few articles about these sort of teams, as they're widely deployed by all the large Internet companies. In my view, it's knowledge work. Sure, it's knowledge work that almost anyone can be trained to do, but there's a skill set required. For example, moderators need to make judgment calls that distinguish between offensive uses and artistic/journalistic/parody uses. There are many cultures and in-groups on Earth, which makes this harder than it might seem at first blush. Not all nudity is obscenity, not all hate speech is blatant.
  - qudat 7y ago
    > For example, moderators need to make judgment calls that distinguish between offensive uses and artistic/journalistic/parody uses. There are many cultures and in-groups on Earth, which makes this harder than it might seem at first blush. Not all nudity is obscenity, not all hate speech is blatant. In these cases I'm not sure it really matters. Over-moderation is going to win overall and I don't think moderators are going to get into that much trouble for being slightly over-zealous. I could be wrong but I feel like after a week of being on the job you'd get the hang of it.
    - 7y ago

anon029102 7y ago

> Can anyone at Facebook speak to whether engineers are aware of the working conditions of moderators, and agitate to improve their lot? They are passively aware but mostly don't care. The engineers who express frustration are in the vast minority. Most engineers are just super happy about their own situations, very focused on PSCs, and try to keep to their own stuff. Any internal frustrations are channelled towards appropriate feedback channels where the anger is watered down.

johnkpaul 7y ago
PSCs?
- tristanperry 7y ago
  From a quick Google, "Performance Summary Cycle" - presumably regular reviews and bonuses.
- deleted 7y ago
  [deleted]
plankers 7y ago
Nobody's making waves when everybody is focused on their own struggle. Gamifying for a docile workforce.

Miner49er 7y ago

Facebook won't do anything to help these moderaters unless the cost of the negative PR exceeds the cost of doing something. Engineers at FB could help achieve this by causing agitation, for sure. Maybe the best solution is for the moderaters to unionize.

plankers 7y ago
If you can think of a practical way to unionize workers on a 1-2 year contract with a high turnover rate, I'd love to hear it. The deck is stacked against employee organization. If I were conspiracy-minded, I might even say this was by design.
- Barrin92 7y ago
  >If you can think of a practical way to unionize workers on a 1-2 year contract with a high turnover rate, I'd love to hear it. Adopt the Danish model, make it mandatory for companies with workforces over a certain size to at least vote on unionisation. Will shift the default from not being unionised to being unionised except for cases were the workforce is actively convinced its detrimental.
  - plankers 7y ago
    I love that idea! But I have a hard time seeing a law like that being passed in the U.S., even in liberal California. Side thought, things like this help me understand why the Danes are so darn happy.
shujito 7y ago
...which it's unlikely to happen because of the NDA's they have to sign. If there's something wrong with this reasoning, please elaborate.
- vageli 7y ago
  > ...which it's unlikely to happen because of the NDA's they have to sign. > If there's something wrong with this reasoning, please elaborate. I would be utterly surprised if an NDA could be used to prevent an employee from exercising their right to organize (in the US at least). Just because you sign something does not make it legally enforceable.

Shivetya 7y ago

anyone who has done moderation for any active forum can tell you how nightmarish it can be. the more popular the platform the more problems you can have. anecdotal, friend who streams on twitch also mods for a few streamers and they even have issues when a stream is in subscriber mode only. simply because anonymity and distance from those being affected empower people to do bad things edit : I am really surprised there is/are not companies which would spring up to provide these services seeing how most of these activities are required by law. however before bemoaning what they are paid, just go look at your local 911 operators who are government employees. just because we think it should be paid more doesn't mean others do.

kevingadd 7y ago
Anonymity is definitely empowerment (especially because Twitch has no effective ban system, so bad actors will create dozens of accounts and keep coming back), but even people acting under their real names will do horrible things. We've seen this with real names on FB/Google+. It seems like there are no easy solutions for this and it's really frustrating: Societal norms are just completely shredded right now.
- nvr219 7y ago
  > but even people acting under their real names will do horrible things. We've seen this with real names on FB/Google+. And on twitter. Their most famous user uses his real name.

Cthulhu_ 7y ago

Re: salaries, don't forget that there's also salaries working on the content moderation problem - taking the huge datasets that the content moderators are reviewing and applying machine learning and content ID and I have no clue what else more to automatically filter out a lot of the content, ideally with such a high certainty rate that the moderators don't see it in the first place. I want to believe the content moderators only end up seeing the 1% of content that is dubious, or brand new content (not published and hashed earlier). I don't understand in that regard why the one video mentioned in the article (about the iguana) wasn't marked as "ok" in the systems, or marked as "on hold for law enforcement" so the moderators didn't have to see it again.

ralusek 7y ago

I don't understand how this isn't a problem perfectly solved by the market. If it's a shit job that isn't worth the pay, people won't do it. If it's an okay job that isn't really worth the pay, they will have high turnover. High turnover has a cost, maybe it's worth it to them. If the turnover is so high or the job is so undesirable, maybe they'll axe the role. If the role is so important to the business that it can't be axed, but people aren't taking the jobs, they'll have to pay more for it. If they can't pay more for it because it cuts too much into profitability, but it's necessary for the company to operate, then they are no longer a viable company in the market. Or they'll have to innovate or adapt. All possibilities are fully encapsulated in very simple dynamics.

sandworm101 7y ago
>> If it's a shit job that isn't worth the pay, people won't do it. That assumes perfect knowledge. As mentioned in the article, these moderators often do not understand what they are getting into until long after they have committed to the job. Once you have quit a previous job, moved, setup your new life, only then do you realize your new "brand consultant" job means watching kittens die all day. Jobs like this also open persons to realities that we normal keep tucked away deep in our brains. We all know that cops don't investigate most reported crimes, but that doesn't come up very often for us. A facebook moderator sees the crimes, sees the lack of response by the police or anyone else, every minute. Repeatedly, every minute. Nobody can appreciate what this does to ones' mental state until you have been there.
- debacle 7y ago
  Nobody moves for $30k a year.
  - skyyler 7y ago
    Are you serious? I know people that have moved for less. Making absolute statements like that doesn't do much for anyone.
  - sandworm101 7y ago

jfindley 7y ago

Given the description of the videos in the article, I'm not sure that merely paying the moderators more would solve it. I think for most people that I'd want to be moderating content (e.g. not sociopaths) making them richer isn't going to do anything to deal with the real reason this is worse than a call centre - the content itself. I'd rather see facebook put more effort into protecting moderators from the content they're viewing. I realise this is a non-trivial problem, but here's a few ideas of things they could do which may help: * Once a video is confirmed bad, fingerprint it and block future uploads. * Provide colour-inversion buttons to help reduce visual impact * Rotate people between areas, so they aren't looking at the same class of bad content constantly * Use ML training to detect which content is likely to be the most severe and for this content, reduce minimum viewing interval to ~ 5 seconds, and ensure that a single individual doesn't get too many of these. * Flag accounts who post bad content too often, and shadow-ban them so their posts are not visible by anyone else (this could go in stages, starting off by blocking re-shares of their posts or something)

quadcore 7y ago
Provide colour-inversion buttons to help reduce visual impact I was thinking also about blurring the picture/video.
- afasadaf 7y ago
  They could have levels, and use moderator consensus to remove the pictures 1. Blurred+Mosaic 2. Contour lines only 3. Windowed/Striped (only showing like 10-25% of the photo) If 2/3 of the moderators above agree that it's a bad photo, it moves onto stage 4: 4. Color filters - Break the photo down into k-means color clusters, moderators choose which colors to see. They can see 100% of the colors, or one, or two, or however many they need. If #4 agrees, the photo goes into the round folder. No need for anyone to ever see the whole photo. It'll be pretty obvious that something was violating the TOS without anyone being able identify anything specific about the photo. We're not talking about 'beyond the shadow of a doubt' here, we're talking about 'yeah, that's probably not ok.'
zanmat0 7y ago

p1esk 7y ago

How about training a classifier to detect and remove potentially bad stuff, then let the users who uploaded it argue that the classifier made a mistake? Only in those cases have human moderators look at it?

Sharlin 7y ago
They already have classifiers, of course. It's not like the moderators go through everything posted to FB–you'd need millions of people to do that. What the mods actually do, presumably, is review content reported by users.
- p1esk 7y ago
  Then perhaps they need better classifiers? I don’t remember seeing any papers from FB about that, so it does not seem like a priority for them. Or just make the existing bad classifiers more aggressive and shift the moderator’s effort to reviewing good content.

sandworm101 7y ago

>> If kiddie porn, gore and animal cruelty flooded the network, it would cease to be a destination visited by people that advertisers will pay to reach. Or the opposite. People flock to the internet to see this stuff, the material that isn't on TV or even dedicated commercial websites. They get to view it in the privacy of their homes, a secret between them an their trusted "friends" on facebook. Facebook knows that if it really cracks down, if cops bust down doors over this stuff, that people will go elsewhere to share horrible material. So facebook does a poor job of moderation, one just good enough to avoid new regulation but not so good as to actually make a difference. Advertisers care about eyeballs. They may claim to not want to be associated with X or Y content, but in reality they would rather be in front of an audience than not.

lapnitnelav 7y ago
Advertisers do care a lot about where their ads are displayed. There's a whole section of adtech dedicated to brand safety. I believe it's over the top[1] but it's something marketing folks do care about. Also the good old pop up ads were created for that reason, aeons ago. [[1] https://www.campaignlive.co.uk/article/advertisers-embrace-news-not-shy-away-brand-safety-paranoia/1458646 https://www.campaignlive.co.uk/article/advertisers-embrace-n...

geggam 7y ago

150k a year in Silicon Valley is similar to 30k a year in the midwest... Where are the moderators living ?

kixiQu 7y ago
This is very much an answered-in-the-article question.
unicornfinder 7y ago
Poland

scarface74 7y ago

$30k/year, no benefits, likely going to acquire mental illnesses Why do you think they don’t get benefits? Facebook contracts out to another company. The company they contract with still is probably providing some benefits

cldellow 7y ago
That's true, the article says Cognizant offers "a 24/7 hotline, full healthcare benefits, and other wellness programs". But at the same time, it quotes the on-site counsellor as having said "I don’t really know how to help you guys" when an employee went to him for help. This leads me to believe that there is a qualitative difference in the calibre of benefits Cognizant provides versus those that Facebook provides. Further, since the job has high turnover and can be expected to put a high toll on an employee's wellbeing, it's also important to me to think about what happens to their benefits if/when they quit. In the US model of employer-linked healthcare, this tends to result in losing your benefits when you need them most. By contrast, other organizations that do similar moderation jobs seem to offer psychological health care treatment even after the employee leaves the company. See https://www.theguardian.com/news/2017/may/25/facebook-moderator-underpaid-overburdened-extreme-content https://www.theguardian.com/news/2017/may/25/facebook-modera... for more info.

Bakary 7y ago

>This is a guess - can anyone at Facebook speak to whether engineers are aware of the working conditions of moderators, and agitate to improve their lot? If you go work at Facebook as a engineer chances are you aren't too bothered by ethical questions

wrongdonf 7y ago

It says right in the article that the content moderators have full medical benefits

cldellow 7y ago
Sorry, yes, I could have expanded on that. You're right, they provide health benefits, in theory. My experience with the US health care system is that there is a wide gulf in health benefits. When I worked for Microsoft in the late 2000s(as an employee, not as "dash trash", as contractors were called), I paid some very small premium like $10-20/month for a plan with 0 deductible, 0 copay, and pretty much every doctor was in network and covered at 100%. When I dislocated my shoulder, some orthopedic surgeon who used to consult for an NBA team looked at it for me. I understand that that was considered quite good insurance and that most people in the USA don't get that level of care, much less at that price. If you read Glassdoor reviews of Cognizant, people complain about the insurance offerings available to employees. https://www.homebaseiowa.gov/sites/homebaseiowa.gov/files/document_uploads/2017%20Cognizant%20US%20Benefits%20Overview.pdf https://www.homebaseiowa.gov/sites/homebaseiowa.gov/files/do... looks like it might be what they offered 2 years ago. It doesn't look very good to me. If I was earning $15/hour, I think I'd pretty much buy the cheapest plan and only use it if I was actively bleeding to death. I'll also link to https://www.theguardian.com/news/2017/may/25/facebook-moderator-underpaid-overburdened-extreme-content https://www.theguardian.com/news/2017/may/25/facebook-modera... again - it describes how other companies that moderate distressing content approach this. It includes the company paying for psychological treatment, including after the employee leaves the company.
- pravda 7y ago
  >I understand that that was considered quite good insurance and that most people in the USA don't get that level of care, much less at that price. AFAIK, Microsoft 'self-insures', in the sense that they simply pay the medical bills for their employees.

meerita 7y ago

"If kiddie porn, gore and animal cruelty flooded the network, it would cease to be a destination visited by people that advertisers will pay to reach." I bet this can be done with ML and IA instead human power. Yet, FB uses their human cattle to censor conservative voices.

mythrowaway1124 7y ago
Good luck! There's at least a billion dollars in it for you if you figure out how to do this.
kartan 7y ago
> I bet this can be done with ML and IA instead human power. IA and ML work in bulk. But they misclassify things that a human easily can discern. Think Tesla VS Google. Google wants a self-driving car, and that is not working yet. Tesla wants a car that helps you to drive and that is good enough. It's order of magnitude easier to be almost always correct that to be correct all the time. And you only need one kiddie porn video in your Facebook feed to start to worry about Facebook.
- meerita 7y ago
  It is matter of time this will happen. Even with self-driving cars. Using people to do this is just killing people. Think all the mental effort that has to be done to censor. Humans aren't that perfect and cannot oversee the entire FB daily activity.

nonwifehaver3 7y ago

How can kiddie porn, gore and animal cruelty "flood the network" for people using Facebook? Don't you normally just see things from your friends, ads, and groups that you are a part of? If they post that, either call the cops, block them on Facebook, don't hang out with them irl, or leave the group as appropriate. Problem solved? I haven't used Facebook in 5+ years so maybe this is no longer accurate.

lazyasciiart 7y ago
As you may have guessed from the downvotes, this is not even close to accurate. It should have been possible for you to realize this was inaccurate simply from how trivial the problem looked to you in comparison to how much effort is being spent on it.
- nonwifehaver3 7y ago
  I genuinely don't understand how an average Facebook user would encounter videos like those described in the article. Is it some recommender system gone awry?
  - lazyasciiart 7y ago
    As far as I know, mostly because people post them as comments on public or widely shared items, along with "get free sunglasses" and every other kind of spam. There are tons of these public posts from pages that do news, memes, politics, trolling, regular advertising, whatever.

hashkb 7y ago

> Facebook is enormously valuable. They made something like $15B in net income in the last four quarters. Their profit is not proof they are valuable, because they are willfully abdicating responsibility for the damage they cause, and, further, involving themselves in government debates over their own regulation. Advertising businesses are inherently suspect, and when one is addictive and potentially enabling election fraud etc, we need to be extra careful in declaring its value. Anyone participating in the debate should have to disclose if they are a daily user or if their company advertises on any social network. (I'm clean) Edit: my opinion of course is that FB, Insta, etc (any advertising product with push notifications and/or infinite scroll) ought to be illegal. In 50 or 100 years we'll all recognize that social networks are like internet cigarettes.

mehrdadn 7y ago
> my opinion of course is that FB, Insta, etc (any advertising product with push notifications and/or infinite scroll) ought to be illegal. In 50 or 100 years we'll all recognize that social networks are like internet cigarettes. Well cigarettes are still not illegal, so we have some ways to go...
- allthecybers 7y ago
  This was an earlier comment that disappeared. Frankly I ask myself consistently "what net value does Facebook add to the world?" other than being a 100% optional vice.

ummwhat 7y ago

I've talked about this somewhere else. What you're seeing is just a manifestation of what I call "the fundamental problem of user created content." Said problem is that warehousing and distributing content scales insanely well but curation does not. Until we have strong AI, curation is a manual process. Moderators just aren't efficient enough at processing content for their output to pay for a full time salary. You can cut costs by making end users into moderators (the Reddit model) but results may vary. This problem applies to other forms of sites and content as well. The app store gets hundreds of submissions in the game category per day. Hosting and distributing that content is easy. Only a tiny fraction of those games are going to be played and rated enough times to show up in a recommendation engine. The bulk of the incoming content stream isn't being matched to interested people at all (sorting by interesting is the same curation issue as filtering by offensive). Literally every content platform is going to have some problem similar to Facebook. We can blame Facebook, but the reality is no one has a good solution. Not even me.

hashkb 7y ago
> We can blame Facebook, but the reality is no one has a good solution. We can take a step backwards and eliminate Facebook, Insta, etc from our societies and enact legislation that will govern the second generation of social networks.
- Jedi72 7y ago
  What legislation? Specifically? What do you want to make law, that they have to hire moderators (which they already do?) or do you want to make it illegal to post "bad things" on the internet?
  - hashkb 7y ago
    Let me start by saying that I am not running for elected office because it's a really hard job; even though I might believe I would do it with more integrity and rationality than the next person, I'm also smart enough to realize that everyone thinks that way and I'd probably end up within a standard deviation of average corruption. What I'm saying, seriously, is that we need a society-wide blameless post-mortem on social networks. It needs to be a slow, careful discussion, where all the stakeholders have their voices heard, and we decide what is good for us all. I don't know that we need them at all, I'm not sure they provide ANY value to ANYONE, but as a non-user I'd of course be open to persuasion by current victims. Erm, users. In the meantime, it should be illegal to operate a social network. Nobody needs to go to prison yet, even though I think it's horrifyingly clear at this point that the ethics are out the window. One obvious one: advertising and public conversation must always be separated; the same way no public schoolteacher may read scripture in class, it should be illegal for an internet service that hosts public conversation (e.g. Twitter) to allow sponsored content. We should also establish guidelines for addiction. After cigarettes, drugs, sugar, etc; we should as a society be prepared to understand the various forms that addictive products can take and regulate them aggressively before they become serious problems. UX patterns like infinite scroll, pull-to-refresh, and push notifications are particularly suspect. I also think we should close the apparent loophole in COPPA that allows parents to post photos of their children on social networks. > that they have to hire moderators (which they already do?) or do you want to make it illegal to post "bad things" on the internet? I tend to come down on the "free speech" side of these issues as much as possible; I would prefer unregulated public fora that (perhaps by requiring identity verification) encouraged good-faith participation in substantial discussions. I think if you are just fooling around, maybe you should head down to the bar and get drunk with your friends and do your shit talking there.

m463 7y ago

Stuff like this should be brought in-house. You can't outsource responsibility. Engineering and AI/pattern matching should be in the loop as a key ingredient so that a terrible image/video won't have to be viewed by human eyes at all, or worst case just once.

laminar_flow 7y ago
I would agree. Having to watch some of this stuff is a great incentive to make the AI better.

deegles 7y ago

I think all companies with user generated content should require every employee to moderate content for some amount of time. 1 hour per quarter would be plenty.

krick 7y ago

> I understand the arguments about supply and demand of labour, but I'd have more respect for Facebook if they demonstrated awareness of this issue. It is a very politically correct statement, which, as it usually happens with politically correct statements, really says nothing. I mean, what exactly do you propose they should do about it? I am the last person to defend Facebook, but, seriously, they are running a business. That is, essentially, finding a way to do/own/produce some stuff while spending less on it than you receive for it. And cost management is all about supply and demand. After all, it's not like Facebook wants to pay engineers $150k/year and for them to be happy. It needs good engineers and the only way to get them is to keep them happy, or else they stop coming. Engineers are not really special, they are not even exactly more necessary than cleaners, and loaders, and content moderators. The only difference is, demand for them is higher than supply. And that would be tragic in a sense, if everyone would be randomly assigned a role and if you draw an "engineer" you are lucky, but if you draw a "content moderator" you are not. Or if we were talking about some factory in some giant village in Vietnam, where no matter how smart you are and no matter what you do, you'll be working much harder job in conditions far, far worse than described for $500/year and you really don't have any choice. But we are not. No, this is Florida, $30K/year and these are people choosing the job of content moderator over a job of an engineer, either because they don't want to or cannot do otherwise. They while many of them may have stories that can be seen as tragic (because who hasn't, really?), they are no more victims than anyone else born into living a life on the Earth. And thus, hardly entitled to anything. Let me be clear: I really hate Facebook and I find companies with a modus operandi like Congizant's distasteful to say the least. And I can see how Cognizant can be held accountable for giving somebody a false hope, luring into doing the job while promising something else. But what exactly do you want for Facebook to do? They want a service as cheap as possible, they get asked a given price, they agree. That's it. It won't save their reputation after an article like this (because, well, public relations...), but it seems just fair.

GoodJokes 7y ago

Are you telling companies exploit labor? Are you telling me meritocracy is a joke? Are you telling me a ton of software devs get paid WAY too much and aren’t that “special.” I think we have hit the nail on the headddddd

nswest23 7y ago

for the record the article states that cognizant, the contractor that FB deals w/for content moderation, does provide health benefits: > Cognizant also offers a 24/7 hotline, full healthcare benefits, and other wellness programs.