11 ms·
A Facebook content moderator’s resignation note
- bellyfullofbac 6y agoIt's worth reading but it doesn't seem very "blistering", it's very calmly written. But hey if you work for BuzzFeed...
- christkv 6y agoThis seems like a set of very reasonable demands.
- thex10 6y agoI was struck by this too. Very reasonable, well explained and decently argued!
- topoftheforts 6y agoIt looks like moderation is one of the biggest problem we're facing in tech, in different ways: - fake reviews which make the ratings useless - scam apps that riddle the various app stores - abusive / illegal content on platforms that can't afford better moderation - moderators who struggle with mental health issues, as per this tweet - footballers receiving racist comments on their socials Is there space for a startup to solve this problem with AI? And should it be a B2B or B2C scenario, as in working with platforms to moderate in the backend or being a user app / extension that does it in the frontend? For example, Fakespot works great with Amazon but it's an extension you have to install, so it's not available on Safari / on Mobile. So it's still quite unknown and I'm not sure how much it can grow
- AbrahamParangi 6y agoThe essential problem is that excellent moderation can be counter to the platform incentives. Fake reviews can be bad for the customer, but potentially neutral or good for the platform. Same with fake engagement. Doing a really good job with moderation will materially and negatively impact KPIs in the short term.
- Juliate 6y agoWhy would AI solve this problem in the first place? We barely understand how biased it is, and even if/why it's necessarily biased, not by design, but by nature. As in, biased, because the reference you use is subjective to the time and society and culture it is built in. Which is different from the whole context 30 years ago, and will be different in 30 years. Why, also, would tech solve a problem tech did not create, but amplified?
- motohagiography 6y agoConsider the economics a bit. The point of moderation is to sustain the business model of the platforms that need it most. It's necessarily going to be a cost centre that will only get optimized down to where it's a kind of impoverished hellscape to work in. The reason it's a cost centre is because it's trivial to stand up a new platform with a higher bar to entry that precludes almost all evil content, with the limiting factor that it won't have exposure to the total size of the legacy platforms that have no bar to entry. So for FB, better moderation is not a differentiator with an ROI, it's a risk management cost. This puts a ceiling on what they'll pay for it. The ideal solution for an FB-like platform that is already committed to open doors user policy means that the cost model/value prop is that the solution must be able to show it's doing something above all. If you sold them an AI solution that "just worked" for 98% of material and had a tiny relative cost, FB's real risk from the 2% that still got through doesn't change, because when FB gets asked by regulators and others what they are doing about it, your incredibly cost-effective AI solution just shows FB isn't committing resources to the problem. When FB can show rooms full of workers at screens on 24/7 shifts responding to the problem (at some tiny foreign hourly wage that their employer with the moderation contract is probably making a 300%+ margin on) that is the solution. These peoples' job is to essentially be psychological human sacrifices to regulators. This also describes most of the economics of the managed security services SOC business model. If you want to get into horror, find out if there is a correlation between the production of evil content and the regional presence of a moderation subcontractor. It's like anti-virus and security companies releasing viruses and exploits, but next level worse. If there is $100m/year market for moderation services, that literally creates the perverse incentive to ensure a flow of evil content. Could there be a CrowdStrike of content moderation? Maybe. Especially in markets like China, Singapore, and increasingly Britain, where policing content could be creating the same regulatory environment that hackers created for security.
- Traubenfuchs 6y agoI'd wager there is a lot of untapped potential in hiring... "mentally divergent" people that enjoy watching this material. As long as they properly tag / rate / perform, there should be no problem with them.
- samstave 6y agoThats an utterly disgusting thought. Fuck that thought.
- sodality2 6y agoI'd prefer to not get pedos off with my hiring policies. Have you considered this may encourage them to act again?
- MattGaiser 6y agoIndeed, although "enjoy" might be too far. Reddit used to have subreddits dedicated to gore and watching people die. There are serial killer groupies out there. An acquaintance of mine works in the judicial system and has no problem screening violent evidence. People who are not horrified in the same way by things clearly exist. For jobs where normal people tend to develop psychological problems but that need doing, seems like we should be looking for people who are less likely to have those issues.
- dcl 6y agomaybe try screening for people that are less likely to be affected by it first? I say try because it probably isn't going to be easy.
- diamondhandle 6y agoPeople who comment on these moderator issues always overlook the huge huge mental red flag that these moderators always discuss: images of children being molested and tortured. Endless images of young children. Being sexually assaulted, penetrated, made to do degrading things. Think about how that statement makes you feel, and then imagine you’ve been tricked into looking at this stuff for money as a “content moderator.” It’s a form of cruel economic punishment. Facebook wouldn’t have these problems if they weren’t obsessed with acquiring as many users as possible. They are the ones who should be forced to pay for the long term mental health of these workers. California ballot initiative, anyone?
- commandlinefan 6y agoI'm shocked by the implication that there's so much of it - that there's enough to damage somebody's mental health. I'm sure numbers are hard to come by, but I'd have guessed that there wouldn't be more than a couple of these cases per month - the problem can't be that widespread, or you'd assume it would spill over into other areas as well.
- mbg721 6y agoThere doesn't have to be that much of it created, if it can be copied for free, and someone is willing to automate posting it. Cynical-me totally believes there are enough people doing that to traumatize a human content-sorter.
- carlhjerpe 6y agoYouTubes contentID system is quite good at detecting just a couple seconds of sound to determine if something is copyrighted or not. It should be quite easy to match copies of content and just get rid of it straight away, meaning they don't have to see duplicates. I would assume they already do this. Which leads me to believe there's actually that much content being produced, I imagine most of it is not produced in the West though.
- 6y ago
- whalesalad 6y agoSome things are not meant to operate at scale. Putting your mouth on the firehose of real human nature can’t be healthy for anyone.
- quenix 6y agoI don't know that it's accurate to describe beheadings and child porn as "real human nature". The scum of nature, perhaps.
- whalesalad 6y agoWho decides where the line is drawn between scum and not scum? Alas, that is why this is human nature. It is all encompassing, you cannot pick and choose. None of us are perfect.
- quacked 6y agoYou live in the ivory tower of civilization. Do you really think you're that different from your ancestors? It's society that has evolved, not the component human parts of it.
- mrweasel 6y agoThe issue brought up by this moderator seems to match what we've heard and read from others, which puts Facebook, and sites like it, in an unfortunate position: Content from a site like Facebook cannot be moderated without causing harm to human moderators, and A.I. is not yet able to do the job. That's a massive issue as lawmakers are pushing for more moderation. It's my belief that Facebook, and others, have sold our politicians on the idea that moderation can be done, at scale, with AI. Those of us who followed along knows that the human moderators have complained that at least Facebooks A.I. simply isn't up to the task. I don't think Facebook and similar sites can exist, not at that scale, without massive human cost.
- deleted 6y ago[deleted]
- mschuster91 6y ago> and A.I. is not yet able to do the job. AI won't be able to do the job until we manage to digitize brains. Stuff like sarcasm, irony, fair use or "dog whistling" (e.g. using "globalists" as a replacement for "jews") is something current AI's can't grasp at all, and even a digitized brain will struggle with these concepts - not to mention the ethical concerns that would come with digitizing a brain. Is it morally acceptable to torture a virtual brain that, in contrast to a human, can't even leave? To reset it to default state when it's too damaged?
- lalaithion 6y agoSee: https://moultano.wordpress.com/2019/10/02/why-do-companies-with-huge-resources-still-have-terrible-moderation/ https://moultano.wordpress.com/2019/10/02/why-do-companies-w...
- ceejayoz 6y agoHell, humans don't even understand sarcasm, irony, fair use, or dog whistling much of the time. Try submitting a dog whistle to Facebook's human moderators and it'll almost certainly get declined as within their standards. As a concrete example: "Congressman Falls for The Onion's Planned Parenthood 'Abortionplex' Story" https://www.theatlantic.com/national/archive/2012/02/congressman-falls-months-old-onion-story-about-planned-parenthood-abortionplex/332189/ https://www.theatlantic.com/national/archive/2012/02/congres...
- fighterpilot 6y agoThey should try showing the moderator a blurred version of the image first, and allow the moderator to slowly unblur it if they need further confirmation. Or use ML to auto blur specific parts that are likely to be gore etc, and allow moderator discretion to remove the occlusion. I bet some pretty basic tricks like this would help a lot. A high resolution image of all the gore (or whatever) inside it is probably not needed to come to a decision in most cases.
- q3k 6y agoDo you really think that this sort of 'basic tricks' is something that didn't occur to the thousands of managers, moderators and engineers working on this?
- ltbarcly3 6y agoHave you ever had a job? Its quite common for companies to store all of their data in excel files on a share drive with no backups. Why would companies do that when there are so many better ideas? The letter makes it clear that these contractors have basically no interaction with FB proper, and are treated as disposable. Is that the sort of environment that leads to tools improved in creative ways just to make it less painful to work there?
- mschuster91 6y agoFront line moderators are seen, treated and paid like expendable cannon fodder. No one cares about them, because if they drop out there's a steady stream of replacements for them. It's similar to Amazon's warehouse pickers, callcenter agents and farm laborers. The less skills your job requires the more abuse employers can get away with.
- ltbarcly3 6y agoI would add that Amazon is extremely innovative in ways to make warehouse employees more productive. They are physically limited in how any employees they can fit into a warehouse, and what geographic region those employees are hired from, so they have to make sure these employees are productive because they can't just hire twice as many to get through peaks if they aren't productive enough. At the same time, these employees are treated poorly and are peeing in bottles while standing at their station because Amazon didn't account for the physical need to urinate. Facebook has none of these restrictions, and these employees don't even work for Amazon, and they are behaving the way you would expect a company to behave with these incentives.
- noen 6y agoIt remains a major issue without end because there are no punitive consequences. Facebook could out, shame, and be a significant aid to prosecution of the individuals who post illegal and obscene content. But there are no regulations compelling them to do so. There's no business incentive to do it themselves. And maybe most important, it's very clear that Facebook has no ethics in their own internal culture to take a stand on principle.
- intended 6y agoHeh - there can’t be punitive consequences, since FB has offloaded the manpower intensive work to outsourcing firms.
- deleted 6y ago[deleted]
- mouzogu 6y agoIt's really sickening how much nsfl content is being uploaded and shared on facebook. Who are these people and what is wrong with them.
- whalesalad 6y agoHave you met a human before?
- deleted 6y ago[deleted]
- z77dj3kl 6y agoThe reason for the NDA is obvious: Facebook doesn't want you to know that there are people doing this job. Facebook doesn't want you to even realize that this job needs doing, they want you to think that the platform is just clean and nice without any of this icky stuff. It's part of what makes Facebook enjoyable for end users, and that they've managed to do: keep the platform clean of most awful things, but it comes at a heavy cost. People don't notice this cost, and that's by design.
- iamben 6y agoIt's Vegas, right (or at least the image it wants to project when I visit as a tourist)? Clean streets, clean casinos, trouble infrequent and quickly dealt with, nice smells, free drinks, happy faces. Everything to keep you putting money on the tables and in the slots -- because showing anything nasty is terrible for business.
- oldsklgdfth 6y agoIs it me or is the current state of the internet akin to physical society that lacks trust systems and law enforcement? In this analogy, are moderators of a given platform equivalent to mall cops/neighborhood watch?
- burlesona 6y agoI wonder how effective a law would be that simply said: "you can't outsource content moderation, and all employees, directors, etc. of any company that does moderation must perform at least X hours of moderation shifts per year." If the Facebook execs had to do a day of content moderation every quarter, I imagine that meaningful change would happen a lot sooner.
- icu 6y agoLet's face it, nothing will change until US lawmakers start forcing change. Unfortunately, US lawmakers by and large don't understand social media (CEO testimonies prove this), and nothing will really change for everyone outside the US until US lawmakers force changes (GDPR still isn't really adhered to).
- 838812052807016 6y agoI wonder how much of the disturbing content is spam from botnets or nefarious groups.