7 ms·

It's probably a bit controversial to say this, but haven't large sites had to deal with these problems for years by the time the likes of Facebook/Twitter/YouTu

by CM30 6y ago

It's probably a bit controversial to say this, but haven't large sites had to deal with these problems for years by the time the likes of Facebook/Twitter/YouTube/Instagram/whatever came around? It's obviously horrible doing this as a job and it's obviously a bit more common on Facebook than say, a large old school internet forum, but... at least in this case people are paid to do this. Reddit mods and old school forum mods have to deal with this stuff for free.

tmpz22 6y ago
Right but the difference is the sheer amount of money involved and the politicization that comes with that. If Facebook was worth 100m nobody would care. If facebook only has 10m users nobody would care. It's the scale that completely breaks the legal and political frameworks the our society leans on.
- wheelie_boy 6y ago
  I'd say another difference is the percentage of your attention is being devoted to this kind of thing. Presumably reddit or phpbb mods aren't spending 40 hr/wk looking at gore.
  - Nasrudith 6y ago
    Wouldn't that suggest then that the "ideal" way to do it is a very part time job then with high volume? Like say 5 hours a week. It would be a scheduling and logistical nightmare of course but it would give more "dispersion time" for trauma.
    - wheelie_boy 6y ago
      I'd be interested to see if the tooling around moderation could also be improved. For example, could the images start heavily blurred, with a circle of clarity that opens when you click. Or something like that, which would harm throughput, but be more humane.
      - prawn 6y ago
        This is a very interesting idea. It could reveal only a portion and see how often that was enough for the moderator to pass judgement.
    - asddubs 6y ago
      tbh I've been a moderator on various old school forums for over a decade, and I've never had to deal with anyone posting any gore at all. I think they're usually too tight knit and reputation based for anyone to feel like they could get away with it. it's not like people posting gore on facebook are doing it to mess with the moderators, that's just a side effect
- hinkley 6y ago
  From my time observing volunteer groups, it's also about leverage and force multipliers. I can go door to door asking people to change something, and dozens of my volunteers will feel useful but overwhelmed by how much work there is to do in a city of a million people. Or I can go to city council meetings, and get more leverage there. Or if I can convince one high profile citizen to participate, others will follow. Or I can get into contact with the mayor, state congressional members, the governor, federal congressional members. It's possible that if I convince the correct five people that they will take half of this burden off of my hands. You'd be crazy not to reach for a lever like that. They are a juicy target, and not just from a punitive standpoint. Of course when you are rebuffed repeatedly by a potential ally, then it's a very human (if questionable) response to seek out some sort of punishment.
  - ece 6y ago
    Do you know what community organizing is? How movements accomplish change? Everything does not have some easy optimization method, only in silicon valley does everyone think it does. edit: What I mean to say is, people have been organizing against FB, AMZN, and tech in general for the past few years now. The last is reactionary but opposition to FB and AMZN has meaningful academia, ngo/gov, and general support.
    - hinkley 6y ago
      I worked with a group that got 100 volunteers at each event who did the work while the regular members provided coordination and resources. I am trying to do something similar on my own now and learning in a very visceral way just how important spreading that load out is. But... that group wouldn't be able to do nearly what it has if it weren't for funding from foundations and the department of neighborhoods, and resources on permanent loan from another public sector organization. You might argue it couldn't exist at all. I think I would say that both are necessary but insufficient.
    - SaltyLemonZest 6y ago
      I think the point is that, if I as an activist am aiming to maximize the number of content moderators treated well, the highest leverage way to push for that is getting people mad at Facebook. There may be a preexisting system of unpaid content moderators who are supported even less, but that's much harder for anyone to address.
- ece 6y ago
  Yeah, monopolies can do that. edit: https://qz.com/1704143/the-antitrust-case-against-facebook/ https://qz.com/1704143/the-antitrust-case-against-facebook/
- whymauri 6y ago
  Another factor is the working conditions at Cognizant. The difference between Reddit moderators and Cognizant employees is that the former are allowed to have bathroom breaks and individual agency. The latter not only have to deal with fairly oppressive workplace policies, but they also rely on this work to live.
untog 6y ago
I think it's fair to say the sheer quantity has changed since Facebook first came around (FB predates smartphones for one!) I also think part of the objection here is around off-shoring. It's similar to other complaints about off-shored jobs: we're farming out unpleasant work to people that get paid less and have much less support (e.g. mental health).
- CM30 6y ago
  > I also think part of the objection here is around off-shoring. It's similar to other complaints about off-shored jobs: we're farming out unpleasant work to people that get paid less and have much less support (e.g. mental health). That's an interesting point, and has me wondering something. If a site like Facebook opened up a volunteer moderator scheme where people could sign up to filter out stuff that broke their rules, would it make things better or worse on a moral level? What about a US/Europe based agency that did this thing onshore as a contract deal?
  - untog 6y ago
    Facebook already hires moderators inside the US via agencies, and they have their own problems: https://www.theverge.com/2019/2/25/18229714/cognizant-facebook-content-moderator-interviews-trauma-working-conditions-arizona https://www.theverge.com/2019/2/25/18229714/cognizant-facebo... As for the volunteer side, it's a really interesting debate. Others in this thread have pointed out Reddit as an example of this in practise. I think the sheer amount of money that Facebook has means that relying on volunteers to (potentially) inflict emotional damage upon themselves wouldn't be a great look.
  - AdmiralGinge 6y ago
    I'd consider volunteering to moderate a particular geographical area, when I was a student Facebook constantly took down the student confessions page I eventually ended up helping to run for the vaguest reasons, while letting actual hate posters run amok. I know I would say that, but we would genuinely get taken down for jokes that wouldn't even make sense out of the tiny university I attended. It was immensely frustrating having to build up the userbase again and again because the moderators of Facebook just weren't/aren't terribly good at their jobs. I'd happily have taken on actual moderation duties rather than had to constantly appeal and get a blank, inhuman "declined" message. I think they tend to take an "if in doubt, remove it" approach which basically allows anyone who doesn't like a niche page for whatever reason to have it taken down through sheer attrition, if you report it enough it'll probably go down eventually even for stupid reasons. I actually ended up writing an app to cut Facebook out of our system altogether (it was kind of similar to Yik Yak with a much better moderation system and better anonymity protection - I really can't be arsed to deal with a GDPR lawsuit...) but we couldn't launch it in part because it took me ages to develop as I've since graduated and work as a full-time developer and because coronavirus meant our launch event got cancelled. I might take it live when students go back this year, but I'm concerned that with nothing much going on it'll fizzle and die because there's nothing interesting to post about. It'll all depend on how the Welsh government lift the coronavirus restrictions.
WrongThinkerNo5 6y ago
I suspect there is a certain interest group that has emerged and grown around this subject that has been elevating and promoting this issue in the public's mind. It would not even surprise me one bit that especially overseas contractors, because there are likely fewer penalties/prohibitions, are paying people to flood, e.g., fb with things like beheading videos in order to produce a cause for their services. It may be cynical, but like I said, it would not surprise me one bit.
rtkwe 6y ago
Very few forums had even a fraction of the number of members as Facebook has users and the amount of content is increased again. Also the type of content they're required to moderate has changed most forums before facebook were largely text and video is a big part of facebook. It's easy to see how video and photo content is more traumatizing than the largely text content of forums.
indigochill 6y ago
>at least in this case people are paid to do this. I've done commercial moderation before and I would sooner go jobless than go back to exposing myself to the kinds of things people have to when doing this professionally. It's the kind of things you wish there was a brain cleaning solution for, but there isn't so it's stuck there basically forever (though I suspect the alcoholism rate is higher among people who have to handle this kind of thing than in the general population). The commercial moderation team I was on actually ended up separating out a task force for handling particularly gnarly content so that people had a way of opting out of being exposed to it (in our case we were responding to user reports rather than randomly stumbling on things, so manually filtering reports was feasible), which I think is the responsible approach since there are plenty of other moderation tasks to handle and some people are more or less sensitive to this content.
- remcob 6y ago
  > wish there was a brain cleaning solution 'Drug-induced amnesia' to reduce long term memory generation is a thing and is sometimes used before traumatic surgical procedures. It needs to be taken in advance and is probably not very healthy. Plus I would have serious ethical concerns with jobs that would encourage that.
  - throwanem 6y ago
    I've had procedures where those drugs are used. They're benzodiazepines, and thus powerfully sedating. You are in no sense able to work while doped to the eyeballs with Versed.
  - mirimir 6y ago
    I've read that ~high-dose opioids after trauma reduces PTSD.
- stickfigure 6y ago
  This seems like something specific to the individual. Maybe this isn't the job for you, but there's probably enough people "made of different stuff" that could handle it? I feel like there's a subtext in this discussion that "nobody can handle it" or "you have to be a monster to work this job" but that's just not true. Just out of college in the mid-90s, I worked for a few months at a small local software company in the last throws of death. As a last grasp at survival, all the programmers switched to doing cold call sales. With phones. I would stare at mutilated children all day long rather than go back to cold calling. I was miserable and I hated every moment of every day until the company (happily, for me) closed up. Yet I know people who somehow think it's perfectly normal to dial random numbers and try to pitch them something. They're made of different stuff. I've talked to people who make the same claim about computer programming. They'd rather starve. Facebook has 10,000 moderators. I'm pretty sure there are at least 10k people in the world who are psychologically fit for content policing. It's doesn't have to be for everyone.
dhosek 6y ago
MySpace had to deal with this. They also had content moderators who were kept in a separate room at HQ who had to view all manner of flagged videos/photos. The turnover was really high even though at the time, MySpace had the kind of cool factor that developers would pick up women at bars with the promise of taking them into the offices after hours (which is a whole other weirdness and the corporate higher ups had to ban the practice after a drunk couple were injured after falling off a table one Saturday night). I don't know that there was anything acting at this kind of scale before MySpace.
- cvhashim 6y ago
  Facebook was a zoo in its early years. Basically a frat house. Google was also a complete boys club and full of misconduct. But you have to be a complete nerd to take your date to your office, even if it’s a nice office.
  - dhosek 6y ago
    It really wasn't a nice office. My wife worked there from 2005–2008 and I'd occasionally pick her up after work. I think the big thing was just that it was MySpace and at the time, it was the epitome of cool. They even had a record label https://en.wikipedia.org/wiki/MySpace_Records https://en.wikipedia.org/wiki/MySpace_Records Unlike Facebook, MySpace was heavily tied into the entertainment industry, especially music, but to a lesser extent film and stand-up comedy which probably contributed to the cool factor. The engineering culture was pretty heavily bro-grammer in a way I've never seen anywhere else.
TonyTrapp 6y ago
I have moderated several forums over the course of 15+ years, and I have never come across this kind of content. But then again, all of them were very specialized in their topics. I still am convinced that we would be better off if we went back to small forums for people sharing common interests rather than having a huge platform. People can police their small group of peers much more easily. It works.
- droffel 6y ago
  Communities grow - I think this kind of strategy only works on a small scale. If it was adopted en masse, it becomes self defeating. Suggesting everyone change groups the moment their current group reaches a certain size ignores the greater point, which is that these large groups are going to exist/emerge regardless.
joe_the_user 6y ago
It's probably a bit controversial to say this, but haven't large sites had to deal with these problems for years by the time the likes of Facebook/Twitter/YouTube/Instagram/whatever came around? I don't think at the same level. Social media usage went from 10% of the population to 79% from 2005 to 2015 and was even lower a few years before this. So current social media companies operate at unprecedented scales. Moreover, the smaller percentage of people usage social media in earlier times tended to be "sophisticated users" who didn't necessarily expect their feed to be systematically filtered. https://www.pewresearch.org/internet/2015/10/08/social-networking-usage-2005-2015/ https://www.pewresearch.org/internet/2015/10/08/social-netwo...

by CM30 6y ago

It's probably a bit controversial to say this, but haven't large sites had to deal with these problems for years by the time the likes of Facebook/Twitter/YouTube/Instagram/whatever came around? It's obviously horrible doing this as a job and it's obviously a bit more common on Facebook than say, a large old school internet forum, but... at least in this case people are paid to do this. Reddit mods and old school forum mods have to deal with this stuff for free.

tmpz22 6y ago

Right but the difference is the sheer amount of money involved and the politicization that comes with that. If Facebook was worth 100m nobody would care. If facebook only has 10m users nobody would care. It's the scale that completely breaks the legal and political frameworks the our society leans on.

wheelie_boy 6y ago
I'd say another difference is the percentage of your attention is being devoted to this kind of thing. Presumably reddit or phpbb mods aren't spending 40 hr/wk looking at gore.
- Nasrudith 6y ago
  Wouldn't that suggest then that the "ideal" way to do it is a very part time job then with high volume? Like say 5 hours a week. It would be a scheduling and logistical nightmare of course but it would give more "dispersion time" for trauma.
  - wheelie_boy 6y ago
    I'd be interested to see if the tooling around moderation could also be improved. For example, could the images start heavily blurred, with a circle of clarity that opens when you click. Or something like that, which would harm throughput, but be more humane.
    - prawn 6y ago
      This is a very interesting idea. It could reveal only a portion and see how often that was enough for the moderator to pass judgement.
  - asddubs 6y ago
    tbh I've been a moderator on various old school forums for over a decade, and I've never had to deal with anyone posting any gore at all. I think they're usually too tight knit and reputation based for anyone to feel like they could get away with it. it's not like people posting gore on facebook are doing it to mess with the moderators, that's just a side effect
hinkley 6y ago
From my time observing volunteer groups, it's also about leverage and force multipliers. I can go door to door asking people to change something, and dozens of my volunteers will feel useful but overwhelmed by how much work there is to do in a city of a million people. Or I can go to city council meetings, and get more leverage there. Or if I can convince one high profile citizen to participate, others will follow. Or I can get into contact with the mayor, state congressional members, the governor, federal congressional members. It's possible that if I convince the correct five people that they will take half of this burden off of my hands. You'd be crazy not to reach for a lever like that. They are a juicy target, and not just from a punitive standpoint. Of course when you are rebuffed repeatedly by a potential ally, then it's a very human (if questionable) response to seek out some sort of punishment.
- ece 6y ago
  Do you know what community organizing is? How movements accomplish change? Everything does not have some easy optimization method, only in silicon valley does everyone think it does. edit: What I mean to say is, people have been organizing against FB, AMZN, and tech in general for the past few years now. The last is reactionary but opposition to FB and AMZN has meaningful academia, ngo/gov, and general support.
  - hinkley 6y ago
    I worked with a group that got 100 volunteers at each event who did the work while the regular members provided coordination and resources. I am trying to do something similar on my own now and learning in a very visceral way just how important spreading that load out is. But... that group wouldn't be able to do nearly what it has if it weren't for funding from foundations and the department of neighborhoods, and resources on permanent loan from another public sector organization. You might argue it couldn't exist at all. I think I would say that both are necessary but insufficient.
  - SaltyLemonZest 6y ago
    I think the point is that, if I as an activist am aiming to maximize the number of content moderators treated well, the highest leverage way to push for that is getting people mad at Facebook. There may be a preexisting system of unpaid content moderators who are supported even less, but that's much harder for anyone to address.
ece 6y ago
Yeah, monopolies can do that. edit: https://qz.com/1704143/the-antitrust-case-against-facebook/ https://qz.com/1704143/the-antitrust-case-against-facebook/
whymauri 6y ago
Another factor is the working conditions at Cognizant. The difference between Reddit moderators and Cognizant employees is that the former are allowed to have bathroom breaks and individual agency. The latter not only have to deal with fairly oppressive workplace policies, but they also rely on this work to live.

untog 6y ago

I think it's fair to say the sheer quantity has changed since Facebook first came around (FB predates smartphones for one!) I also think part of the objection here is around off-shoring. It's similar to other complaints about off-shored jobs: we're farming out unpleasant work to people that get paid less and have much less support (e.g. mental health).

CM30 6y ago
> I also think part of the objection here is around off-shoring. It's similar to other complaints about off-shored jobs: we're farming out unpleasant work to people that get paid less and have much less support (e.g. mental health). That's an interesting point, and has me wondering something. If a site like Facebook opened up a volunteer moderator scheme where people could sign up to filter out stuff that broke their rules, would it make things better or worse on a moral level? What about a US/Europe based agency that did this thing onshore as a contract deal?
- untog 6y ago
  Facebook already hires moderators inside the US via agencies, and they have their own problems: https://www.theverge.com/2019/2/25/18229714/cognizant-facebook-content-moderator-interviews-trauma-working-conditions-arizona https://www.theverge.com/2019/2/25/18229714/cognizant-facebo... As for the volunteer side, it's a really interesting debate. Others in this thread have pointed out Reddit as an example of this in practise. I think the sheer amount of money that Facebook has means that relying on volunteers to (potentially) inflict emotional damage upon themselves wouldn't be a great look.
- AdmiralGinge 6y ago
  I'd consider volunteering to moderate a particular geographical area, when I was a student Facebook constantly took down the student confessions page I eventually ended up helping to run for the vaguest reasons, while letting actual hate posters run amok. I know I would say that, but we would genuinely get taken down for jokes that wouldn't even make sense out of the tiny university I attended. It was immensely frustrating having to build up the userbase again and again because the moderators of Facebook just weren't/aren't terribly good at their jobs. I'd happily have taken on actual moderation duties rather than had to constantly appeal and get a blank, inhuman "declined" message. I think they tend to take an "if in doubt, remove it" approach which basically allows anyone who doesn't like a niche page for whatever reason to have it taken down through sheer attrition, if you report it enough it'll probably go down eventually even for stupid reasons. I actually ended up writing an app to cut Facebook out of our system altogether (it was kind of similar to Yik Yak with a much better moderation system and better anonymity protection - I really can't be arsed to deal with a GDPR lawsuit...) but we couldn't launch it in part because it took me ages to develop as I've since graduated and work as a full-time developer and because coronavirus meant our launch event got cancelled. I might take it live when students go back this year, but I'm concerned that with nothing much going on it'll fizzle and die because there's nothing interesting to post about. It'll all depend on how the Welsh government lift the coronavirus restrictions.

WrongThinkerNo5 6y ago

I suspect there is a certain interest group that has emerged and grown around this subject that has been elevating and promoting this issue in the public's mind. It would not even surprise me one bit that especially overseas contractors, because there are likely fewer penalties/prohibitions, are paying people to flood, e.g., fb with things like beheading videos in order to produce a cause for their services. It may be cynical, but like I said, it would not surprise me one bit.

rtkwe 6y ago

Very few forums had even a fraction of the number of members as Facebook has users and the amount of content is increased again. Also the type of content they're required to moderate has changed most forums before facebook were largely text and video is a big part of facebook. It's easy to see how video and photo content is more traumatizing than the largely text content of forums.

indigochill 6y ago

>at least in this case people are paid to do this. I've done commercial moderation before and I would sooner go jobless than go back to exposing myself to the kinds of things people have to when doing this professionally. It's the kind of things you wish there was a brain cleaning solution for, but there isn't so it's stuck there basically forever (though I suspect the alcoholism rate is higher among people who have to handle this kind of thing than in the general population). The commercial moderation team I was on actually ended up separating out a task force for handling particularly gnarly content so that people had a way of opting out of being exposed to it (in our case we were responding to user reports rather than randomly stumbling on things, so manually filtering reports was feasible), which I think is the responsible approach since there are plenty of other moderation tasks to handle and some people are more or less sensitive to this content.

remcob 6y ago
> wish there was a brain cleaning solution 'Drug-induced amnesia' to reduce long term memory generation is a thing and is sometimes used before traumatic surgical procedures. It needs to be taken in advance and is probably not very healthy. Plus I would have serious ethical concerns with jobs that would encourage that.
- throwanem 6y ago
  I've had procedures where those drugs are used. They're benzodiazepines, and thus powerfully sedating. You are in no sense able to work while doped to the eyeballs with Versed.
- mirimir 6y ago
  I've read that ~high-dose opioids after trauma reduces PTSD.
stickfigure 6y ago
This seems like something specific to the individual. Maybe this isn't the job for you, but there's probably enough people "made of different stuff" that could handle it? I feel like there's a subtext in this discussion that "nobody can handle it" or "you have to be a monster to work this job" but that's just not true. Just out of college in the mid-90s, I worked for a few months at a small local software company in the last throws of death. As a last grasp at survival, all the programmers switched to doing cold call sales. With phones. I would stare at mutilated children all day long rather than go back to cold calling. I was miserable and I hated every moment of every day until the company (happily, for me) closed up. Yet I know people who somehow think it's perfectly normal to dial random numbers and try to pitch them something. They're made of different stuff. I've talked to people who make the same claim about computer programming. They'd rather starve. Facebook has 10,000 moderators. I'm pretty sure there are at least 10k people in the world who are psychologically fit for content policing. It's doesn't have to be for everyone.

dhosek 6y ago

MySpace had to deal with this. They also had content moderators who were kept in a separate room at HQ who had to view all manner of flagged videos/photos. The turnover was really high even though at the time, MySpace had the kind of cool factor that developers would pick up women at bars with the promise of taking them into the offices after hours (which is a whole other weirdness and the corporate higher ups had to ban the practice after a drunk couple were injured after falling off a table one Saturday night). I don't know that there was anything acting at this kind of scale before MySpace.

cvhashim 6y ago
Facebook was a zoo in its early years. Basically a frat house. Google was also a complete boys club and full of misconduct. But you have to be a complete nerd to take your date to your office, even if it’s a nice office.
- dhosek 6y ago
  It really wasn't a nice office. My wife worked there from 2005–2008 and I'd occasionally pick her up after work. I think the big thing was just that it was MySpace and at the time, it was the epitome of cool. They even had a record label https://en.wikipedia.org/wiki/MySpace_Records https://en.wikipedia.org/wiki/MySpace_Records Unlike Facebook, MySpace was heavily tied into the entertainment industry, especially music, but to a lesser extent film and stand-up comedy which probably contributed to the cool factor. The engineering culture was pretty heavily bro-grammer in a way I've never seen anywhere else.

TonyTrapp 6y ago

I have moderated several forums over the course of 15+ years, and I have never come across this kind of content. But then again, all of them were very specialized in their topics. I still am convinced that we would be better off if we went back to small forums for people sharing common interests rather than having a huge platform. People can police their small group of peers much more easily. It works.

droffel 6y ago
Communities grow - I think this kind of strategy only works on a small scale. If it was adopted en masse, it becomes self defeating. Suggesting everyone change groups the moment their current group reaches a certain size ignores the greater point, which is that these large groups are going to exist/emerge regardless.

joe_the_user 6y ago

It's probably a bit controversial to say this, but haven't large sites had to deal with these problems for years by the time the likes of Facebook/Twitter/YouTube/Instagram/whatever came around? I don't think at the same level. Social media usage went from 10% of the population to 79% from 2005 to 2015 and was even lower a few years before this. So current social media companies operate at unprecedented scales. Moreover, the smaller percentage of people usage social media in earlier times tended to be "sophisticated users" who didn't necessarily expect their feed to be systematically filtered. https://www.pewresearch.org/internet/2015/10/08/social-networking-usage-2005-2015/ https://www.pewresearch.org/internet/2015/10/08/social-netwo...