12 ms·

The Human Toll of Protecting the Internet from the Worst of Humanity

250 points by mathgenius 10y ago

stephengillie 10y ago
Are there any projects working to automate the identification of offensive material with Machine Vision or Machine Learning? Replacing these peoples' jobs with computers might actually be a humane act.
- kartan 10y ago
  As the article states, "Although algorithms and artificial intelligence have helped streamline the process of moderation, most technology companies that host user-generated content employ moderators like Soto to screen video, text, and images, to see if they violate company guidelines." So the algorithms are used to create the list of dubious material. But in the end you need a human being to look at the picture/video and call the police as needed. The day that AI can do that for us, they will do any other job also.
- shakna 10y ago
  I really wish there was. I did forensics for a couple years, working with the police force to access and/or recover incriminating evidence. I really, really wish that a machine could replace a person in that circumstance. And to a certain extent, it is possible. One of the really unfortunate parts of the process, is to know a video is incriminating, someone at some point has to see that video. It was part of my job to analyse and catalogue the evidence I recovered, such as saying who was in a video, and what happened to who inside it. ML will hopefully take over this at some point, but it is difficult for a computer, at this stage, to tell if say, penetration, is involved. It's moving in that direction, and that is absolutely fantastic. No person should be required to witness this crap. Hopefully, we can offload it to something that can, one day, be better than us at it.
  - deleted 10y ago
    [deleted]
  - jasonkostempski 10y ago
    I'm sorry anyone has to see stuff like that, but there is no way in hell machines can be allowed to start "confirming" such things without several people having to review it.
- brynedwards 10y ago
  There's this by Yahoo: https://github.com/yahoo/open_nsfw https://github.com/yahoo/open_nsfw
  - drdaeman 10y ago
    > For this reason, the model we describe below focuses only on one type of NSFW content: pornographic images. It tries to detect a subset of pornography. Others may have different opinions, but personally I wouldn't relate porn to a category of "worst of humanity".
    - brynedwards 10y ago
      Good point, I guess detecting something as e.g. showing child abuse would be much more difficult.
- DannyBee 10y ago
  Yes, there are a lot. For example, NCMEC maintains databases of child porn hashes using photodna and other things. Machine learning and other algorithms are used to identify images that end up in this database (and others). Sadly, teaching machines what exploitation looks like is hard and it's not as simple as "identify things that have explicit sexual activity" and then "filter those things by people who appear to be 6 years old". People still often have to make a final determination because categorization is ... hard (and honestly, nobody wants to be the guy handling mislabeling, etc)
  - village-idiot 10y ago
    The cost of false negatives in this area is so painfully high too.
- monista 10y ago
  Most obvious problem is: define offensive. What other people take as offense doesn't look offensive to me at all, and vice versa. There are other problems too. The only way to fight offenses is to restrict phrasing to Common Universal Neutral Terms. Of course, message consisting only of Common Universal Neutral Term Sequences cannot possibly contain an offense!
kartan 10y ago
> constant exposure to the worst of humanity on a daily basis takes an undeniable toll. This is one of the troubles police officers find in their daily work. When they are not part of the society where they work it is hard for them to empathize, as they just see the worst parts of that society. In the police case also to have members of the police that are part of the communities they serve. But counseling also is needed to be able to deal with a biased view of reality. People don't call the police to show them how nice a neighbor is, or to invite them to a party. Their experience is, like for the guy in the article, to see only the bad parts as the good ones are filtered away.
- KON_Air 10y ago
  I would dare say nothing of worth being done stop child abuse anywhere in world would compounds the effect of being exposed.
  - otakucode 10y ago
    Child advocacy groups are exasperated. The tactics that have been shown to actually help kids aren't sexy enough for law enforcement or politicians. Busting actual abusers is one bust, and its almost always of a parent or other close relative of the victim. Whereas they can get dozens of people trading pictures online in one fell swoop, and the public tends to treat that as just as useful, but research shows it just isn't. Those people aren't the ones abusing kids in the vast majority of cases. To stop child abuse, the most effective tactics are education and empowerment. Educating kids about the realistic dangers is uncomfortable and decidedly not something people are comfortable with. And empowerment - telling them that their body is their own property and no one, not even their parents, has the right to make them do anything they don't want to - goes against the narrative people want to believe about children being incapable of making those decisions (as if they're difficult). To stop abuse, you have to realize what abuse is happening. And its abuse by parents first and foremost. A close second is close family members and family friends. Strangers simply aren't the problem. They're a statistical anomaly that is certainly a problem, but one that education does away with quickly and easily. That leaves kids who are afraid of seeing their family destroyed and potentially being tossed into the foster care system. Kids need to know that they will be taken care of, and to be made confident and competent enough to stand up and call out abuse when it happens. It's depressing, slow, intensive work. And law enforcement and politicians are instead too easily convinced to abandon it entirely and just go for the quick busts of extremely dubious benefit to kids.
drinkjuice 10y ago
Dare I guess that these people generally get paid minimum wage? > Tech companies don’t like to talk about the details of content moderation, so it’s difficult to judge how well they’re caring for the psychological health of moderators. The number of such stories which include said companies declining comment (not this one, but anecdotally that's how it seems to be) is kind of a red flag though.
- manarth 10y ago
  "Often, the moderators are workers in developing countries, like the Philippines or India" It looks like those two countries do have a legal minimum wage [1], although it's pretty low - $0.28 for India, $0.61 for the Phillipines. No idea whether those numbers reflect living wages for those countries though, or what a content moderator might actually earn. [1] http://www.businessinsider.com/a-look-at-minimum-wages-around-the-world-2013-8?IR=T http://www.businessinsider.com/a-look-at-minimum-wages-aroun...
  - douche 10y ago
    If $.28 is really the minimum wage in India, then so much starts to make sense. No wonder that wild incompetence is the order of the day. I couldn't hire a kid to mow my lawn at that rate.
    - savanaly 10y ago
      Keep in mind cost of living is lower.
_callcc 10y ago
I think we'll see more stratification on the internet in the coming years with subscriptions, etc. There's probably even a market right now for a paid version of reddit or HN. The sad fact is that the internet has gradually become a shithole as the entirety of humanity has joined in. Twitter, for example, is being dragged down by low quality user accounts that add nothing but noise. It's actively hurting the stock price.
- bitlax 10y ago
  > The sad fact is that the internet has gradually become a shithole as the entirety of humanity has joined in. In what year do you feel the Internet was better than it is now?
  - drdaeman 10y ago
    1974 or few years after, maybe? After that a law of large numbers must've kicked in (and I don't think human nature had changed any much in last 3-4 decades).
    - tjr225 10y ago
      http://www.catb.org/jargon/html/S/September-that-never-ended.html http://www.catb.org/jargon/html/S/September-that-never-ended...
  - wott 10y ago
    Pre- or beside-Internet, the days of FidoNet; the days of Usenet before its decadence; the days before Web 2.0; the days before business shifted from traditional forms to Internet; the days before smartphones, etc. There's no clear point, but when it was loosely "reserved" to a certain "elite" (financial, technical, cultural, scientific) which showed genuine interest in it, it was in a much better condition than now. It sounds terrible to say that, but as the masses joined in, the media dumbed down and what it carries dumbed down as well. Another step towards dumbness happened when kids were massively allowed to enter the arena. Again, this sounds harsh, but that's the pitiful reality of humankind. And businesses and capitalism know very well how to encourage the most vile behaviours in order to make more profit. It also shifted from a confidence & collaboration mode to insecurity & defiance. Now it's "oh my God how this protocol / library is insecure, how could one design such a stupid thing?" Well, I am not sure who is most stupid. Remember when a 'troll' was just someone who was just pulling a prank teasing a few other people? Well now the same word means a stupid, racist, hateful, mean guy who drowns out a forum or a site with his feces and whose attacks impact real life of its victims. Which brings me to tell that another big difference is that when Internet was disconnected from real life, it was much better. That was a big turning point too. You used to be able to write everything under your real name and not fear some acquaintance, school mate, employer or any other peeping Tom or enemy would search for it. So there's not a single year to point out, but during the last 15 (20?) years a series of steps gradually turned Internet in the shithole (and supermarket) it is now. And none of those steps was positive.
wtbob 10y ago
The title is an instance where automatically substituting 'humanity' for 'mankind' leads to a contradiction in terms: humanity is the quality of being humane, or benevolence, and there's nothing bad or malevolent about it.
- edmccard 10y ago
  >humanity is the quality of being humane, or benevolence That's one definition; my dictionary also lists "the human race; mankind; people."
wallace_f 10y ago
Political censorship cleverly wrapped in a much more palatable package delivered to you for your acceptance.
- Brakenshire 10y ago
  Yes, in the same way that not inviting a neo-nazi or an anarcho-syndicalist, or a vegan, to talk on your radio programme is political censorship. Private people and organizations are well within their rights to set their own rules. If you want no moderation that's your choice, go to 4chan or a similar site.
  - anonnyj 10y ago
    There's moderation (bans for being off topic, bating, coordinated websit raiding, NSFW in the wrong place, etc) But yeah, there's less in the way of censorship. The odd moderator with an agenda still sneaks in, but they seem to get outed.
  - drdaeman 10y ago
    [Edit: parent was edited, I was replying to a house example] Not exactly. The comparision doesn't account for scale, and in case of largest platforms (like Facebook) it's more like "leave our country bloc"... except for such comparisions are probably a bad idea, because there are always too many differences that break them. ----- But parent comment has a point anyway. Political censorship had (ab)used moderation systems in that way, at the very least in some countries. It's not unheard of cases of censorship slipping under disguise of "protecting from bad/unhealthy things". I can think of a few examples from Russia.
    - Brakenshire 10y ago
      Yes, apologies for that, I changed it a few minutes after posting to be more constructive/reasonable, the original comment was flamebait on reflection. Facebook shouldn't be some kind of universal substrate on the internet anyway. Alternatives should be encouraged, and people can make their own choice about the kind of environment they want to spend their time in. If you're saying that's impossible, and facebook is some sort of utility, or common carrier, then you can have rights which are enforced upon the space. Ultimately, either it is a public space (which needs public rights analogous to what exists in the real world, enforced through constitutional or democratic means) or it is a private space, and the owners are free to regulate as they wish, and customers are free to make their own decisions, where to visit and where to spend their time.
RangerScience 10y ago
Huh. I'm reminded of that story about the guy who's shown surveillance footage of the middle east, and there's a computer that watches his brain because his brain will react to there being things like people in the images faster than he can be consciously aware of it. Would it work to use such a system to improve filtering while reducing the human cost? Or would it do something akin to what subliminal advertising is credited with, and just push the damage to the unconscious level?
- halomru 10y ago
  I don't think such a system would be notably better than standard mashine learning. Reliable image categorisation means you often have to take a second look and interpret context. The internet is filled with images that Intentionally lead to a wrong first impression and can only be recognized when consciously looking at them
apatters 10y ago
Instead of inflicting PTSD on content moderators, why not give the job to someone who actually wants to see this type of content? They must exist, otherwise the content wouldn't spread in the first place. And/or crowdsource it so that more people are spending less time doing it? Viewing a little bit of this content is probably less traumatic than viewing it for 40 hours a week. Of course I realize it'd probably be a regulatory minefield, a thorny area morally, and any specific plan would likely draw critics. But it seems a lot better to me than scarring people for life with a full time barrage of things that repulse them. For instance, why not offer convicted pedophiles jobs which involve screening out child porn? That would save an innocent person from the trauma involved and it might even allow the pedophile to feel like he was redeeming himself to some degree by saving others from viewing that material. From my understanding many people with that fetish feel a lot of remorse about it and they're not exactly trying to get other people into it.
- 77pt77 10y ago
  >For instance, why not offer convicted pedophiles jobs which involve screening out child porn? That would save an innocent person from the trauma involved and it might even allow the pedophile to feel like he was redeeming himself to some degree by saving others from viewing that material. Or just allow the pedophile to indulge in his paraphilia. I'm really not comfortable with this...
  - jessaustin 10y ago
    At some point it will be possible to create very realistic CP without using children. Actually it might already be possible to do that. One would have thought that would solve most of the problem: create "artificial" CP with strict controls to ensure children are not involved, and let the weirdos watch as much as they want. (Admittedly, there would still probably be some super-weirdos who would be totally turned off by seeing the "Attestation of No Children Involved" message at the beginning of their CP...) But this comment seems to imply that there would be a problem with that? What is the harm in movies that harm no one?
    - 10y ago
sidcool 10y ago
Sometimes I feel humanity does not deserve all the wonderful things that science and technology provides because people will find a bad use of it. Won't be surprised if someone creates a solar bomb.
- 20after4 10y ago
  The hydrogen bomb could be thought of as a solar bomb. The sun is made almost entirely of hydrogen.
losteverything 10y ago
The internet is a voluntary vice used by some to earn a living. It's not food, shelter, clothing AND the internet. Staying away from the internet is the next health craze.
- village-idiot 10y ago
  Maybe. It's also so crucial to so many careers that you need to be in a pretty special place to pull that off constantly, along with all the money it takes to entertain yourself sans internet.
  - stephancoral 10y ago
    A library card is free in many cities.
- lsd5you 10y ago
  Couldn't agree more. It is more like we need to protect humanity from the internet.
  - severine 10y ago
    Yeah, let only the powerful use it /s
    - mr_spothawk 10y ago
      We could put up a giant content firewall, and only allow the well connected to access the "real internet".
- bsaunder 10y ago
  The internet isn't a vice. Over use of social networks and various applications on it may be. The internet in many forms has become the primary means of human communication. Communication is a fundamental need of human societies. I view excessive social network usage (to the point of unhealthy) to be analogous with over eating and the obesity it leads to. Underlying both is a basic need. We do better collectively to encourage others to behave responsibly with moderation and enjoyment. In the extreme we should guard our freedoms, yet penalize harmful abuse.
jedanbik 10y ago
Can we get a collective thank you for the moderators on this website? It's a hard job with lots of nuance-against-shouting-people, and now has probably been the hardest time to be a moderator in the history of the internet.
- mr_spothawk 10y ago
  +1. I'm constantly delighted (and informed/encouraged/challenged/etc...) by the content available to me through this community, and by the hard work of its moderators. Thanks.
EekSnakePond 10y ago
As always, the risk is simple: deny the expression of human nature at your own peril. Those who hunt monsters are apt to become one.
Girlang 10y ago
"Protecting the Internet"? Who watches the watchmen?
bidoof___ 10y ago
Disqus and social sign-on are the great cancer. Every local news site on the planet that outsources authentication to whoever's got a valid Facebook, Google+, Twitter, etc., is opening themselves to troll farms landsliding both opinions and values out of existence in favor of mindless repeatable bites, "the Sleeper Effect." This is twice as true of Twitter discussions. Hostile nations are employing legions of both paid users with a posting quota and bots. They are taking advantage of globalism to spread fear and infest rivals with a self-destructive nationalistic ideology which appears to be grassroots and organic. Just the other day I saw "easyvpn" on here, a one command shell utility to hop VPNs to exit from any desired country. I assume far more anonymous and powerful alternatives exist if you have the cash. This kind of thing paired with "persona management software" is a real problem. Step back, stop out-sourcing authentication on your websites (even if it means you have to do the time learning some amount of practical cryptography, it's safer than handing the keys off in this climate), and stop conceding massive tracts of digital "space" to a hundred thousand Ru & Romanian twitter accounts claiming to be Joe America tired of liberals while deflecting to "Correct the Record" and "but emails" every time you call them out. Step it up with security, boys, that means some degree of human intelligence. The Internet is growing up and it's going through a pretty nasty phase right now. Build a wall? How about build a wall around your website so Facebook social sign-on isn't letting Ivan roleplay half of Texas on a popular news story.
- empressplay 10y ago
  It's a good thing all those Trump supporters were bots, if they had been real he might have got elected! (Oh, wait...)
  - bidoof___ 10y ago
    You misunderstand the impact of social media marketing.
- deleted 10y ago
  [deleted]
- sesqu 10y ago
  I don't think social sign-on is that big of a problem. It certainly makes it easier to comment on news stories, but if you're spending the effort to create thousands of social profiles already, what's a few more e-mail registrations using your Facebook email address? You could restrict comments to subscribers, but that's anti-social to a degree that I expect is too severe for marketers.
ocschwar 10y ago
What I'm not clear on is why these mods had to look at all at some of these pictures and videos. There are published hashes of files known to contain child porn and the like. sha1sum matches? Bit length matches? Contact LEOs with relevant details, delete the post, block the accounts, and move on.
zw123456 10y ago
I have seen articles similar to this one a number of times. The thing that I wonder about is if there are certain personality types that would be less sensitive than others while still able to remain objective in terms of applying the standards. For example, would a sociopath be unaffected by the content? Just wondering if a psychological screening for the right personality type to do the job would help. BTW, thank you to the people that do that hard job including the HN screeners.
- kem 10y ago
  This is a bit within the realm of my research, so I feel confident about commenting: yes, it's possible. There might be a catch-22 in that the people who would be most resilient are probably those who would be least concerned about the welfare of victims. However, I don't know that's really perfectly accurate: I think you could find some medium, or identify those who empathize but tend not to become easily upset. Empathy and emotional response are kind of dissociable. Interestingly, the rates of stress disorder in those populations that they cite in the article are kind of in the ballpark of what you see in epidemiological studies of trauma-exposed populations, like combat exposed populations, areas subject to frequent terrorism, natural disaster survivors and so forth. My impression based on what they're seeing is that it's comparable to a job with high trauma exposure rates. It might be a prime opportunity to study trauma response and treatment, actually, in that often these things have an element of unpredictability to them. If content moderation is something that's reliably characterized by traumatic exposure, you could maybe try to study content moderators to identify predictors of and preventions for stress response.
manigandham 10y ago
Wired also did a good writeup in 2014: https://www.wired.com/2014/10/content-moderation/ https://www.wired.com/2014/10/content-moderation/
ajeet_dhaliwal 10y ago
In the games industry dealing with abusive online players can get demoralising too. In a lobbby and someone has black sounding voice? Abuse. A woman's voice? Abuse. It becomes an ethical and business issue because these are paying customers on services like Xbox Live and PlayStation Plus. There a ToS and abusive people are a minority but in gaming it can be a large minority. Developers do write algorithms to help deal with these people (won't go into details intentionally) but without outright banning them its not always full proof.
uptownfunk 10y ago
I'm traumatized by the iPhone ui of that site with the ads.. couldn't even make it through the article..

250 points by mathgenius 10y ago

Are there any projects working to automate the identification of offensive material with Machine Vision or Machine Learning? Replacing these peoples' jobs with computers might actually be a humane act.

kartan 10y ago
As the article states, "Although algorithms and artificial intelligence have helped streamline the process of moderation, most technology companies that host user-generated content employ moderators like Soto to screen video, text, and images, to see if they violate company guidelines." So the algorithms are used to create the list of dubious material. But in the end you need a human being to look at the picture/video and call the police as needed. The day that AI can do that for us, they will do any other job also.
shakna 10y ago
I really wish there was. I did forensics for a couple years, working with the police force to access and/or recover incriminating evidence. I really, really wish that a machine could replace a person in that circumstance. And to a certain extent, it is possible. One of the really unfortunate parts of the process, is to know a video is incriminating, someone at some point has to see that video. It was part of my job to analyse and catalogue the evidence I recovered, such as saying who was in a video, and what happened to who inside it. ML will hopefully take over this at some point, but it is difficult for a computer, at this stage, to tell if say, penetration, is involved. It's moving in that direction, and that is absolutely fantastic. No person should be required to witness this crap. Hopefully, we can offload it to something that can, one day, be better than us at it.
- deleted 10y ago
  [deleted]
- jasonkostempski 10y ago
  I'm sorry anyone has to see stuff like that, but there is no way in hell machines can be allowed to start "confirming" such things without several people having to review it.
brynedwards 10y ago
There's this by Yahoo: https://github.com/yahoo/open_nsfw https://github.com/yahoo/open_nsfw
- drdaeman 10y ago
  > For this reason, the model we describe below focuses only on one type of NSFW content: pornographic images. It tries to detect a subset of pornography. Others may have different opinions, but personally I wouldn't relate porn to a category of "worst of humanity".
  - brynedwards 10y ago
    Good point, I guess detecting something as e.g. showing child abuse would be much more difficult.
DannyBee 10y ago
Yes, there are a lot. For example, NCMEC maintains databases of child porn hashes using photodna and other things. Machine learning and other algorithms are used to identify images that end up in this database (and others). Sadly, teaching machines what exploitation looks like is hard and it's not as simple as "identify things that have explicit sexual activity" and then "filter those things by people who appear to be 6 years old". People still often have to make a final determination because categorization is ... hard (and honestly, nobody wants to be the guy handling mislabeling, etc)
- village-idiot 10y ago
  The cost of false negatives in this area is so painfully high too.
monista 10y ago
Most obvious problem is: define offensive. What other people take as offense doesn't look offensive to me at all, and vice versa. There are other problems too. The only way to fight offenses is to restrict phrasing to Common Universal Neutral Terms. Of course, message consisting only of Common Universal Neutral Term Sequences cannot possibly contain an offense!

kartan 10y ago

> constant exposure to the worst of humanity on a daily basis takes an undeniable toll. This is one of the troubles police officers find in their daily work. When they are not part of the society where they work it is hard for them to empathize, as they just see the worst parts of that society. In the police case also to have members of the police that are part of the communities they serve. But counseling also is needed to be able to deal with a biased view of reality. People don't call the police to show them how nice a neighbor is, or to invite them to a party. Their experience is, like for the guy in the article, to see only the bad parts as the good ones are filtered away.

KON_Air 10y ago
I would dare say nothing of worth being done stop child abuse anywhere in world would compounds the effect of being exposed.
- otakucode 10y ago
  Child advocacy groups are exasperated. The tactics that have been shown to actually help kids aren't sexy enough for law enforcement or politicians. Busting actual abusers is one bust, and its almost always of a parent or other close relative of the victim. Whereas they can get dozens of people trading pictures online in one fell swoop, and the public tends to treat that as just as useful, but research shows it just isn't. Those people aren't the ones abusing kids in the vast majority of cases. To stop child abuse, the most effective tactics are education and empowerment. Educating kids about the realistic dangers is uncomfortable and decidedly not something people are comfortable with. And empowerment - telling them that their body is their own property and no one, not even their parents, has the right to make them do anything they don't want to - goes against the narrative people want to believe about children being incapable of making those decisions (as if they're difficult). To stop abuse, you have to realize what abuse is happening. And its abuse by parents first and foremost. A close second is close family members and family friends. Strangers simply aren't the problem. They're a statistical anomaly that is certainly a problem, but one that education does away with quickly and easily. That leaves kids who are afraid of seeing their family destroyed and potentially being tossed into the foster care system. Kids need to know that they will be taken care of, and to be made confident and competent enough to stand up and call out abuse when it happens. It's depressing, slow, intensive work. And law enforcement and politicians are instead too easily convinced to abandon it entirely and just go for the quick busts of extremely dubious benefit to kids.

drinkjuice 10y ago

Dare I guess that these people generally get paid minimum wage? > Tech companies don’t like to talk about the details of content moderation, so it’s difficult to judge how well they’re caring for the psychological health of moderators. The number of such stories which include said companies declining comment (not this one, but anecdotally that's how it seems to be) is kind of a red flag though.

manarth 10y ago
"Often, the moderators are workers in developing countries, like the Philippines or India" It looks like those two countries do have a legal minimum wage [1], although it's pretty low - $0.28 for India, $0.61 for the Phillipines. No idea whether those numbers reflect living wages for those countries though, or what a content moderator might actually earn. [1] http://www.businessinsider.com/a-look-at-minimum-wages-around-the-world-2013-8?IR=T http://www.businessinsider.com/a-look-at-minimum-wages-aroun...
- douche 10y ago
  If $.28 is really the minimum wage in India, then so much starts to make sense. No wonder that wild incompetence is the order of the day. I couldn't hire a kid to mow my lawn at that rate.
  - savanaly 10y ago
    Keep in mind cost of living is lower.

_callcc 10y ago

I think we'll see more stratification on the internet in the coming years with subscriptions, etc. There's probably even a market right now for a paid version of reddit or HN. The sad fact is that the internet has gradually become a shithole as the entirety of humanity has joined in. Twitter, for example, is being dragged down by low quality user accounts that add nothing but noise. It's actively hurting the stock price.

bitlax 10y ago
> The sad fact is that the internet has gradually become a shithole as the entirety of humanity has joined in. In what year do you feel the Internet was better than it is now?
- drdaeman 10y ago
  1974 or few years after, maybe? After that a law of large numbers must've kicked in (and I don't think human nature had changed any much in last 3-4 decades).
  - tjr225 10y ago
    http://www.catb.org/jargon/html/S/September-that-never-ended.html http://www.catb.org/jargon/html/S/September-that-never-ended...
- wott 10y ago
  Pre- or beside-Internet, the days of FidoNet; the days of Usenet before its decadence; the days before Web 2.0; the days before business shifted from traditional forms to Internet; the days before smartphones, etc. There's no clear point, but when it was loosely "reserved" to a certain "elite" (financial, technical, cultural, scientific) which showed genuine interest in it, it was in a much better condition than now. It sounds terrible to say that, but as the masses joined in, the media dumbed down and what it carries dumbed down as well. Another step towards dumbness happened when kids were massively allowed to enter the arena. Again, this sounds harsh, but that's the pitiful reality of humankind. And businesses and capitalism know very well how to encourage the most vile behaviours in order to make more profit. It also shifted from a confidence & collaboration mode to insecurity & defiance. Now it's "oh my God how this protocol / library is insecure, how could one design such a stupid thing?" Well, I am not sure who is most stupid. Remember when a 'troll' was just someone who was just pulling a prank teasing a few other people? Well now the same word means a stupid, racist, hateful, mean guy who drowns out a forum or a site with his feces and whose attacks impact real life of its victims. Which brings me to tell that another big difference is that when Internet was disconnected from real life, it was much better. That was a big turning point too. You used to be able to write everything under your real name and not fear some acquaintance, school mate, employer or any other peeping Tom or enemy would search for it. So there's not a single year to point out, but during the last 15 (20?) years a series of steps gradually turned Internet in the shithole (and supermarket) it is now. And none of those steps was positive.

wtbob 10y ago

The title is an instance where automatically substituting 'humanity' for 'mankind' leads to a contradiction in terms: humanity is the quality of being humane, or benevolence, and there's nothing bad or malevolent about it.

edmccard 10y ago
>humanity is the quality of being humane, or benevolence That's one definition; my dictionary also lists "the human race; mankind; people."

wallace_f 10y ago

Political censorship cleverly wrapped in a much more palatable package delivered to you for your acceptance.

Brakenshire 10y ago
Yes, in the same way that not inviting a neo-nazi or an anarcho-syndicalist, or a vegan, to talk on your radio programme is political censorship. Private people and organizations are well within their rights to set their own rules. If you want no moderation that's your choice, go to 4chan or a similar site.
- anonnyj 10y ago
  There's moderation (bans for being off topic, bating, coordinated websit raiding, NSFW in the wrong place, etc) But yeah, there's less in the way of censorship. The odd moderator with an agenda still sneaks in, but they seem to get outed.
- drdaeman 10y ago
  [Edit: parent was edited, I was replying to a house example] Not exactly. The comparision doesn't account for scale, and in case of largest platforms (like Facebook) it's more like "leave our country bloc"... except for such comparisions are probably a bad idea, because there are always too many differences that break them. ----- But parent comment has a point anyway. Political censorship had (ab)used moderation systems in that way, at the very least in some countries. It's not unheard of cases of censorship slipping under disguise of "protecting from bad/unhealthy things". I can think of a few examples from Russia.
  - Brakenshire 10y ago
    Yes, apologies for that, I changed it a few minutes after posting to be more constructive/reasonable, the original comment was flamebait on reflection. Facebook shouldn't be some kind of universal substrate on the internet anyway. Alternatives should be encouraged, and people can make their own choice about the kind of environment they want to spend their time in. If you're saying that's impossible, and facebook is some sort of utility, or common carrier, then you can have rights which are enforced upon the space. Ultimately, either it is a public space (which needs public rights analogous to what exists in the real world, enforced through constitutional or democratic means) or it is a private space, and the owners are free to regulate as they wish, and customers are free to make their own decisions, where to visit and where to spend their time.

RangerScience 10y ago

Huh. I'm reminded of that story about the guy who's shown surveillance footage of the middle east, and there's a computer that watches his brain because his brain will react to there being things like people in the images faster than he can be consciously aware of it. Would it work to use such a system to improve filtering while reducing the human cost? Or would it do something akin to what subliminal advertising is credited with, and just push the damage to the unconscious level?

halomru 10y ago
I don't think such a system would be notably better than standard mashine learning. Reliable image categorisation means you often have to take a second look and interpret context. The internet is filled with images that Intentionally lead to a wrong first impression and can only be recognized when consciously looking at them

apatters 10y ago

Instead of inflicting PTSD on content moderators, why not give the job to someone who actually wants to see this type of content? They must exist, otherwise the content wouldn't spread in the first place. And/or crowdsource it so that more people are spending less time doing it? Viewing a little bit of this content is probably less traumatic than viewing it for 40 hours a week. Of course I realize it'd probably be a regulatory minefield, a thorny area morally, and any specific plan would likely draw critics. But it seems a lot better to me than scarring people for life with a full time barrage of things that repulse them. For instance, why not offer convicted pedophiles jobs which involve screening out child porn? That would save an innocent person from the trauma involved and it might even allow the pedophile to feel like he was redeeming himself to some degree by saving others from viewing that material. From my understanding many people with that fetish feel a lot of remorse about it and they're not exactly trying to get other people into it.

77pt77 10y ago
>For instance, why not offer convicted pedophiles jobs which involve screening out child porn? That would save an innocent person from the trauma involved and it might even allow the pedophile to feel like he was redeeming himself to some degree by saving others from viewing that material. Or just allow the pedophile to indulge in his paraphilia. I'm really not comfortable with this...
- jessaustin 10y ago
  At some point it will be possible to create very realistic CP without using children. Actually it might already be possible to do that. One would have thought that would solve most of the problem: create "artificial" CP with strict controls to ensure children are not involved, and let the weirdos watch as much as they want. (Admittedly, there would still probably be some super-weirdos who would be totally turned off by seeing the "Attestation of No Children Involved" message at the beginning of their CP...) But this comment seems to imply that there would be a problem with that? What is the harm in movies that harm no one?
  - 10y ago

sidcool 10y ago

Sometimes I feel humanity does not deserve all the wonderful things that science and technology provides because people will find a bad use of it. Won't be surprised if someone creates a solar bomb.

20after4 10y ago
The hydrogen bomb could be thought of as a solar bomb. The sun is made almost entirely of hydrogen.

losteverything 10y ago

The internet is a voluntary vice used by some to earn a living. It's not food, shelter, clothing AND the internet. Staying away from the internet is the next health craze.

village-idiot 10y ago
Maybe. It's also so crucial to so many careers that you need to be in a pretty special place to pull that off constantly, along with all the money it takes to entertain yourself sans internet.
- stephancoral 10y ago
  A library card is free in many cities.
lsd5you 10y ago
Couldn't agree more. It is more like we need to protect humanity from the internet.
- severine 10y ago
  Yeah, let only the powerful use it /s
  - mr_spothawk 10y ago
    We could put up a giant content firewall, and only allow the well connected to access the "real internet".
bsaunder 10y ago
The internet isn't a vice. Over use of social networks and various applications on it may be. The internet in many forms has become the primary means of human communication. Communication is a fundamental need of human societies. I view excessive social network usage (to the point of unhealthy) to be analogous with over eating and the obesity it leads to. Underlying both is a basic need. We do better collectively to encourage others to behave responsibly with moderation and enjoyment. In the extreme we should guard our freedoms, yet penalize harmful abuse.

jedanbik 10y ago

Can we get a collective thank you for the moderators on this website? It's a hard job with lots of nuance-against-shouting-people, and now has probably been the hardest time to be a moderator in the history of the internet.

mr_spothawk 10y ago
+1. I'm constantly delighted (and informed/encouraged/challenged/etc...) by the content available to me through this community, and by the hard work of its moderators. Thanks.

EekSnakePond 10y ago

As always, the risk is simple: deny the expression of human nature at your own peril. Those who hunt monsters are apt to become one.

Girlang 10y ago

"Protecting the Internet"? Who watches the watchmen?

bidoof___ 10y ago

Disqus and social sign-on are the great cancer. Every local news site on the planet that outsources authentication to whoever's got a valid Facebook, Google+, Twitter, etc., is opening themselves to troll farms landsliding both opinions and values out of existence in favor of mindless repeatable bites, "the Sleeper Effect." This is twice as true of Twitter discussions. Hostile nations are employing legions of both paid users with a posting quota and bots. They are taking advantage of globalism to spread fear and infest rivals with a self-destructive nationalistic ideology which appears to be grassroots and organic. Just the other day I saw "easyvpn" on here, a one command shell utility to hop VPNs to exit from any desired country. I assume far more anonymous and powerful alternatives exist if you have the cash. This kind of thing paired with "persona management software" is a real problem. Step back, stop out-sourcing authentication on your websites (even if it means you have to do the time learning some amount of practical cryptography, it's safer than handing the keys off in this climate), and stop conceding massive tracts of digital "space" to a hundred thousand Ru & Romanian twitter accounts claiming to be Joe America tired of liberals while deflecting to "Correct the Record" and "but emails" every time you call them out. Step it up with security, boys, that means some degree of human intelligence. The Internet is growing up and it's going through a pretty nasty phase right now. Build a wall? How about build a wall around your website so Facebook social sign-on isn't letting Ivan roleplay half of Texas on a popular news story.

empressplay 10y ago
It's a good thing all those Trump supporters were bots, if they had been real he might have got elected! (Oh, wait...)
- bidoof___ 10y ago
  You misunderstand the impact of social media marketing.
deleted 10y ago
[deleted]
sesqu 10y ago
I don't think social sign-on is that big of a problem. It certainly makes it easier to comment on news stories, but if you're spending the effort to create thousands of social profiles already, what's a few more e-mail registrations using your Facebook email address? You could restrict comments to subscribers, but that's anti-social to a degree that I expect is too severe for marketers.

ocschwar 10y ago

What I'm not clear on is why these mods had to look at all at some of these pictures and videos. There are published hashes of files known to contain child porn and the like. sha1sum matches? Bit length matches? Contact LEOs with relevant details, delete the post, block the accounts, and move on.

zw123456 10y ago

I have seen articles similar to this one a number of times. The thing that I wonder about is if there are certain personality types that would be less sensitive than others while still able to remain objective in terms of applying the standards. For example, would a sociopath be unaffected by the content? Just wondering if a psychological screening for the right personality type to do the job would help. BTW, thank you to the people that do that hard job including the HN screeners.

kem 10y ago
This is a bit within the realm of my research, so I feel confident about commenting: yes, it's possible. There might be a catch-22 in that the people who would be most resilient are probably those who would be least concerned about the welfare of victims. However, I don't know that's really perfectly accurate: I think you could find some medium, or identify those who empathize but tend not to become easily upset. Empathy and emotional response are kind of dissociable. Interestingly, the rates of stress disorder in those populations that they cite in the article are kind of in the ballpark of what you see in epidemiological studies of trauma-exposed populations, like combat exposed populations, areas subject to frequent terrorism, natural disaster survivors and so forth. My impression based on what they're seeing is that it's comparable to a job with high trauma exposure rates. It might be a prime opportunity to study trauma response and treatment, actually, in that often these things have an element of unpredictability to them. If content moderation is something that's reliably characterized by traumatic exposure, you could maybe try to study content moderators to identify predictors of and preventions for stress response.

manigandham 10y ago

Wired also did a good writeup in 2014: https://www.wired.com/2014/10/content-moderation/ https://www.wired.com/2014/10/content-moderation/

ajeet_dhaliwal 10y ago

In the games industry dealing with abusive online players can get demoralising too. In a lobbby and someone has black sounding voice? Abuse. A woman's voice? Abuse. It becomes an ethical and business issue because these are paying customers on services like Xbox Live and PlayStation Plus. There a ToS and abusive people are a minority but in gaming it can be a large minority. Developers do write algorithms to help deal with these people (won't go into details intentionally) but without outright banning them its not always full proof.

uptownfunk 10y ago

I'm traumatized by the iPhone ui of that site with the ads.. couldn't even make it through the article..