14 ms·

The murky history of moderation and how it’s shaping the future of free speech

263 points by pmcpinto 11y ago

niels_olson 11y ago
This makes the criticisms of Wikipedia look pretty bland. I had never thought about what life in these trenches could look like. Trenches indeed.
- dredmorbius 11y ago
  Realise that pretty much anything that hits YouTube or FB also hits Wikpedia. (I'm trying hard to avoid making plays on "pediwikia" without success. And not faulting Wikipedia but rather those who'll post that kind of content, or other rot, if they can.)
- dclowd9901 11y ago
  Seriously. I can't count the number of times a day I see something marked NSFL and think, "nope, not even gonna look." But these people have to. I wouldn't wish that on anyone.
proksoup 11y ago
> The details of moderation practices are routinely hidden from public view, siloed within companies and treated as trade secrets when it comes to users and the public. Despite persistent calls from civil society advocates for transparency, social media companies do not publish details of their internal content moderation guidelines; no major platform has made such guidelines public. It's really nice to see light shining on this topic. Truly, this content moderation is the equivalent (equivalent for the internet medium) of the boundaries of debate enforced by the editors and owners of major media corporations on our previous mediums.
- dmckeon 11y ago
  If the details of moderation rules were public, companies would be facing additional labor costs dealing with posters pushing hard on the edges of those rules. At some level, users can choose from a variety of forums with different moderation methods, or none, and companies can try to fit their moderation policies and practices to their users' expectations. All of that develops in a cross between evolution and arms race, with lots of moving targets - global variation, changing social mores, changing technology, the latest taboos and moral panics. Exposing the details of how the sausage is made would help the people who want to put pebbles (or worse) in the sausage more than it would help the makers or consumers. If one finds the sausage too bland, there is likely a spicier one just a few clicks away. Or one could grind one's own sausage and compete!
  - proksoup 11y ago
    What you describe sounds like security by obscurity.
    - dmckeon 11y ago
      Publishing moderation policy details would be like crowd-sourcing cryptography software development - creating a fractally increased attack surface with a less profitable business model. Social media companies operate in a market economy. These are not public utilities or government programs. Feel free to propose a more cost-effective approach.
      - proksoup 11y ago
        I think exposing the rules will make them stronger.
        intended 11y ago
        Moderation is not cyber security. Virii don't turn around and argue that your AV is unfairly applying their rules to them.
    - dang 11y ago
      I think the rules are quite different when dealing with human processes rather than algorithmic ones.
      - proksoup 11y ago
        Could you elaborate, the same principle seems to apply to me, the costs of dealing with people attacking the edge strengthens the edge, no?
    - ihnorton 11y ago
      Or plausible deniability.
  - parenthephobia 11y ago
deleted 11y ago
[deleted]
ckelly 11y ago
> In April of 2005, they tested their first upload. By October, they had posted their first one million-view hit: Brazilian soccer phenom Ronaldinho trying out a pair of gold cleats. Weeks later, Google paid an unprecedented $1.65 billion to buy the site. This article misstates when Google acquired YouTube. It was October 2006, not October 2005: https://en.wikipedia.org/wiki/YouTube#Company_history https://en.wikipedia.org/wiki/YouTube#Company_history
- a301 11y ago
  They also mistake the traffic stats for /r/AskReddit as being traffic for all of Reddit: > According to a source close to the moderation process at Reddit, the climate there is far worse. Despite the site’s size and influence — attracting some 4 to 5 million page views a day[1] — Reddit has a full-time staff of only around 75 people, leaving Redditors to largely police themselves [1] https://www.reddit.com/r/AskReddit/about/traffic https://www.reddit.com/r/AskReddit/about/traffic
  - reycharles 11y ago
    > last month, reddit had 243,632,148 unique visitors hailing from over 212 different countries viewing a total of 8,137,128,592 pages Which is about 270 million page views per day. https://www.reddit.com/about/ https://www.reddit.com/about/
  - takno 11y ago
    Given that they don't have a large marketing team, 75 people would be a huge team to deal with 4 to 5 million page views
- giancarlostoro 11y ago
  I'm glad you caught this too, I thought it was WAY too soon, I knew YouTube when it had relatively grown in popularity and quite a bit before Google's acquisition of it.
- takno 11y ago
  By the time I read it this had been corrected and acknowledged in the footnotes, so I guess it's good to know the system works
xnull2guest 11y ago
Many of the problems faced by Facebook, Youtube and others is that they own the digital commons and manage and curate it for the people trying to communicate with one another. These issues come up for companies whose business models are deciding what people should see and hear. This makes them incredibly powerful, but it brings them into the bog. This isn't a new problem, other than internet scale. Media companies have long - in having to decide what news to print - been faced with content moderation. Finally, content moderation is inherently political. ModSquad - one of many booming companies in the social media moderation space - serves the US State Department, who makes no bones about the fact that it uses all elements of national power to affect what populations think and what access to information they have (it also runs the Bureau of International Information Programs - the State Department half of the US propaganda programme). Like newspapers, TV shows, cable companies, and radio programs before it - this new media sees itself filtering content for the public - with all of the complexities, power, and grief that entails.
- skybrian 11y ago
  Yep, filtering is inevitable. Lots of early Internet folks started out pro-free-speech and tried to be as hands-off as they could. But even if you're a stubborn libertarian, eventually you find out that it doesn't work. Sooner or later you're going to need a moderation team (or do it all yourself), and there are going to be tough political calls. The bigger the site, the more crap you see and the more tough calls there are.
jrapdx3 11y ago
This is a very good article and quite an eye opener. As an internet/web "user" for twenty years or so I'm familiar with the existence of "moderation", but had only vague awareness of what moderators endured in that role. Reminiscent of traumatized 911 operators, it would be unsurprising that a subset of moderators develop PTSD-like symptoms related to that experience. Workers' comments quoted in the article lead me to speculate that there are significant mental health epidemiological questions re: current and former moderators. An effective automated approaches will provide a big advantage in narrowing down the "judgement space" that must decided by humans. To the extent that's possible, the key benefit provided is reducing the exposure of moderators to stressful situations as the article describes, and indeed that's a very helpful development. As the article points out the whole domain of moderation practices is a minefield. But now I wonder if there isn't also a risk of automated review making classification errors re: user behavior. Plausibly automated systems can be tuned finely enough to avoid serious errors, and support human oversight to catch and more easily resolve edge cases. Automated moderation systems will need to have such qualities in order to be able to reduce human burden as intended.
- justifier 11y ago
  south park had a bit where they dealt with this sort of workspace ptsd butters was in charge of filtering cartman's comments so cartman would only see material that would be unable to offend him cartman seemed to flourish with his new perceived support and popularity and butters deteriorated i thought it compelling to look at how they chose to do butters 'this material is offensive! you should deal with it instead of me' cartman was characteristically unconcerned about butter's well being as long as the supportive comments kept coming in i understand the satire loses some weight when you acknowledge that those that hire these moderators are mostly companies with a vested interest in keeping their users blissfully unaware
  - staticfish 11y ago
    it's amazing how dead on-the-nail the southpark team do serious, humour ridden topical commentary on serious social and political issues. I think they'll go down as two of the greatest commentators of our generation.
James001 11y ago
That's just a beautiful layout on that page. Amazing typography and colour.
- pandler 11y ago
  The parallax was a bit too much for me though. I ended up opening the console and pausing the javascript.
  - Dr_tldr 11y ago
    They didn't do pure CSS parallax? I guess in this case it's for the best, but generally that's Doing It Wrong.
- Freak_NL 11y ago
  Glad you like it, but I actually stopped reading the article due to the atrocious typography and gimmicks. Those shadows behind the asides that move as you move your mouse are extremely distracting, and they overlap with the body text. For the body font they depend on users having Helvetica or Arial installed to render (neither of which I have) instead of using @font-face, so the text looks out of place. Also, none of the background images loaded for me until I tried with a browser without ad-blocker and third-party tracking protection.
  - giancarlostoro 11y ago
    I'm using noscript so it looks fine to me, the site's a bit broken though but that's fine by me, if you have FireFox you could always try "Reader View" to read the article without all the glitter.
    - elcapitan 11y ago
      It would actually be a nice feature if I could add certain domains to a "reader view" list in the browser, so that they always use that view directly. Medium, The Verge, all those giant blown up designs would be much more readable.
justifier 11y ago
i feel like this topic is a logic minefield seemingly least of all the question of under what criteria does censorship become moderation?
- dang 11y ago
  The difference may have more to do with whether you like a particular instance of it or not. "Censorship" is a pejorative and "curation" is an honorific and "moderation" is either that or neutral.
  - justifier 11y ago
    an example of the logic minefield i understand your sentiment, it is concise and well developed but saying 'censorship' is a pejorative of 'moderation', or worse 'curation', seems dangerously disingenuous censorship is a thing > the practice of officially examining books, movies, etc., and suppressing unacceptable parts. that definition lacks any sort of caveat of disapproval or association with moderation or curation i would suggest you can make the same point in more honest language by saying: moderation is censorship you agree with, or approve of, or tolerate, or enable
    - aptwebapps 11y ago
      Did 'dang edit his comment? Is there a material difference between "... whether you like a particular instance of it or not ..." and "... moderation is censorship you agree with, or approve of, or tolerate, or enable" ?
      - justifier 11y ago
        dang's comment is the same as it was when i responded the 'material difference' that i was trying to discuss was less about the first part of dang's comment and more about the second > "Censorship" is a pejorative and "curation" is an honorific and "moderation" is either that or neutral. calling censorship a pejorative, expressing contempt or disapproval, and calling moderation 'nuetral', to me, lends itself to the interpretation that the super set is moderation and censorship is a form of moderation you disapprove of 'censorship=-moderation' juxtaposed with 'moderation=+censorship' it's funny to have received a response from dang, the hn moderator, because i have this.. self censoring :p.. feeling that any opinion i express in response will somehow be associated with dang's work on this site i want to note, that though i am sure plenty of work goes on behind the scenes that i am unaware of, the times i have seen dang step into a thread and moderate explicitly it has been done with commendable tact and respect both for the community and the issue or user being addressed that said, > "Censorship" is a pejorative is a terrifying sentence to me
dang 11y ago
We've changed the title from the linkbait headline to the informative subtitle and re-upped* this post. I don't think we've ever done that for a story that already had 100+ upvotes, but this article is so astonishingly in-depth, and the topic so little known, that it deserves a deeper discussion. * Described at https://news.ycombinator.com/item?id=10705926 https://news.ycombinator.com/item?id=10705926 and the other links there.
- nkurz 11y ago
  Thanks, it's a great article that deserves even more attention. Is there still a plan to make public the list of re-upped articles? I think it would be a good resource. Please at least keep a log of them for future use.
- nickpsecurity 11y ago
  I appreciate you doing that. This article is phenomenal work as it's so thoroughly researched with so many perspectives and details published. It's such a void of a topic to most despite the fact that it actively shapes democracy, innovation, and evolution of human culture. I might have to re-read it a few times to let it all sink in haha.
- jrapdx3 11y ago
  I entirely concur with both your points, especially about how little appreciated this subject has been. I was dimly aware of it, I mean it's no secret there's a lot of really terrible stuff on the web, and it's shocked me when I've stumbled into it. It's disquieting to realize I never thought about what it takes to keep a site like YouTube from being a complete cesspool. Never crossed my mind, but the question should've occurred to me, why didn't it? I wish I had an answer. Of course, now I know how good a job moderators have been doing, and better understanding of the toll that role can take. They deserve a lot of respect. The idea this is a subject worthy of systematic study sticks with me though not yet clear about the next steps. Any case, the implications of the story cover a large space in economics, politics, ethics, law, and medicine. It will be fascinating to see how it develops.
- Mz 11y ago
  Thank you. You do good work around here.
mc32 11y ago
Flickr seems to address the issues in a slightly different way. When uploading, you self "sensor" (classify) in that you can classify uploads as "mature" or more run of the mill. Plus you have groups which provide guidelines and have group moderators who may bump things out of the pool, ban people, etc. In addition, users can flag content and moderators can appeal to flickr to ban users for harassment, etc. So, while exploitation, criminal activity, etc. can still be problematic, the issue of free speech as it pertains politics is less of an issue because you can upload the content to a group which has group policies allowing such content. So, for example, you might find a group which is welcoming of videos pictures showing police brutality, or the like or gang brutality [so long as the content isn't an excuse to portray something else] All this is to say Flickr allows/allowed for more grass roots approach to moderation, other than meta moderation where the usual rules apply --ie to criminal acts, exploitation, etc.
dba7dba 11y ago
1. Breaking rules When the article starts out with the story of Youtube, you read about the lists the moderators working off of to keep certain materials out of Youtube. And something was missing on the list. Any guess? Yes, copyrighted material. I knew a guy who used to work at a competitor of Youtube. That startup was started by a Hollywood veteran (naturally), meaning he was very conscious of respecting copyrights. Youtube didn't care about copyrights, which started the upward spiral of more viewers, more uploads, and more viewers. The end story is that Youtube took off, being bought off at a billion and making the founders rich. Mixed feelings about this. Break rules to beat others in the game, and you end up winning. And most will agree this is not cool. And yet, because of rule breakers (some, not all), our world is advancing. A comment I read in a comment section, that I saw that was funny. "When someone else cheats, it's adultery. When I cheat, it's romance" After this this story, I've come up with following: "When someone else break rules, it's cheating. When I break rules, it's innovation" 2. No chance for moderation And then there's the other part in the world of moderation, in which certain stories are not even given a chance for moderators to review. NYT (New York Times) has abundant and quality comments on news articles. However, if you hang out there long enough, you will notice a trend. Certain stories that don't help the goal of NYT don't get comments section at all. And if a story's comment section gets filled up with comments that seem to be hurting the agenda of that particular article, the comments section closes rather quickly. And then there's Fox News. They don't even allow any way to post comments. This kind of social engineering has been going on since humans became social animals, but with technology in the mix, those in power get more powerful.
- erichocean 11y ago
  > "When someone else break rules, it's cheating. When I break rules, it's innovation" IIRC pg wrote an essay defending that concept, more-or-less (and where by "I", he meant "startups").
- CM30 11y ago
  Your point about breaking the rules to become successful is an old, old trick by community managers. I knew an awful lot of large forums that originally took off by posting ROMs/torrents/music/film downloads and then carefully excised them when the site had tens of thousands of members and might actually be visible to the copyright owners. It's a pretty good way to become large rather quickly, since illegal or legal questionable content brings in a lot of visitors. Then you just remove it, and watch the network effect/inflated stats lure people to your site. "IIRC pg wrote an essay defending that concept, more-or-less (and where by "I", he meant "startups"). " I also heard it mentioned as 'asking for forgiveness rather than permission'. Or more cynically, get big enough that you can afford to fight the inevitable lawsuits.
hackuser 11y ago
This seems like a trending topic. Some related materials: * The dark side of Guardian comments https://news.ycombinator.com/item?id=11478361 https://news.ycombinator.com/item?id=11478361 * Why has the Guardian declared war on internet freedom? http://www.spiked-online.com/newsite/article/why-has-the-guardian-declared-war-on-internet-freedom/18247 http://www.spiked-online.com/newsite/article/why-has-the-gua... * The New Man of 4chan https://news.ycombinator.com/item?id=11510758 https://news.ycombinator.com/item?id=11510758
paganel 11y ago
Excellent article. Just the paragraph bellow deserves its own extended commentary: > The screener was instructed to take down videos depicting drug-related violence in Mexico, while those of political violence in Syria and Russia were to remain live. This distinction angered him. Regardless of the country, people were being murdered in what were, in effect, all civil wars. "[B]asically," he said, "our policies are meant to protect certain groups." Among other things it puts Putin's decision to grab VKontakte from its founder in a new light. Presumably videos of Mexican violence (supported by American guns and by the Americans' desire for drugs) are not banned on VK, while Russian violence I suppose is now easily banned. It also makes him look less like a paranoid for accusing the US special services of having "created the Internet" (http://www.theguardian.com/world/2014/apr/24/vladimir-putin-web-breakup-internet-cia http://www.theguardian.com/world/2014/apr/24/vladimir-putin-...)
hackuser 11y ago
This part is troubling: In May 2014, Dave Willner and Dan Kelmenson, a software engineer at Facebook, patented a 3D-modeling technology[1] for content moderation ... First, the model identifies a set of malicious groups - say neo-Nazis, child pornographers, or rape promoters. The model then identifies users who associate with those groups through their online interactions. Next, the model searches for other groups associated with those users and analyzes those groups "based on occurrences of keywords associated with the type of malicious activity and manual verification by experts." This way, companies can identify additional or emerging malicious online activity It's not hard to imagine how this could be misused: Automated guilt-by-association, and within 3 degrees of separation, triggering investigation by authorities (corporate or otherwise). EDIT: And also this tool of automated censorship, though I doubt it will surprise many HN readers PhotoDNA works by processing an image every two milliseconds and is highly accurate. ... Then PhotoDNA extracts from each image a numeric signature that is unique to that image, "like your human DNA is to you." Whenever an image is uploaded, whether to Facebook or Tumblr or Twitter, and so on, he says, "its photoDNA is extracted and compared to our known ... images. Matches are automatically detected by a computer and reported ... for a follow-up investigation" Currently used for child exploitation images, but for what else too and by whom? They add that it's being applied to something more political: Farid is now working with tech companies and nonprofit groups to develop similar technology that will identify extremism and terrorist threats online - whether expressed in speech, image, video, or audio. [1] http://www.google.com.gh/patents/US20080256602 http://www.google.com.gh/patents/US20080256602
- CM30 11y ago
  This is a really good point, and this sort of thing could well be happening already. There has already been controversy over negative posts related to immigration being removed from Facebook due to a deal with the government. And there's already been talk of people inside Facebook asking whether they're 'doing enough' to stop a Donald Trump presidency: http://gizmodo.com/facebook-employees-asked-mark-zuckerberg-if-they-should-1771012990 http://gizmodo.com/facebook-employees-asked-mark-zuckerberg-... As well as cases where people were visited by police after making right wing comments online: http://www.breitbart.com/london/2016/04/07/police-raid-social-media-posts/ http://www.breitbart.com/london/2016/04/07/police-raid-socia... (Yes, I know it's from Breitbart, but the point still stands) And there's a lot of talk about Twitter censorship, blockbots blocking people based on what accounts they follow rather than any actions and all that stuff. Similar systems are already being abused by social networks, and this type of technology will only make cases like this more common.
deleted 11y ago
[deleted]
cb18 11y ago
Many large(and small) subreddits are highly censored. r/worldnews r/news r/europe to name a few. Here's a recent example, not even the most egregious, just a recent one. https://www.reddit.com/r/worldnews/comments/4ew68z/half_of_all_jobless_in_sweden_are_foreigners/ https://www.reddit.com/r/worldnews/comments/4ew68z/half_of_a... Noticed how many xx deep comment threads voted high up in the comment stack have been excised from the discussion. It's disgusting. It's very often the case that the top voted contributions have been 'disappeared.' Again, just disgusting. What exactly is the point of a 'democratic' news site, if there is this constant intervention from unaccountable authorities, constantly policing what information and opinions are allowed to be discussed. Any information outside the scope of a narrow ideological agenda is summarily terminated. Is this the public square of the future that we want? kn0thing, spez, What are your thoughts?
- DanBC 11y ago
  ...and yet when anyone talks about a good subreddit it's almost always a sub that has strict rules and vigorous moderation, or a tiny sub that has a tiny number of users.
  - cb18 11y ago
    Well, I'm saying those are bad subs, because they have 'vigorous' moderation. So subs with 'strict, vigorous moderation' is not a feature that automatically connotes, good sub.
    - azernik 11y ago
      Well, they're the subreddits that get cited whenever anyone talks about the glorious goodness of reddit: /r/ama, /r/iama, /r/askscience, /r/syriancivilwar (cited by news organizations as a source and informing their methodology)... Moderation doesn't automatically connote good subs, but the correlation is very strong.
- braythwayt 11y ago
  And yet, you can start your own damn subreddit. You can talk about anything you want, even violent, openly racist stuff. You can decide not to moderate it at all. So clearly, you can talk about nearly anything you want on reddit. Your problem is that you want to be able to talk about anything you want in somebody else’s subreddit. It’s as if you move onto my street, live in a perfectly good house, but complain about the fact that you can’t do whatever you like in my house. Freedom to say whatever you want is not the same thing as forcing other people to pay attention to it. You need to either find like-minded people moderating a like-minded subreddit, or find a way to get other people interested in your own subreddit. This business of demanding that everybody else give you a platform because “freedom of speech” is flat-out wrong.
rustynails 11y ago
I find The Verge extremely offensive. To pick a few bytes from This article, "Abusive men threaten spouses" (aka domestic violence is a man thing), "social media and mobile tech cause real harm to women". At least The verge are consistent. They have been sexist for as long as I have known them and their campaigns directed squarely at males (regardless of age). When The Verge viciously attacked Matt Taylor, I felt ill for days. I could not believe that a media outlet was so callous in its aim to destroy a person and their career ... . Then, they posted an article about rape on campus. Rolling Stone pulled the referred article (the woman who wrote the Rolling Stone article had a history of lying and she wasn't a student where she said she was, the university acknowledged incidents but denied ever speaking to this woman, etc). The verge was infuriated as the media's suppression (regardless of the truth) because it matched a picture they were trying to paint. This now explains why the media are so sexist - it's no accident that domestic violence is supposedly "men only". It's no surprise that boys are actively discriminated against in schools with STEM. It's linked to these moderators described in the article. It's not the moderation itself but the personal prejudices of the leaders of the moderators. The sexism and manipulation I see today on the internet is unprecedented in my lifetime. In the 60s, I never saw a teacher ignore students based on gender - but today I hear about boys frequently being ignored "to level the playing field" because the media tell them it's ok and worse, necessary. I spoke with one teacher recently because he was directed to discriminate against boys and he refused to discriminate and resigned in protest. I've had other teachers (male and female) strongly hint this happens frequently. This ruins children's lives! I've essen these policies in action on my own children too. The question I constantly asked myself was why. This article on moderation was my theory and now it confirms what I thought. The question is how do you challenge these manipulative propagandists bent on gender wars who selectively filter the content that gets shown? Decent people need to take an active stand. I'm glad to see there is some pushback against the feminist propaganda in the last few weeks. People are staring to question "supposed facts" and selective language such as "domestic violence" is a great example. This Verge article uses the same manipulative formulas that we continue to see (men:bad). I implore everyone to raise the awareness of the extreme prejudice of feminism and to oppose it assertively without threats. People need to denounce prejudice by anyone (despite what political correctness says). Feminism doesn't deserve protection in the way that the KKK don't deserve it. You aren't born a feminist or KKK, but you sure can choose to propagate the intolerance. I took my stance when I saw that young boys are impacted by a gender war that never should have started.
vonnik 11y ago
This is one example of a job where neural networks/AI can, and should, replace humans. Video and image classification is basically a solved problem. Numerous companies have already trained deep convolutional networks to recognize pornography and other forms of content unwanted on their platforms.
- CM30 11y ago
  Maybe I'm missing something here, but I suspect this would be a technical and political minefield. Sure, you can recognise certain types of abuse images, but then what about content like that in fiction? It's very logical that something seems as problematic in real life would be fine in say, a Call of Duty game or a blockbuster movie or some other form of fiction. Heck, there have been cases where scenes in movies and games have been 'reappropriated' as real life military or terrorist events by clueless nations and groups. For example: http://www.shacknews.com/article/93359/footage-of-six-isis-soldiers-killed-by-sniper-is-really-from-medal-of-honor http://www.shacknews.com/article/93359/footage-of-six-isis-s... So your system would have to figure out not just whether something is seen as 'offensive' or 'against the rules', but whether its from a fictional work that might be allowed on the site.
schoen 11y ago
This article made me wonder about the range of things that these companies are dealing with under the rubric of "moderation" or "trust and safety", particularly the way that companies can have so many different kinds of incentives to remove content from a platform. (Three that come to mind are "this may create legal liability", "this may damage our brand", and "this isn't what many of our users would like to see".) If the Internet or the way that people commonly use it is going to continue to get more centralized, I hope platforms improve their ability to distinguish between the problem of "things people don't want to see" and "things people don't want other people to see", or, to put it another way, between "please don't show me things like this" and "please don't allow people to publish things like this". While users themselves might not consistently draw the distinction between the two, platforms, in principle, could. The article touched in this issue in its discussion of the extreme variation in cultural norms, and the likelihood that one culture's vulgarity is another culture's lyr—er, that one culture's distaste for something will lead to considerable pressure for platforms to squelch it for everybody. A common idea is that platforms have a right and even a responsibility to define their own community standards and then people can choose platforms that they prefer or that best suit them, much as people can choose the newspaper whose editorial policy and biases they find most agreeable. I think this notion has a lot to recommend it but it seems less comfortable as the patterns of people's usage of the Internet becomes ever more centralized; it also raises the question of what things can be considered infrastructural enough that people can reasonably expect (or at least accept) complete content-neutrality from them. The article also provided an interesting reminder that most people are unlikely to be comfortable using communications systems that have no moderation or filtering at all. Among other things, that's an interesting challenge for decentralized and censorship-resistant systems; to become more popular and practical, they'll need to be paired with some ways that people can avoid unwanted communication, beyond spam.
scotty79 11y ago
I don't think perfect moderation to some global average morality is the future of humanity. It's TV mentality. I think the future is keeping vulnerable groups apart from offensive groups. Hellbaning and honeypot bots are the future. White supremacist can talk to other like him all day long and there's no harm done. Ones that would like to talk to black people or just people who get offended by racism should get algorithmically administered silence or honeypot bots that will react in best way possible to defuse and calm down without engaging human in distasteful content. Plus side of this solutio is that you can keep tabs on potentially dangerous people and react if they escalate or brag about physical harm they do.
- rodgerd 11y ago
  > White supremacist can talk to other like him all day long and there's no harm done. Right up until one of them decides it's time to leave his little echo chamber and murder a bunch of kids on an island because he and his Internet buddies have convinced themselves that's the right thing to do.
  - michaelmrose 11y ago
    Between 1984 and bad people doing bad things I choose the later with no regrets.
    - scotty79 11y ago
      Worst thing in 1984 was not surveilance but rewriting the truth. You can't have progress if you are not open and honest. Pretending that there's just one morality that everyone normal must adhere to is more 1984 than any of my ideas.
  - scotty79 11y ago
    You'd rather prefer they'd coordinate their attack on a channel you don't monitor? Do you think you can stop stupid people from doing harmful things by banning them from expressing their stupid opinions online? Brevik is exception not the rule and it could be all avoided if he was allowed to speak freely and discuss his demented ideas with people that don't mind so he can be monitored.
microcolonel 11y ago
>Their stories reveal how the boundaries of free speech were drawn during a period of explosive growth for a high-stakes public domain, one that did not exist for most of human history. I feel like this is a pretty big jump to make. There's a difference between deciding not to publish and amplify child sexual assault, and the "boundaries" of free speech. Whether their speech is free or not is irrelevant; YouTube has no obligation to publish it.
michaelmrose 11y ago
Whomever is responsible for the boxes that follow your mouse ought to have their commit privileges revoked.
- takno 11y ago
  Oh, is that what was meant to be happening? I was reading on mobile and the page just kept randomly scrolling, and when I came back to it after 8 hours a bunch of boxes had spilled semi transparent shadows over the text. It's funny how 20 years of improving tech have led us to a situation where we now just have to ignore huge layout issues and people shouting incredibly unpleasant things
  - yomly 11y ago
    Glad someone else took issue with this. We're in a constant race to improve battery life and processing power so we can waste energy on these pointless "innovations". I'm all for improving UX, but I constantly wonder how much longer our batteries would last if they weren't being drained by gratuitous features like this one...
- bobwaycott 11y ago
  On mobile, it appears to be tied to the gyroscope or accelerometer. They only move as I tilt/move my phone. Keeping it fixed and upright removed the floating boxes. Very strange. It took me 2/3 of the article to even figure out it was my movement making them move.
firebones 11y ago
I wonder if there could be a protection ring [1] model of moderation and content classification--one that protected business interests yet classified objectionable material based on content while letting service providers retain common carrier status. In such a world, service providers would choose a level of filtering that fit their business needs, but would also let the free market decide what minimum standards were acceptable. YouTube's moderation expenses would simply focus on keeping them at level X (with a clear separation from some other site). Consumers, producers, and law enforcement would simply tune their dial to the level of content they found acceptable. It seems like it would promote competition while effectively pricing free speech. [1] https://en.wikipedia.org/wiki/Protection_ring https://en.wikipedia.org/wiki/Protection_ring
halugalugh 11y ago
This kind of censorship is the USAs version of Chinese censorship. Self imposed first amendment restriction. Very troubling.
Joeboy 11y ago
What is going on with Facebook's moderation of Kurdish / anti-Turkish content? Why is it that according to the leaked standards doc[1] (which as far as I can tell is genuine) Turkey gets to ban stuff it doesn't like on Facebook? Presumably it's OK to post a map showing Palestine, or a United Ireland, or other disputed regions on FB. Why can't you post a map of Kurdistan? Aside from Holocaust denial, no other political position seems to have been singled out in this way. [1] http://i.dailymail.co.uk/i/pix/2012/02/21/article-2104424-11D8ADDB000005DC-298_964x733.jpg http://i.dailymail.co.uk/i/pix/2012/02/21/article-2104424-11...
- tim333 11y ago
  I'm guessing here but Turkey/Erdogan are particularly aggressive in attacking stuff they don't like. See for example Germany being made to prosecute one of their comedians and "In 2012 the Committee to Protect Journalists (CPJ) ranked Turkey as the worst journalist jailer in the world (ahead of Iran and China), with 49 journalists sitting in jail." Also more ammusingly the doctor facing 5 years for posting pictures of Erdogan next to pictures of Gollum http://www.theguardian.com/world/2015/dec/03/lord-of-rings-director-insult-to-erdogan-mistaken-as-gollum-as-charcter-is http://www.theguardian.com/world/2015/dec/03/lord-of-rings-d... They are probably threatening to arrest people / ban Facebook if there are any maps of Kurdistan posted.
arca_vorago 11y ago
I've posted before about my theories regarding optimum commenting and moderation systems, but one thing I've recently been thinking about is some sort of logic/rationality analysis engine (perhaps through nlp/ml?). What I really want is to wade through the extra verbal fluff of a comment and get to its real points, so that I can determine if they are rational and logical. My primary method is by looking for fallacies which are tell-tell signs of a bad argument. That still doesnt "fix" moderation, but I think such a system could be used to get rid of consistently illogical trolls to reduce the moderator workload at least. Ideally, I imagine exclusivity of posting to be barrier 1, and moderation to be barrier 2. Some strange combinantion of hn Rules and /. Randomized moderation along with /. Tagging styles, would probably be the best. Another factor to consider is that sockpuppetry has thrown the democratic balance of such systems off, and Im not yet sure how to deal with that.
- intended 11y ago
  The assumption being that forums and trolls are static. Trolls adapt and with enough innuendo, the innocuous will be loaded with dark meaning. Plus someone has to wade through the sea of false positives.
rurban 11y ago
Oh, that Neda video was uploaded by me to Youtube, because the Iranians who brought it out of the country couldn't. They never talked to me about it.
mbrutsch 11y ago
Little different from any historical thought police - we know what's best for you, citizen, move along, nothing to see here.
- bobwaycott 11y ago
  This is vastly different from historical/fictional thought-policing. Did you read the article and really wind up with this takeaway?
bobwaycott 11y ago
Wow. There is just so much in this article to wrap one's head around. I feel somewhat ashamed that I've never really thought about moderators at all, much less of the psychological and emotional effects they must endure as a result of the content they must see. I know it's work I wouldn't want to perform. I'm grateful for this investigation and report. There's a ton to unpack and think through.
pessimizer 11y ago
I didn't even realize that all of these sites were deleting racist and "hate" speech. That's totally reprehensible. Any site that would do that would also delete blasphemy and violations of lese majeste. The worst part is that they justify it with philosophical jumbles like "open policies [are] stifling free expression," as if one person speaking prevents another from also speaking. edit: also, the "trillion or so dollars of value" that an "expert" ascribed to Section 230 is simply a euphemism for ads. Nothing else. edit2: Wow. I just realized that the reason I've stopped getting racist hits on search engines in recent years is because they've been removed. Part of the range of discourse disappeared and I didn't even realize it. That's also got to be the reason that comment sections in papers like the Boston Globe, Chicago Tribune, and Washington Post have become racist cesspools. Forums where they could have their mutual appreciation societies away from normal people have been completely hidden from view. Am I the only person who didn't think the internet was broken before all of this arbitration?
brashrat 11y ago
I thought this headline was referring to what I think is a more insidious form of moderation, the type that's practiced here on ycombinator with regard to style, for example, negativity is punished broadly here. The reason I find it more insidious is that it affects the speech of and acts to silence even moderate people. Extreme speech is easy to spot; but when moderate speech encounters headwinds that divert its path, that's creepy. I understand the goals here, don't need them explained, just pointing out that slashdot in its heyday was a terrific and informative resource and it did not resort to stifling.

263 points by pmcpinto 11y ago

This makes the criticisms of Wikipedia look pretty bland. I had never thought about what life in these trenches could look like. Trenches indeed.

dredmorbius 11y ago
Realise that pretty much anything that hits YouTube or FB also hits Wikpedia. (I'm trying hard to avoid making plays on "pediwikia" without success. And not faulting Wikipedia but rather those who'll post that kind of content, or other rot, if they can.)
dclowd9901 11y ago
Seriously. I can't count the number of times a day I see something marked NSFL and think, "nope, not even gonna look." But these people have to. I wouldn't wish that on anyone.

proksoup 11y ago

> The details of moderation practices are routinely hidden from public view, siloed within companies and treated as trade secrets when it comes to users and the public. Despite persistent calls from civil society advocates for transparency, social media companies do not publish details of their internal content moderation guidelines; no major platform has made such guidelines public. It's really nice to see light shining on this topic. Truly, this content moderation is the equivalent (equivalent for the internet medium) of the boundaries of debate enforced by the editors and owners of major media corporations on our previous mediums.

dmckeon 11y ago
If the details of moderation rules were public, companies would be facing additional labor costs dealing with posters pushing hard on the edges of those rules. At some level, users can choose from a variety of forums with different moderation methods, or none, and companies can try to fit their moderation policies and practices to their users' expectations. All of that develops in a cross between evolution and arms race, with lots of moving targets - global variation, changing social mores, changing technology, the latest taboos and moral panics. Exposing the details of how the sausage is made would help the people who want to put pebbles (or worse) in the sausage more than it would help the makers or consumers. If one finds the sausage too bland, there is likely a spicier one just a few clicks away. Or one could grind one's own sausage and compete!
- proksoup 11y ago
  What you describe sounds like security by obscurity.
  - dmckeon 11y ago
    Publishing moderation policy details would be like crowd-sourcing cryptography software development - creating a fractally increased attack surface with a less profitable business model. Social media companies operate in a market economy. These are not public utilities or government programs. Feel free to propose a more cost-effective approach.
    - proksoup 11y ago
      I think exposing the rules will make them stronger.
      - intended 11y ago
        Moderation is not cyber security. Virii don't turn around and argue that your AV is unfairly applying their rules to them.
  - dang 11y ago
    I think the rules are quite different when dealing with human processes rather than algorithmic ones.
    - proksoup 11y ago
      Could you elaborate, the same principle seems to apply to me, the costs of dealing with people attacking the edge strengthens the edge, no?
  - ihnorton 11y ago
    Or plausible deniability.
- parenthephobia 11y ago

deleted 11y ago

[deleted]

ckelly 11y ago

> In April of 2005, they tested their first upload. By October, they had posted their first one million-view hit: Brazilian soccer phenom Ronaldinho trying out a pair of gold cleats. Weeks later, Google paid an unprecedented $1.65 billion to buy the site. This article misstates when Google acquired YouTube. It was October 2006, not October 2005: https://en.wikipedia.org/wiki/YouTube#Company_history https://en.wikipedia.org/wiki/YouTube#Company_history

a301 11y ago
They also mistake the traffic stats for /r/AskReddit as being traffic for all of Reddit: > According to a source close to the moderation process at Reddit, the climate there is far worse. Despite the site’s size and influence — attracting some 4 to 5 million page views a day[1] — Reddit has a full-time staff of only around 75 people, leaving Redditors to largely police themselves [1] https://www.reddit.com/r/AskReddit/about/traffic https://www.reddit.com/r/AskReddit/about/traffic
- reycharles 11y ago
  > last month, reddit had 243,632,148 unique visitors hailing from over 212 different countries viewing a total of 8,137,128,592 pages Which is about 270 million page views per day. https://www.reddit.com/about/ https://www.reddit.com/about/
- takno 11y ago
  Given that they don't have a large marketing team, 75 people would be a huge team to deal with 4 to 5 million page views
giancarlostoro 11y ago
I'm glad you caught this too, I thought it was WAY too soon, I knew YouTube when it had relatively grown in popularity and quite a bit before Google's acquisition of it.
takno 11y ago
By the time I read it this had been corrected and acknowledged in the footnotes, so I guess it's good to know the system works

xnull2guest 11y ago

Many of the problems faced by Facebook, Youtube and others is that they own the digital commons and manage and curate it for the people trying to communicate with one another. These issues come up for companies whose business models are deciding what people should see and hear. This makes them incredibly powerful, but it brings them into the bog. This isn't a new problem, other than internet scale. Media companies have long - in having to decide what news to print - been faced with content moderation. Finally, content moderation is inherently political. ModSquad - one of many booming companies in the social media moderation space - serves the US State Department, who makes no bones about the fact that it uses all elements of national power to affect what populations think and what access to information they have (it also runs the Bureau of International Information Programs - the State Department half of the US propaganda programme). Like newspapers, TV shows, cable companies, and radio programs before it - this new media sees itself filtering content for the public - with all of the complexities, power, and grief that entails.

skybrian 11y ago
Yep, filtering is inevitable. Lots of early Internet folks started out pro-free-speech and tried to be as hands-off as they could. But even if you're a stubborn libertarian, eventually you find out that it doesn't work. Sooner or later you're going to need a moderation team (or do it all yourself), and there are going to be tough political calls. The bigger the site, the more crap you see and the more tough calls there are.

jrapdx3 11y ago

This is a very good article and quite an eye opener. As an internet/web "user" for twenty years or so I'm familiar with the existence of "moderation", but had only vague awareness of what moderators endured in that role. Reminiscent of traumatized 911 operators, it would be unsurprising that a subset of moderators develop PTSD-like symptoms related to that experience. Workers' comments quoted in the article lead me to speculate that there are significant mental health epidemiological questions re: current and former moderators. An effective automated approaches will provide a big advantage in narrowing down the "judgement space" that must decided by humans. To the extent that's possible, the key benefit provided is reducing the exposure of moderators to stressful situations as the article describes, and indeed that's a very helpful development. As the article points out the whole domain of moderation practices is a minefield. But now I wonder if there isn't also a risk of automated review making classification errors re: user behavior. Plausibly automated systems can be tuned finely enough to avoid serious errors, and support human oversight to catch and more easily resolve edge cases. Automated moderation systems will need to have such qualities in order to be able to reduce human burden as intended.

justifier 11y ago
south park had a bit where they dealt with this sort of workspace ptsd butters was in charge of filtering cartman's comments so cartman would only see material that would be unable to offend him cartman seemed to flourish with his new perceived support and popularity and butters deteriorated i thought it compelling to look at how they chose to do butters 'this material is offensive! you should deal with it instead of me' cartman was characteristically unconcerned about butter's well being as long as the supportive comments kept coming in i understand the satire loses some weight when you acknowledge that those that hire these moderators are mostly companies with a vested interest in keeping their users blissfully unaware
- staticfish 11y ago
  it's amazing how dead on-the-nail the southpark team do serious, humour ridden topical commentary on serious social and political issues. I think they'll go down as two of the greatest commentators of our generation.

James001 11y ago

That's just a beautiful layout on that page. Amazing typography and colour.

pandler 11y ago
The parallax was a bit too much for me though. I ended up opening the console and pausing the javascript.
- Dr_tldr 11y ago
  They didn't do pure CSS parallax? I guess in this case it's for the best, but generally that's Doing It Wrong.
Freak_NL 11y ago
Glad you like it, but I actually stopped reading the article due to the atrocious typography and gimmicks. Those shadows behind the asides that move as you move your mouse are extremely distracting, and they overlap with the body text. For the body font they depend on users having Helvetica or Arial installed to render (neither of which I have) instead of using @font-face, so the text looks out of place. Also, none of the background images loaded for me until I tried with a browser without ad-blocker and third-party tracking protection.
- giancarlostoro 11y ago
  I'm using noscript so it looks fine to me, the site's a bit broken though but that's fine by me, if you have FireFox you could always try "Reader View" to read the article without all the glitter.
  - elcapitan 11y ago
    It would actually be a nice feature if I could add certain domains to a "reader view" list in the browser, so that they always use that view directly. Medium, The Verge, all those giant blown up designs would be much more readable.

justifier 11y ago

i feel like this topic is a logic minefield seemingly least of all the question of under what criteria does censorship become moderation?

dang 11y ago
The difference may have more to do with whether you like a particular instance of it or not. "Censorship" is a pejorative and "curation" is an honorific and "moderation" is either that or neutral.
- justifier 11y ago
  an example of the logic minefield i understand your sentiment, it is concise and well developed but saying 'censorship' is a pejorative of 'moderation', or worse 'curation', seems dangerously disingenuous censorship is a thing > the practice of officially examining books, movies, etc., and suppressing unacceptable parts. that definition lacks any sort of caveat of disapproval or association with moderation or curation i would suggest you can make the same point in more honest language by saying: moderation is censorship you agree with, or approve of, or tolerate, or enable
  - aptwebapps 11y ago
    Did 'dang edit his comment? Is there a material difference between "... whether you like a particular instance of it or not ..." and "... moderation is censorship you agree with, or approve of, or tolerate, or enable" ?
    - justifier 11y ago
      dang's comment is the same as it was when i responded the 'material difference' that i was trying to discuss was less about the first part of dang's comment and more about the second > "Censorship" is a pejorative and "curation" is an honorific and "moderation" is either that or neutral. calling censorship a pejorative, expressing contempt or disapproval, and calling moderation 'nuetral', to me, lends itself to the interpretation that the super set is moderation and censorship is a form of moderation you disapprove of 'censorship=-moderation' juxtaposed with 'moderation=+censorship' it's funny to have received a response from dang, the hn moderator, because i have this.. self censoring :p.. feeling that any opinion i express in response will somehow be associated with dang's work on this site i want to note, that though i am sure plenty of work goes on behind the scenes that i am unaware of, the times i have seen dang step into a thread and moderate explicitly it has been done with commendable tact and respect both for the community and the issue or user being addressed that said, > "Censorship" is a pejorative is a terrifying sentence to me

dang 11y ago

We've changed the title from the linkbait headline to the informative subtitle and re-upped* this post. I don't think we've ever done that for a story that already had 100+ upvotes, but this article is so astonishingly in-depth, and the topic so little known, that it deserves a deeper discussion. * Described at https://news.ycombinator.com/item?id=10705926 https://news.ycombinator.com/item?id=10705926 and the other links there.

nkurz 11y ago
Thanks, it's a great article that deserves even more attention. Is there still a plan to make public the list of re-upped articles? I think it would be a good resource. Please at least keep a log of them for future use.
nickpsecurity 11y ago
I appreciate you doing that. This article is phenomenal work as it's so thoroughly researched with so many perspectives and details published. It's such a void of a topic to most despite the fact that it actively shapes democracy, innovation, and evolution of human culture. I might have to re-read it a few times to let it all sink in haha.
jrapdx3 11y ago
I entirely concur with both your points, especially about how little appreciated this subject has been. I was dimly aware of it, I mean it's no secret there's a lot of really terrible stuff on the web, and it's shocked me when I've stumbled into it. It's disquieting to realize I never thought about what it takes to keep a site like YouTube from being a complete cesspool. Never crossed my mind, but the question should've occurred to me, why didn't it? I wish I had an answer. Of course, now I know how good a job moderators have been doing, and better understanding of the toll that role can take. They deserve a lot of respect. The idea this is a subject worthy of systematic study sticks with me though not yet clear about the next steps. Any case, the implications of the story cover a large space in economics, politics, ethics, law, and medicine. It will be fascinating to see how it develops.
Mz 11y ago
Thank you. You do good work around here.

mc32 11y ago

Flickr seems to address the issues in a slightly different way. When uploading, you self "sensor" (classify) in that you can classify uploads as "mature" or more run of the mill. Plus you have groups which provide guidelines and have group moderators who may bump things out of the pool, ban people, etc. In addition, users can flag content and moderators can appeal to flickr to ban users for harassment, etc. So, while exploitation, criminal activity, etc. can still be problematic, the issue of free speech as it pertains politics is less of an issue because you can upload the content to a group which has group policies allowing such content. So, for example, you might find a group which is welcoming of videos pictures showing police brutality, or the like or gang brutality [so long as the content isn't an excuse to portray something else] All this is to say Flickr allows/allowed for more grass roots approach to moderation, other than meta moderation where the usual rules apply --ie to criminal acts, exploitation, etc.

dba7dba 11y ago

1. Breaking rules When the article starts out with the story of Youtube, you read about the lists the moderators working off of to keep certain materials out of Youtube. And something was missing on the list. Any guess? Yes, copyrighted material. I knew a guy who used to work at a competitor of Youtube. That startup was started by a Hollywood veteran (naturally), meaning he was very conscious of respecting copyrights. Youtube didn't care about copyrights, which started the upward spiral of more viewers, more uploads, and more viewers. The end story is that Youtube took off, being bought off at a billion and making the founders rich. Mixed feelings about this. Break rules to beat others in the game, and you end up winning. And most will agree this is not cool. And yet, because of rule breakers (some, not all), our world is advancing. A comment I read in a comment section, that I saw that was funny. "When someone else cheats, it's adultery. When I cheat, it's romance" After this this story, I've come up with following: "When someone else break rules, it's cheating. When I break rules, it's innovation" 2. No chance for moderation And then there's the other part in the world of moderation, in which certain stories are not even given a chance for moderators to review. NYT (New York Times) has abundant and quality comments on news articles. However, if you hang out there long enough, you will notice a trend. Certain stories that don't help the goal of NYT don't get comments section at all. And if a story's comment section gets filled up with comments that seem to be hurting the agenda of that particular article, the comments section closes rather quickly. And then there's Fox News. They don't even allow any way to post comments. This kind of social engineering has been going on since humans became social animals, but with technology in the mix, those in power get more powerful.

erichocean 11y ago
> "When someone else break rules, it's cheating. When I break rules, it's innovation" IIRC pg wrote an essay defending that concept, more-or-less (and where by "I", he meant "startups").
CM30 11y ago
Your point about breaking the rules to become successful is an old, old trick by community managers. I knew an awful lot of large forums that originally took off by posting ROMs/torrents/music/film downloads and then carefully excised them when the site had tens of thousands of members and might actually be visible to the copyright owners. It's a pretty good way to become large rather quickly, since illegal or legal questionable content brings in a lot of visitors. Then you just remove it, and watch the network effect/inflated stats lure people to your site. "IIRC pg wrote an essay defending that concept, more-or-less (and where by "I", he meant "startups"). " I also heard it mentioned as 'asking for forgiveness rather than permission'. Or more cynically, get big enough that you can afford to fight the inevitable lawsuits.

hackuser 11y ago

This seems like a trending topic. Some related materials: * The dark side of Guardian comments https://news.ycombinator.com/item?id=11478361 https://news.ycombinator.com/item?id=11478361 * Why has the Guardian declared war on internet freedom? http://www.spiked-online.com/newsite/article/why-has-the-guardian-declared-war-on-internet-freedom/18247 http://www.spiked-online.com/newsite/article/why-has-the-gua... * The New Man of 4chan https://news.ycombinator.com/item?id=11510758 https://news.ycombinator.com/item?id=11510758

paganel 11y ago

Excellent article. Just the paragraph bellow deserves its own extended commentary: > The screener was instructed to take down videos depicting drug-related violence in Mexico, while those of political violence in Syria and Russia were to remain live. This distinction angered him. Regardless of the country, people were being murdered in what were, in effect, all civil wars. "[B]asically," he said, "our policies are meant to protect certain groups." Among other things it puts Putin's decision to grab VKontakte from its founder in a new light. Presumably videos of Mexican violence (supported by American guns and by the Americans' desire for drugs) are not banned on VK, while Russian violence I suppose is now easily banned. It also makes him look less like a paranoid for accusing the US special services of having "created the Internet" (http://www.theguardian.com/world/2014/apr/24/vladimir-putin-web-breakup-internet-cia http://www.theguardian.com/world/2014/apr/24/vladimir-putin-...)

hackuser 11y ago

This part is troubling: In May 2014, Dave Willner and Dan Kelmenson, a software engineer at Facebook, patented a 3D-modeling technology[1] for content moderation ... First, the model identifies a set of malicious groups - say neo-Nazis, child pornographers, or rape promoters. The model then identifies users who associate with those groups through their online interactions. Next, the model searches for other groups associated with those users and analyzes those groups "based on occurrences of keywords associated with the type of malicious activity and manual verification by experts." This way, companies can identify additional or emerging malicious online activity It's not hard to imagine how this could be misused: Automated guilt-by-association, and within 3 degrees of separation, triggering investigation by authorities (corporate or otherwise). EDIT: And also this tool of automated censorship, though I doubt it will surprise many HN readers PhotoDNA works by processing an image every two milliseconds and is highly accurate. ... Then PhotoDNA extracts from each image a numeric signature that is unique to that image, "like your human DNA is to you." Whenever an image is uploaded, whether to Facebook or Tumblr or Twitter, and so on, he says, "its photoDNA is extracted and compared to our known ... images. Matches are automatically detected by a computer and reported ... for a follow-up investigation" Currently used for child exploitation images, but for what else too and by whom? They add that it's being applied to something more political: Farid is now working with tech companies and nonprofit groups to develop similar technology that will identify extremism and terrorist threats online - whether expressed in speech, image, video, or audio. [1] http://www.google.com.gh/patents/US20080256602 http://www.google.com.gh/patents/US20080256602

CM30 11y ago
This is a really good point, and this sort of thing could well be happening already. There has already been controversy over negative posts related to immigration being removed from Facebook due to a deal with the government. And there's already been talk of people inside Facebook asking whether they're 'doing enough' to stop a Donald Trump presidency: http://gizmodo.com/facebook-employees-asked-mark-zuckerberg-if-they-should-1771012990 http://gizmodo.com/facebook-employees-asked-mark-zuckerberg-... As well as cases where people were visited by police after making right wing comments online: http://www.breitbart.com/london/2016/04/07/police-raid-social-media-posts/ http://www.breitbart.com/london/2016/04/07/police-raid-socia... (Yes, I know it's from Breitbart, but the point still stands) And there's a lot of talk about Twitter censorship, blockbots blocking people based on what accounts they follow rather than any actions and all that stuff. Similar systems are already being abused by social networks, and this type of technology will only make cases like this more common.

deleted 11y ago

[deleted]

cb18 11y ago

Many large(and small) subreddits are highly censored. r/worldnews r/news r/europe to name a few. Here's a recent example, not even the most egregious, just a recent one. https://www.reddit.com/r/worldnews/comments/4ew68z/half_of_all_jobless_in_sweden_are_foreigners/ https://www.reddit.com/r/worldnews/comments/4ew68z/half_of_a... Noticed how many xx deep comment threads voted high up in the comment stack have been excised from the discussion. It's disgusting. It's very often the case that the top voted contributions have been 'disappeared.' Again, just disgusting. What exactly is the point of a 'democratic' news site, if there is this constant intervention from unaccountable authorities, constantly policing what information and opinions are allowed to be discussed. Any information outside the scope of a narrow ideological agenda is summarily terminated. Is this the public square of the future that we want? kn0thing, spez, What are your thoughts?

DanBC 11y ago
...and yet when anyone talks about a good subreddit it's almost always a sub that has strict rules and vigorous moderation, or a tiny sub that has a tiny number of users.
- cb18 11y ago
  Well, I'm saying those are bad subs, because they have 'vigorous' moderation. So subs with 'strict, vigorous moderation' is not a feature that automatically connotes, good sub.
  - azernik 11y ago
    Well, they're the subreddits that get cited whenever anyone talks about the glorious goodness of reddit: /r/ama, /r/iama, /r/askscience, /r/syriancivilwar (cited by news organizations as a source and informing their methodology)... Moderation doesn't automatically connote good subs, but the correlation is very strong.
braythwayt 11y ago
And yet, you can start your own damn subreddit. You can talk about anything you want, even violent, openly racist stuff. You can decide not to moderate it at all. So clearly, you can talk about nearly anything you want on reddit. Your problem is that you want to be able to talk about anything you want in somebody else’s subreddit. It’s as if you move onto my street, live in a perfectly good house, but complain about the fact that you can’t do whatever you like in my house. Freedom to say whatever you want is not the same thing as forcing other people to pay attention to it. You need to either find like-minded people moderating a like-minded subreddit, or find a way to get other people interested in your own subreddit. This business of demanding that everybody else give you a platform because “freedom of speech” is flat-out wrong.

rustynails 11y ago

I find The Verge extremely offensive. To pick a few bytes from This article, "Abusive men threaten spouses" (aka domestic violence is a man thing), "social media and mobile tech cause real harm to women". At least The verge are consistent. They have been sexist for as long as I have known them and their campaigns directed squarely at males (regardless of age). When The Verge viciously attacked Matt Taylor, I felt ill for days. I could not believe that a media outlet was so callous in its aim to destroy a person and their career ... . Then, they posted an article about rape on campus. Rolling Stone pulled the referred article (the woman who wrote the Rolling Stone article had a history of lying and she wasn't a student where she said she was, the university acknowledged incidents but denied ever speaking to this woman, etc). The verge was infuriated as the media's suppression (regardless of the truth) because it matched a picture they were trying to paint. This now explains why the media are so sexist - it's no accident that domestic violence is supposedly "men only". It's no surprise that boys are actively discriminated against in schools with STEM. It's linked to these moderators described in the article. It's not the moderation itself but the personal prejudices of the leaders of the moderators. The sexism and manipulation I see today on the internet is unprecedented in my lifetime. In the 60s, I never saw a teacher ignore students based on gender - but today I hear about boys frequently being ignored "to level the playing field" because the media tell them it's ok and worse, necessary. I spoke with one teacher recently because he was directed to discriminate against boys and he refused to discriminate and resigned in protest. I've had other teachers (male and female) strongly hint this happens frequently. This ruins children's lives! I've essen these policies in action on my own children too. The question I constantly asked myself was why. This article on moderation was my theory and now it confirms what I thought. The question is how do you challenge these manipulative propagandists bent on gender wars who selectively filter the content that gets shown? Decent people need to take an active stand. I'm glad to see there is some pushback against the feminist propaganda in the last few weeks. People are staring to question "supposed facts" and selective language such as "domestic violence" is a great example. This Verge article uses the same manipulative formulas that we continue to see (men:bad). I implore everyone to raise the awareness of the extreme prejudice of feminism and to oppose it assertively without threats. People need to denounce prejudice by anyone (despite what political correctness says). Feminism doesn't deserve protection in the way that the KKK don't deserve it. You aren't born a feminist or KKK, but you sure can choose to propagate the intolerance. I took my stance when I saw that young boys are impacted by a gender war that never should have started.

vonnik 11y ago

This is one example of a job where neural networks/AI can, and should, replace humans. Video and image classification is basically a solved problem. Numerous companies have already trained deep convolutional networks to recognize pornography and other forms of content unwanted on their platforms.

CM30 11y ago
Maybe I'm missing something here, but I suspect this would be a technical and political minefield. Sure, you can recognise certain types of abuse images, but then what about content like that in fiction? It's very logical that something seems as problematic in real life would be fine in say, a Call of Duty game or a blockbuster movie or some other form of fiction. Heck, there have been cases where scenes in movies and games have been 'reappropriated' as real life military or terrorist events by clueless nations and groups. For example: http://www.shacknews.com/article/93359/footage-of-six-isis-soldiers-killed-by-sniper-is-really-from-medal-of-honor http://www.shacknews.com/article/93359/footage-of-six-isis-s... So your system would have to figure out not just whether something is seen as 'offensive' or 'against the rules', but whether its from a fictional work that might be allowed on the site.

schoen 11y ago

This article made me wonder about the range of things that these companies are dealing with under the rubric of "moderation" or "trust and safety", particularly the way that companies can have so many different kinds of incentives to remove content from a platform. (Three that come to mind are "this may create legal liability", "this may damage our brand", and "this isn't what many of our users would like to see".) If the Internet or the way that people commonly use it is going to continue to get more centralized, I hope platforms improve their ability to distinguish between the problem of "things people don't want to see" and "things people don't want other people to see", or, to put it another way, between "please don't show me things like this" and "please don't allow people to publish things like this". While users themselves might not consistently draw the distinction between the two, platforms, in principle, could. The article touched in this issue in its discussion of the extreme variation in cultural norms, and the likelihood that one culture's vulgarity is another culture's lyr—er, that one culture's distaste for something will lead to considerable pressure for platforms to squelch it for everybody. A common idea is that platforms have a right and even a responsibility to define their own community standards and then people can choose platforms that they prefer or that best suit them, much as people can choose the newspaper whose editorial policy and biases they find most agreeable. I think this notion has a lot to recommend it but it seems less comfortable as the patterns of people's usage of the Internet becomes ever more centralized; it also raises the question of what things can be considered infrastructural enough that people can reasonably expect (or at least accept) complete content-neutrality from them. The article also provided an interesting reminder that most people are unlikely to be comfortable using communications systems that have no moderation or filtering at all. Among other things, that's an interesting challenge for decentralized and censorship-resistant systems; to become more popular and practical, they'll need to be paired with some ways that people can avoid unwanted communication, beyond spam.

scotty79 11y ago

I don't think perfect moderation to some global average morality is the future of humanity. It's TV mentality. I think the future is keeping vulnerable groups apart from offensive groups. Hellbaning and honeypot bots are the future. White supremacist can talk to other like him all day long and there's no harm done. Ones that would like to talk to black people or just people who get offended by racism should get algorithmically administered silence or honeypot bots that will react in best way possible to defuse and calm down without engaging human in distasteful content. Plus side of this solutio is that you can keep tabs on potentially dangerous people and react if they escalate or brag about physical harm they do.

rodgerd 11y ago
> White supremacist can talk to other like him all day long and there's no harm done. Right up until one of them decides it's time to leave his little echo chamber and murder a bunch of kids on an island because he and his Internet buddies have convinced themselves that's the right thing to do.
- michaelmrose 11y ago
  Between 1984 and bad people doing bad things I choose the later with no regrets.
  - scotty79 11y ago
    Worst thing in 1984 was not surveilance but rewriting the truth. You can't have progress if you are not open and honest. Pretending that there's just one morality that everyone normal must adhere to is more 1984 than any of my ideas.
- scotty79 11y ago
  You'd rather prefer they'd coordinate their attack on a channel you don't monitor? Do you think you can stop stupid people from doing harmful things by banning them from expressing their stupid opinions online? Brevik is exception not the rule and it could be all avoided if he was allowed to speak freely and discuss his demented ideas with people that don't mind so he can be monitored.

microcolonel 11y ago

>Their stories reveal how the boundaries of free speech were drawn during a period of explosive growth for a high-stakes public domain, one that did not exist for most of human history. I feel like this is a pretty big jump to make. There's a difference between deciding not to publish and amplify child sexual assault, and the "boundaries" of free speech. Whether their speech is free or not is irrelevant; YouTube has no obligation to publish it.

michaelmrose 11y ago

Whomever is responsible for the boxes that follow your mouse ought to have their commit privileges revoked.

takno 11y ago
Oh, is that what was meant to be happening? I was reading on mobile and the page just kept randomly scrolling, and when I came back to it after 8 hours a bunch of boxes had spilled semi transparent shadows over the text. It's funny how 20 years of improving tech have led us to a situation where we now just have to ignore huge layout issues and people shouting incredibly unpleasant things
- yomly 11y ago
  Glad someone else took issue with this. We're in a constant race to improve battery life and processing power so we can waste energy on these pointless "innovations". I'm all for improving UX, but I constantly wonder how much longer our batteries would last if they weren't being drained by gratuitous features like this one...
bobwaycott 11y ago
On mobile, it appears to be tied to the gyroscope or accelerometer. They only move as I tilt/move my phone. Keeping it fixed and upright removed the floating boxes. Very strange. It took me 2/3 of the article to even figure out it was my movement making them move.

firebones 11y ago

I wonder if there could be a protection ring [1] model of moderation and content classification--one that protected business interests yet classified objectionable material based on content while letting service providers retain common carrier status. In such a world, service providers would choose a level of filtering that fit their business needs, but would also let the free market decide what minimum standards were acceptable. YouTube's moderation expenses would simply focus on keeping them at level X (with a clear separation from some other site). Consumers, producers, and law enforcement would simply tune their dial to the level of content they found acceptable. It seems like it would promote competition while effectively pricing free speech. [1] https://en.wikipedia.org/wiki/Protection_ring https://en.wikipedia.org/wiki/Protection_ring

halugalugh 11y ago

This kind of censorship is the USAs version of Chinese censorship. Self imposed first amendment restriction. Very troubling.

Joeboy 11y ago

What is going on with Facebook's moderation of Kurdish / anti-Turkish content? Why is it that according to the leaked standards doc[1] (which as far as I can tell is genuine) Turkey gets to ban stuff it doesn't like on Facebook? Presumably it's OK to post a map showing Palestine, or a United Ireland, or other disputed regions on FB. Why can't you post a map of Kurdistan? Aside from Holocaust denial, no other political position seems to have been singled out in this way. [1] http://i.dailymail.co.uk/i/pix/2012/02/21/article-2104424-11D8ADDB000005DC-298_964x733.jpg http://i.dailymail.co.uk/i/pix/2012/02/21/article-2104424-11...

tim333 11y ago
I'm guessing here but Turkey/Erdogan are particularly aggressive in attacking stuff they don't like. See for example Germany being made to prosecute one of their comedians and "In 2012 the Committee to Protect Journalists (CPJ) ranked Turkey as the worst journalist jailer in the world (ahead of Iran and China), with 49 journalists sitting in jail." Also more ammusingly the doctor facing 5 years for posting pictures of Erdogan next to pictures of Gollum http://www.theguardian.com/world/2015/dec/03/lord-of-rings-director-insult-to-erdogan-mistaken-as-gollum-as-charcter-is http://www.theguardian.com/world/2015/dec/03/lord-of-rings-d... They are probably threatening to arrest people / ban Facebook if there are any maps of Kurdistan posted.

arca_vorago 11y ago

I've posted before about my theories regarding optimum commenting and moderation systems, but one thing I've recently been thinking about is some sort of logic/rationality analysis engine (perhaps through nlp/ml?). What I really want is to wade through the extra verbal fluff of a comment and get to its real points, so that I can determine if they are rational and logical. My primary method is by looking for fallacies which are tell-tell signs of a bad argument. That still doesnt "fix" moderation, but I think such a system could be used to get rid of consistently illogical trolls to reduce the moderator workload at least. Ideally, I imagine exclusivity of posting to be barrier 1, and moderation to be barrier 2. Some strange combinantion of hn Rules and /. Randomized moderation along with /. Tagging styles, would probably be the best. Another factor to consider is that sockpuppetry has thrown the democratic balance of such systems off, and Im not yet sure how to deal with that.

intended 11y ago
The assumption being that forums and trolls are static. Trolls adapt and with enough innuendo, the innocuous will be loaded with dark meaning. Plus someone has to wade through the sea of false positives.

rurban 11y ago

Oh, that Neda video was uploaded by me to Youtube, because the Iranians who brought it out of the country couldn't. They never talked to me about it.

mbrutsch 11y ago

Little different from any historical thought police - we know what's best for you, citizen, move along, nothing to see here.

bobwaycott 11y ago
This is vastly different from historical/fictional thought-policing. Did you read the article and really wind up with this takeaway?

bobwaycott 11y ago

Wow. There is just so much in this article to wrap one's head around. I feel somewhat ashamed that I've never really thought about moderators at all, much less of the psychological and emotional effects they must endure as a result of the content they must see. I know it's work I wouldn't want to perform. I'm grateful for this investigation and report. There's a ton to unpack and think through.

pessimizer 11y ago

I didn't even realize that all of these sites were deleting racist and "hate" speech. That's totally reprehensible. Any site that would do that would also delete blasphemy and violations of lese majeste. The worst part is that they justify it with philosophical jumbles like "open policies [are] stifling free expression," as if one person speaking prevents another from also speaking. edit: also, the "trillion or so dollars of value" that an "expert" ascribed to Section 230 is simply a euphemism for ads. Nothing else. edit2: Wow. I just realized that the reason I've stopped getting racist hits on search engines in recent years is because they've been removed. Part of the range of discourse disappeared and I didn't even realize it. That's also got to be the reason that comment sections in papers like the Boston Globe, Chicago Tribune, and Washington Post have become racist cesspools. Forums where they could have their mutual appreciation societies away from normal people have been completely hidden from view. Am I the only person who didn't think the internet was broken before all of this arbitration?

brashrat 11y ago

I thought this headline was referring to what I think is a more insidious form of moderation, the type that's practiced here on ycombinator with regard to style, for example, negativity is punished broadly here. The reason I find it more insidious is that it affects the speech of and acts to silence even moderate people. Extreme speech is easy to spot; but when moderate speech encounters headwinds that divert its path, that's creepy. I understand the goals here, don't need them explained, just pointing out that slashdot in its heyday was a terrific and informative resource and it did not resort to stifling.