19 ms·

Censorship 2.0: Shadowy forces controlling online conversations

255 points by r721 12y ago

r721 12y ago
Presentation slides (pdf, 55mb): http://conference.hitb.org/hitbsecconf2014kul/materials/D2T1%20-%20Haroon%20Meer%20Azhar%20Desai%20and%20Marco%20Slaviero%20-%20Weapons%20of%20Mass%20Distraction.pdf http://conference.hitb.org/hitbsecconf2014kul/materials/D2T1... PDFy mirror: https://pdf.yt/d/Rw7qpWWchX0cd05f https://pdf.yt/d/Rw7qpWWchX0cd05f
snarfy 12y ago
Reminds me of megaphone: http://en.wikipedia.org/wiki/Megaphone_desktop_tool http://en.wikipedia.org/wiki/Megaphone_desktop_tool
legutierr 12y ago
Are the sock puppets that frequent HN simply much harder to detect? It often surprises me that HN is not a target for these kinds of people; I like to think that I have a good sense of when this kind of thing is going on, but probably I'm less sophisticated than I think I am.
- gburt 12y ago
  It is widely claimed (in comment threads) that many tech. companies directly or indirectly operate this sort of targeting here. I'm not certain there's any evidence of this though.
  - jacquesm 12y ago
    There is plenty of evidence, including some very high level interventions to put a stop to some of the worst of it. No need to name names but it happened in the past it is happening in the present and it will most likely continue as long as HN has any sizeable (or perceived as influential) audience.
- penprog 12y ago
  hn is without a doubt a target for these kinds of people. I'm honestly surprised that this thread hasn't been flag killed considering how it shows how blatantly idf are gaming this system. edit: nothing against Israel and Jews in general, just that there was some pretty clear evidence specific to the IDF in the article
- jacquesm 12y ago
  HN is definitely a target for these kind of people. It has gone so far as high level contacts between the bosses of these kind of people and the stewards to sort things out. That doesn't mean that it has stopped nor does it mean there won't be repetitions. Astroturfing, burying, and derailing all happen. The scariest aspect to me is not those that are paid to do this sort of thing, because I can see their motivation but their little helpers in the form of the gullible public. From the above you could probably conclude I'm a cynical old bastard these days and you wouldn't be to far off. I suspect that any forum with more than a few thousand participants has elements that are trying to actively shape the conversation to serve their goals on it.
  - legutierr 12y ago
    > The scariest aspect to me...are their little helpers in the form of the gullible public Well, given my rather naive comment above, I have to assume that I fall into this category to some extent. What would you suggest people do differently?
    - jacquesm 12y ago
      Be skeptical about everything you read online and offline (but moreso online), check up on sources rather than to use citations helpfully provided to underpin some bit of information and in general read until the cows come home to make sure you have a solid foundation from which to make your own judgment, especially when it comes to history and any attempts to re-write it.
  - deleted 12y ago
    [deleted]
higherpurpose 12y ago
That's a pretty scary concept when we have the rise of the unstoppable, unaccountable and highly secretive spy agencies. Things will probably get even worse when Strong AI exists.
- jacquesm 12y ago
  When -> if. And if strong AI comes in to play the world will change in ways that any kind of speculation as to whether 'things will get worse or better' is totally moot.
woah 12y ago
Seems like it would be an important public service to game as much online discussion as possible, persistently. Right now, the vulnerability of these discussion media is not detrimental to those running them. Make gameability of discussion media something that is actually visible, and the vulnerabilities will be patched.
- jevgeni 12y ago
  Kind of hard to follow, what you are trying to say. Synergy.
  - woah 12y ago
    Persistent, ridiculous gaming of discussion media would be good for society, as it would force it to become gaming resistant.
    - lol123456 12y ago
      If you take away sock puppets you take away anonymity as well. Even if you make everyone use their name and face and display pic and ip address and location, there is no guarantee that a group of people will stop censoring another group with or without sock puppets. If you care to count there are at-least 5 controls on HN or any forum since 1990. 1. What gets on homepage ( Stickies ) 2. Admin control 3. Firewall 4. Voting 5. Moderators If you take social groups into consideration then on HN you have at least 1. Programmers 2. Startup People 3. Technical People 4. Geeks 5. Hackers 6. Security Hackers 7. College Students 8. Academic People 9. Trend followers 10. Trolls ( my people ) These 10 groups fight amongst each other with the help of 5 controls above. Every forum is always like this, since its inception-middle-death. No one really wins but the constant bickering about forces each group to outdo each other. Whether it is progress or going about in circles only time will tell. Just because you can patch the fucking software doesn't mean people will change. You will probably add one more control level to make into a total of 6 and then nothing. Some other egghead will write about censorship 3.0 It is best to accept internet as it is.
fidotron 12y ago
Of course, this place isn't immune to it. That includes all the usual suspect discussions (NSA, Palestine, China, Russia etc.) but more visibly with the employees of certain companies. One of the more curious things you can do on HN to distract from a story is to start a tangential flamewar in the comments on it, but my single favourite way I see to bury things around here is to ask for detailed citations to data for something which really doesn't need it, as that results either in a flamewar or if left the post loses any momentum completely.
- waterlesscloud 12y ago
  There should be a word for requests for data citations that are really just disguised admissions of having no meaningful response to a point.
  - ddingus 12y ago
    I use the word "dodge" and that is one of a number of things people do to avoid having to deal with information they find difficult to process. A while back, I decided to compile a list of these behaviors seen on a smaller, guilty pleasure type of site I enjoy. Here are the major ones seen very frequently: 1. Subject change 2. False comparison 3. Claim of too many opponents 4. Silence 5. Claim to either be a member of, or opponents of discussion being members of club, with said membership somehow required to complete what is otherwise a non-rational conclusion. 6. Claim of overall subject and or implications of subject complexity being greater than scope of discussion; therefore, it’s all a waste of time. (You would not understand) 7. Transformation of rational point at hand to emotional one. 8. The bible. 9. Redefinition of common words. 10. Personal attack. 11. Obsfucation. (however you spell it) 12. Excess verbosity. What I did was post this, put names on it, including my own, and then start citing it when the behavior is seen. I expected people to moderate the behaviors and perhaps improve dialog. What actually happened was the behaviors intensified! They created new ones, and started using them in combination: Subject change, excess verbosity, silence was very common. Blow up the dialog to prevent realizations or consensus, then go silent to let it all fade away. Then do it again, and again... Now, something good did happen. Participants who get it, tended to add their own observations to the list, and a bond formed between them. This helped differentiate the dialog some, and high clarity material, rational material was easier to identify and that dialog did improve. But the noise makers also improved, making better noise, more frequently, division went up, and the overall tone of the place was harder to enjoy. This led me to another realization: We very often want to frame things as debate when the truth is we are often doing advocacy. Advocacy has three components: Reason, character, emotion. Advocacy is a super set of debate. Effective advocates employ all three and here is the kicker: They will trump an effective debater in a dialog. Interesting isn't it?
penprog 12y ago
The first thing this article reminded me of was ghost in the shell season 2. In the show the Intelligence Agency of the government used this sort of info manipulation to increase animosity towards refugees to the point where the public was in support of basically slaughtering them. In the show this agency also analyzed public opinion using info obtained from discussion forums to gauge the public's acceptance of government policy and they would use the information manipulation with sock puppets to change public opinion in their favor.
- mrottenkolber 12y ago
  One of the best shows ever. The whole Stand Alone Complex concept is kind of unique to GitS, is it? Edit: Oh and btw, ghost in the lisp: http://medias.ircam.fr/x03b42f http://medias.ircam.fr/x03b42f RPG talking about his work on the mass media manipulation machine of DARPA.
  - cjslep 12y ago
    Yes and no[0]. For reference, the definition of a Stand Alone Complex from wikipedia [1]: Stand Alone Complex (スタンド・アローン・コンプレックス Sutando Arōn Konpurekkusu) eventually came to represent a phenomenon where unrelated, yet very similar actions of individuals create a seemingly concerted effort. The ideology of radical Islam from the viewpoint of neoconservatives in the USA in the 00 decade was of "shadowy underground terrorist cells all linked together and we should all be afraid of it", when in reality it was much more akin to the Stand Alone Complex: disparate radical groups all over the world used a common idea to act on their own but without communicating to one another. When viewed by the intelligence community, of course they weren't independent isolated events influenced by the same root idea; they were a centralized enemy. They projected thwir own centralized nature to the fragmented radical Islamic movement of the early 00's. Eventually, that radical Islamic movement grew into the identity provided them by the intelligence community as a self-fulfilling prophecy of sorts. [0]So yes in name, no in practice if you buy into the viewpoint above. [1] http://en.wikipedia.org/wiki/Philosophy_of_Ghost_in_the_Shell#Stand_Alone_Complex http://en.wikipedia.org/wiki/Philosophy_of_Ghost_in_the_Shel...
lizzard 12y ago
I've been talking about this for years; we have to at the least be aware that there is wide scale sockpuppetry and astroturfing for political as well as corporate ends. Better tools for disinformation detection may help. One person running many personas is much easier to detect than the more sophisticated model we have now.
raggaeturf 12y ago
If you doubt this, then just criticize Google on HN or Reddit... Google has thousands of employees and sock puppets on all social media discussions. It's especially blatant on Reddit. Google is a charity and everything they do is perfect and for the good of humanity. They've never been involved in massive spying and with monopolistic practices concerning their spammy ads and search engine preference for ebay, amazon, trip advisor and others.
- pekk 12y ago
  Big companies cultivate the mindshare of product fans and employees who then feel naturally inclined to argue for their favorite companies, that's not always meaningfully different from astroturfing, but this doesn't necessarily mean that what you are reading is coming from some Conversation Control Department. Sometimes lots of people on the Internet just disagree with your opinion.
- lingben 12y ago
  same thing for facebook, criticise the effectiveness of their ads and watch what happens
- sologoub 12y ago
  Negative comment, 2 karma and this is an only comment you made... You look like a sock puppet to me :)
  - penprog 12y ago
    maybe you are the sockpuppet?
    - jerf 12y ago
      Behold the futility of the sock puppet accusation in practice. (Preempting "jerf is a sock puppet". Let's skip that.)
      - deleted 12y ago
        [deleted]
pekk 12y ago
Whenever someone says something you disagree with, just accuse them of being a sockpuppet of a shadowy conspiracy. Then you can censor their view very effectively.
arca_vorago 12y ago
This is more insidious than I think even most HN readers realize. In the intelligence community, the big black sheep people don't really talk about is psychological operations, or psyops, units. They are the same breed that used research from MK ULTRA style programs to learn how to deprogram humans (some claims on how to reprogram them), and part of that is a scientific analysis of mass social community discourse. Previously, domestic propaganda was illegal, though it was often engaged in anyway now that the law against domestic propaganda has been rescinded that means the remnants of older programs like Operation Mockingbird, where the CIA inserted people into news outlets across the board for intel gathering and disinfo pushing, have now evolved to a level where they are actively manipulating public discourse on all kinds of subjects. A very good example of this is the post JFK assassination issue. It's pretty clear to any intellectually honest person that there are some very large issues that were never addressed regarding the assassination, because the CIA purposeful put into action an operation to discredit any alternative theory's as "conspiracy theory" and anyone who talked in that manner was labeled as a crackpot "conspiracy theorist" and then a large campaign to undermine any logical discussion of "conspiracy theory", essentially by putting it, as Chomsky would say, "outside the accepted range of debate". Wonder why the fourth estate, aka journalism is in such a sad state of affairs? It's because the state has reached (quietly) so far into it that even the handful of real journalist are having a hard time. They have keep up technologically to keep sources safe, which proves almost impossible these days. They have to deal with pushback from the editors bosses of a glarily political nature, they have to deal with being cut off from sources if they happen to piss off the wrong people with a too-strong story. This is another reason the surveillance society is so insidious. I have said it before and I will say it again, surveillance, censorship, subtle covert influence of public discourse, are not about security. It's about control. A lot of people argue about 1984 vs Brave New World. What I say is that it's a Brave New World, unless you attempt to resist, then it's 1984. This is the state we live in, the real question is "Are the people going to let this happen?" I'm no longer so reliant on the "long arc of justice", and am getting pretty pessimistic.
- maxerickson 12y ago
  The difficult thing is figuring out how much of it is insidious and dangerous and how much of it is The Men Who Stare at Goats.
lol123456 12y ago
A social group implies at least 2 persons. Even if two people are in a room, and they start having a conversation what you find is that they will try to * Censor / Re-Imagine themselves to present their POV in a better way * Censor / Manipulate the other person to force their POV unto them. The online is a complete extension of the offline, so its perfectly normal for all this to happen, even all the time !
intended 12y ago
All the discussion here pointing out possible shill armies aside - is there some way to actually stop it. How do we move conversations - Forward.
- sitkack 12y ago
  We voice our thoughts and opinions verbally, with our neighbors, coworkers and friends. Convincing people online in discussion forums is not the best use of our rhetorical time, I would argue that most discussion forums are actually honey pots, whether intended or not.
notacoward 12y ago
This is why I'm not particularly thrilled about programs passing the Turing test. As soon as we have programs that can write credibly human-sounding comments on sites like this, many conversations will be controlled by whoever can afford the largest sockpuppet farm. Also, for every post on the actual threads of interest, there will be dozens of comments on every thread designed to increase karma. Automated karma whoring might even be more injurious than the direct intervention, because it will increase the groupthink tendencies already present. Some might say we're already half way there. I disagree. Yes, Google or Apple employees might swamp an occasional thread here and there, either openly or covertly, but what I'm talking about would be a whole order of magnitude worse and constant.
- r0s 12y ago
  It seems like any kind of mass-comment system would be easy to detect. Simply limiting the post rate and comparing to other posts would work just like current anti-span filters now.
  - Oblouk 12y ago
    This is if all the comments came from the same IP and if the puppeteer was stupid enough to post all the comments at once. Additionally if the AI can beat the turing test then comparing the posts would not be practical. Passing the turing test signifies that it is not possible to distinguish the comments from human comments by looking at the comments alone.
    - r0s 12y ago
      You don't need to detect a machine, astroturfing now is mostly a manual process. Recognizing the same topics repeated is enough, even if it happens over time. The longer the time between posts, the less effective the campaign. The less cohesive the topic, the less effective the campaign. This already exists somewhat, machine or human is almost irrelevant.
      - lukifer 12y ago
InfiniteRand 12y ago
I'm not sure how alarmed I am by this. I certainly have seen this in action on Wikipedia at least, if not on the scale described, but I have little doubt it occurs and I can think of a couple events whose online conversations show evidence of this sort of manipulation. To some degree I admit, when I see the overwhelming online conversation swaying in one direction, it does influence. On the other hand, partially because I know online conversation manipulation is out there, I become less prone to read comments. I used to always read comments to Economist articles for example, now I rarely do. Ultimately, if there is online conversation manipulation but people barely pay attention to it, it will be overwhelmed by all the other random sources of influence in our lives. Whether, traffic, etc. all play some influence on our opinions, if the impact of the online conversation manipulation is less than that, I am not alarmed. There are two other aspects of this worth thinking about: 1. What is the problem we have with online conversation manipulation? Is it the sock-puppets alone? If you replace sock-puppets with volunteers, is that different enough? If you take volunteers who otherwise wouldn't participate in that online conversation, is that different enough? What are the aspects of this that are different from a legitimate campaign to change the opinions of others about a position you think is profoundly wrong. 2. One sad aspect of this is that it dilutes the impact of anonymous or semi-anonymous voices on the web. This was a short-cut to influence for ordinary web users. On the other hand, this is in a way natural, whenever the impact of a medium becomes popularly recognized, more people will try to utilize that medium, and its impact will be diluted. In the same way, just having a website used to be a short-cut to influence, but now has little effect.
- stemc43 12y ago
  US gov. actually admitted that it has a trolling network - it used to be called "viral peace". They claim it operates on "terrorist websites". And of course they give no definition of "terrorist websites" - But I think it's safe to assume reddit, twitter, facebook are terrorist websites.
  - schoen 12y ago
    They also have some openly governmental accounts called "Think Again Turn Away", like https://twitter.com/ThinkAgain_DOS https://twitter.com/ThinkAgain_DOS https://www.facebook.com/ThinkAgainTurnAway https://www.facebook.com/ThinkAgainTurnAway They used to directly engage in debate with people more, while now they seem to have switched to a more posting-only style.
wyck 12y ago
This has been going on in one form of another throughout history. But if you want to sway opinion on the internet, it's not that easy. There are a lot of voices and noise in the mix, compared to how it was a mere 20 years ago. This is certainly important, but in the context of history, how people consume information is on a much better track, in my opinion things are actually getting better if you look at the big picture.
fiatmoney 12y ago
2013's "Most Reddit Addicted City" was Eglin Air Force Base. http://www.redditblog.com/2013/05/get-ready-for-global-reddit-meetup-day.html http://www.redditblog.com/2013/05/get-ready-for-global-reddi...
- superuser2 12y ago
  Air Force Bases are also full of technologically inclined 18 to 25 year old men, i.e the Reddit demographic.
Animats 12y ago
I've made this point before about search engine ranking. Most of the popularity signals used by search engines can be, and are, faked. I have two papers on this: "Social is bad for search, and search is bad for social" (2012), or how Google's use of social signals backfired, badly. http://www.sitetruth.com/doc/socialisbadforsearch09.pdf http://www.sitetruth.com/doc/socialisbadforsearch09.pdf How Google's use of social signals backfired, badly. "'Places' spam - the new front in the spam wars", or how Google's use of "local" information backfired. "http://www.sitetruth.com/doc/placesspam10.pdf" http://www.sitetruth.com/doc/placesspam10.pdf" (2010) Google's merge of data from Google Maps into the main search engine results created a whole new branch of local SEO spam. The fundamental problem is that the creation of fake online identities is cheap and easy. This can be partially fixed by taking a tough line on identity, but that's hard for services which are either big or have only a casual connection with their users. Facebook ran into the gay agenda enforcing a real names policy. The mobile guys can at least make people buy a phone to fake an identity. (A phone number is not enough; you can rent fake phone numbers. See "http://www.attlines.com http://www.attlines.com) An app that phones home with too much user information, though, is hard to fake cheaply. Yelp can tell if your phone has been to the place you're rating.
- jordigh 12y ago
  > the gay agenda I'm not sure if you know about this, but this is typically a term of derision and a strawman by anti-gay and homophobic groups. http://en.wikipedia.org/wiki/Homosexual_agenda http://en.wikipedia.org/wiki/Homosexual_agenda
  - seanp2k2 12y ago
    +1; there are also lots of other marginalized groups of people who stand to lose something if aliases cannot be used. Edit: for example: http://www.theguardian.com/commentisfree/2014/sep/29/facebooks-real-names-policy-is-legal-but-its-also-problematic-for-free-speech http://www.theguardian.com/commentisfree/2014/sep/29/faceboo...
    - teddyh 12y ago
patcon 12y ago
Call me crazy, but it /has/ crossed my mind that #gamergate might have been a large-scale test of the ability to create widespread conflict and some degree of community crisis from thin air through sock-puppetry... EDIT: I honestly don't totally understand what gamergate is, but am in awe of the resurging energy of the whole thing, and hence the observation
- deleted 12y ago
  [deleted]
Udik 12y ago
Hmmm. The only verified case of sophisticated, organized sockpuppetry in the wild mentioned by the article seem to be related to the Israeli- Palestinian conflict and anti-semitism, and in every case they seem to be orchestrated to favour Israel. But all the comments here seem concentrated on the NSA, AIs, and even abortion. Now downvote me :)
- krapp 12y ago
  Then clearly either you, or the article, or both are just another reeking layer in the American government's propaganda onion. Edward Snowden clearly proved, scientifically, that the NSA/CIA/Military Industrial Complex controls everything and everyone I don't personally agree with, especially on the internet. Now go tell your controllers your mission has failed, Udik. If that even is your real name.

255 points by r721 12y ago

r721 12y ago

Presentation slides (pdf, 55mb): http://conference.hitb.org/hitbsecconf2014kul/materials/D2T1%20-%20Haroon%20Meer%20Azhar%20Desai%20and%20Marco%20Slaviero%20-%20Weapons%20of%20Mass%20Distraction.pdf http://conference.hitb.org/hitbsecconf2014kul/materials/D2T1... PDFy mirror: https://pdf.yt/d/Rw7qpWWchX0cd05f https://pdf.yt/d/Rw7qpWWchX0cd05f

snarfy 12y ago

Reminds me of megaphone: http://en.wikipedia.org/wiki/Megaphone_desktop_tool http://en.wikipedia.org/wiki/Megaphone_desktop_tool

legutierr 12y ago

Are the sock puppets that frequent HN simply much harder to detect? It often surprises me that HN is not a target for these kinds of people; I like to think that I have a good sense of when this kind of thing is going on, but probably I'm less sophisticated than I think I am.

gburt 12y ago
It is widely claimed (in comment threads) that many tech. companies directly or indirectly operate this sort of targeting here. I'm not certain there's any evidence of this though.
- jacquesm 12y ago
  There is plenty of evidence, including some very high level interventions to put a stop to some of the worst of it. No need to name names but it happened in the past it is happening in the present and it will most likely continue as long as HN has any sizeable (or perceived as influential) audience.
penprog 12y ago
hn is without a doubt a target for these kinds of people. I'm honestly surprised that this thread hasn't been flag killed considering how it shows how blatantly idf are gaming this system. edit: nothing against Israel and Jews in general, just that there was some pretty clear evidence specific to the IDF in the article
jacquesm 12y ago
HN is definitely a target for these kind of people. It has gone so far as high level contacts between the bosses of these kind of people and the stewards to sort things out. That doesn't mean that it has stopped nor does it mean there won't be repetitions. Astroturfing, burying, and derailing all happen. The scariest aspect to me is not those that are paid to do this sort of thing, because I can see their motivation but their little helpers in the form of the gullible public. From the above you could probably conclude I'm a cynical old bastard these days and you wouldn't be to far off. I suspect that any forum with more than a few thousand participants has elements that are trying to actively shape the conversation to serve their goals on it.
- legutierr 12y ago
  > The scariest aspect to me...are their little helpers in the form of the gullible public Well, given my rather naive comment above, I have to assume that I fall into this category to some extent. What would you suggest people do differently?
  - jacquesm 12y ago
    Be skeptical about everything you read online and offline (but moreso online), check up on sources rather than to use citations helpfully provided to underpin some bit of information and in general read until the cows come home to make sure you have a solid foundation from which to make your own judgment, especially when it comes to history and any attempts to re-write it.
- deleted 12y ago
  [deleted]

higherpurpose 12y ago

That's a pretty scary concept when we have the rise of the unstoppable, unaccountable and highly secretive spy agencies. Things will probably get even worse when Strong AI exists.

jacquesm 12y ago
When -> if. And if strong AI comes in to play the world will change in ways that any kind of speculation as to whether 'things will get worse or better' is totally moot.

woah 12y ago

Seems like it would be an important public service to game as much online discussion as possible, persistently. Right now, the vulnerability of these discussion media is not detrimental to those running them. Make gameability of discussion media something that is actually visible, and the vulnerabilities will be patched.

jevgeni 12y ago
Kind of hard to follow, what you are trying to say. Synergy.
- woah 12y ago
  Persistent, ridiculous gaming of discussion media would be good for society, as it would force it to become gaming resistant.
  - lol123456 12y ago
    If you take away sock puppets you take away anonymity as well. Even if you make everyone use their name and face and display pic and ip address and location, there is no guarantee that a group of people will stop censoring another group with or without sock puppets. If you care to count there are at-least 5 controls on HN or any forum since 1990. 1. What gets on homepage ( Stickies ) 2. Admin control 3. Firewall 4. Voting 5. Moderators If you take social groups into consideration then on HN you have at least 1. Programmers 2. Startup People 3. Technical People 4. Geeks 5. Hackers 6. Security Hackers 7. College Students 8. Academic People 9. Trend followers 10. Trolls ( my people ) These 10 groups fight amongst each other with the help of 5 controls above. Every forum is always like this, since its inception-middle-death. No one really wins but the constant bickering about forces each group to outdo each other. Whether it is progress or going about in circles only time will tell. Just because you can patch the fucking software doesn't mean people will change. You will probably add one more control level to make into a total of 6 and then nothing. Some other egghead will write about censorship 3.0 It is best to accept internet as it is.

fidotron 12y ago

Of course, this place isn't immune to it. That includes all the usual suspect discussions (NSA, Palestine, China, Russia etc.) but more visibly with the employees of certain companies. One of the more curious things you can do on HN to distract from a story is to start a tangential flamewar in the comments on it, but my single favourite way I see to bury things around here is to ask for detailed citations to data for something which really doesn't need it, as that results either in a flamewar or if left the post loses any momentum completely.

waterlesscloud 12y ago
There should be a word for requests for data citations that are really just disguised admissions of having no meaningful response to a point.
- ddingus 12y ago
  I use the word "dodge" and that is one of a number of things people do to avoid having to deal with information they find difficult to process. A while back, I decided to compile a list of these behaviors seen on a smaller, guilty pleasure type of site I enjoy. Here are the major ones seen very frequently: 1. Subject change 2. False comparison 3. Claim of too many opponents 4. Silence 5. Claim to either be a member of, or opponents of discussion being members of club, with said membership somehow required to complete what is otherwise a non-rational conclusion. 6. Claim of overall subject and or implications of subject complexity being greater than scope of discussion; therefore, it’s all a waste of time. (You would not understand) 7. Transformation of rational point at hand to emotional one. 8. The bible. 9. Redefinition of common words. 10. Personal attack. 11. Obsfucation. (however you spell it) 12. Excess verbosity. What I did was post this, put names on it, including my own, and then start citing it when the behavior is seen. I expected people to moderate the behaviors and perhaps improve dialog. What actually happened was the behaviors intensified! They created new ones, and started using them in combination: Subject change, excess verbosity, silence was very common. Blow up the dialog to prevent realizations or consensus, then go silent to let it all fade away. Then do it again, and again... Now, something good did happen. Participants who get it, tended to add their own observations to the list, and a bond formed between them. This helped differentiate the dialog some, and high clarity material, rational material was easier to identify and that dialog did improve. But the noise makers also improved, making better noise, more frequently, division went up, and the overall tone of the place was harder to enjoy. This led me to another realization: We very often want to frame things as debate when the truth is we are often doing advocacy. Advocacy has three components: Reason, character, emotion. Advocacy is a super set of debate. Effective advocates employ all three and here is the kicker: They will trump an effective debater in a dialog. Interesting isn't it?

penprog 12y ago

The first thing this article reminded me of was ghost in the shell season 2. In the show the Intelligence Agency of the government used this sort of info manipulation to increase animosity towards refugees to the point where the public was in support of basically slaughtering them. In the show this agency also analyzed public opinion using info obtained from discussion forums to gauge the public's acceptance of government policy and they would use the information manipulation with sock puppets to change public opinion in their favor.

mrottenkolber 12y ago
One of the best shows ever. The whole Stand Alone Complex concept is kind of unique to GitS, is it? Edit: Oh and btw, ghost in the lisp: http://medias.ircam.fr/x03b42f http://medias.ircam.fr/x03b42f RPG talking about his work on the mass media manipulation machine of DARPA.
- cjslep 12y ago
  Yes and no[0]. For reference, the definition of a Stand Alone Complex from wikipedia [1]: Stand Alone Complex (スタンド・アローン・コンプレックス Sutando Arōn Konpurekkusu) eventually came to represent a phenomenon where unrelated, yet very similar actions of individuals create a seemingly concerted effort. The ideology of radical Islam from the viewpoint of neoconservatives in the USA in the 00 decade was of "shadowy underground terrorist cells all linked together and we should all be afraid of it", when in reality it was much more akin to the Stand Alone Complex: disparate radical groups all over the world used a common idea to act on their own but without communicating to one another. When viewed by the intelligence community, of course they weren't independent isolated events influenced by the same root idea; they were a centralized enemy. They projected thwir own centralized nature to the fragmented radical Islamic movement of the early 00's. Eventually, that radical Islamic movement grew into the identity provided them by the intelligence community as a self-fulfilling prophecy of sorts. [0]So yes in name, no in practice if you buy into the viewpoint above. [1] http://en.wikipedia.org/wiki/Philosophy_of_Ghost_in_the_Shell#Stand_Alone_Complex http://en.wikipedia.org/wiki/Philosophy_of_Ghost_in_the_Shel...

lizzard 12y ago

I've been talking about this for years; we have to at the least be aware that there is wide scale sockpuppetry and astroturfing for political as well as corporate ends. Better tools for disinformation detection may help. One person running many personas is much easier to detect than the more sophisticated model we have now.

raggaeturf 12y ago

If you doubt this, then just criticize Google on HN or Reddit... Google has thousands of employees and sock puppets on all social media discussions. It's especially blatant on Reddit. Google is a charity and everything they do is perfect and for the good of humanity. They've never been involved in massive spying and with monopolistic practices concerning their spammy ads and search engine preference for ebay, amazon, trip advisor and others.

pekk 12y ago
Big companies cultivate the mindshare of product fans and employees who then feel naturally inclined to argue for their favorite companies, that's not always meaningfully different from astroturfing, but this doesn't necessarily mean that what you are reading is coming from some Conversation Control Department. Sometimes lots of people on the Internet just disagree with your opinion.
lingben 12y ago
same thing for facebook, criticise the effectiveness of their ads and watch what happens
sologoub 12y ago
Negative comment, 2 karma and this is an only comment you made... You look like a sock puppet to me :)
- penprog 12y ago
  maybe you are the sockpuppet?
  - jerf 12y ago
    Behold the futility of the sock puppet accusation in practice. (Preempting "jerf is a sock puppet". Let's skip that.)
    - deleted 12y ago
      [deleted]

pekk 12y ago

Whenever someone says something you disagree with, just accuse them of being a sockpuppet of a shadowy conspiracy. Then you can censor their view very effectively.

arca_vorago 12y ago

This is more insidious than I think even most HN readers realize. In the intelligence community, the big black sheep people don't really talk about is psychological operations, or psyops, units. They are the same breed that used research from MK ULTRA style programs to learn how to deprogram humans (some claims on how to reprogram them), and part of that is a scientific analysis of mass social community discourse. Previously, domestic propaganda was illegal, though it was often engaged in anyway now that the law against domestic propaganda has been rescinded that means the remnants of older programs like Operation Mockingbird, where the CIA inserted people into news outlets across the board for intel gathering and disinfo pushing, have now evolved to a level where they are actively manipulating public discourse on all kinds of subjects. A very good example of this is the post JFK assassination issue. It's pretty clear to any intellectually honest person that there are some very large issues that were never addressed regarding the assassination, because the CIA purposeful put into action an operation to discredit any alternative theory's as "conspiracy theory" and anyone who talked in that manner was labeled as a crackpot "conspiracy theorist" and then a large campaign to undermine any logical discussion of "conspiracy theory", essentially by putting it, as Chomsky would say, "outside the accepted range of debate". Wonder why the fourth estate, aka journalism is in such a sad state of affairs? It's because the state has reached (quietly) so far into it that even the handful of real journalist are having a hard time. They have keep up technologically to keep sources safe, which proves almost impossible these days. They have to deal with pushback from the editors bosses of a glarily political nature, they have to deal with being cut off from sources if they happen to piss off the wrong people with a too-strong story. This is another reason the surveillance society is so insidious. I have said it before and I will say it again, surveillance, censorship, subtle covert influence of public discourse, are not about security. It's about control. A lot of people argue about 1984 vs Brave New World. What I say is that it's a Brave New World, unless you attempt to resist, then it's 1984. This is the state we live in, the real question is "Are the people going to let this happen?" I'm no longer so reliant on the "long arc of justice", and am getting pretty pessimistic.

maxerickson 12y ago
The difficult thing is figuring out how much of it is insidious and dangerous and how much of it is The Men Who Stare at Goats.

lol123456 12y ago

A social group implies at least 2 persons. Even if two people are in a room, and they start having a conversation what you find is that they will try to * Censor / Re-Imagine themselves to present their POV in a better way * Censor / Manipulate the other person to force their POV unto them. The online is a complete extension of the offline, so its perfectly normal for all this to happen, even all the time !

intended 12y ago

All the discussion here pointing out possible shill armies aside - is there some way to actually stop it. How do we move conversations - Forward.

sitkack 12y ago
We voice our thoughts and opinions verbally, with our neighbors, coworkers and friends. Convincing people online in discussion forums is not the best use of our rhetorical time, I would argue that most discussion forums are actually honey pots, whether intended or not.

notacoward 12y ago

This is why I'm not particularly thrilled about programs passing the Turing test. As soon as we have programs that can write credibly human-sounding comments on sites like this, many conversations will be controlled by whoever can afford the largest sockpuppet farm. Also, for every post on the actual threads of interest, there will be dozens of comments on every thread designed to increase karma. Automated karma whoring might even be more injurious than the direct intervention, because it will increase the groupthink tendencies already present. Some might say we're already half way there. I disagree. Yes, Google or Apple employees might swamp an occasional thread here and there, either openly or covertly, but what I'm talking about would be a whole order of magnitude worse and constant.

r0s 12y ago
It seems like any kind of mass-comment system would be easy to detect. Simply limiting the post rate and comparing to other posts would work just like current anti-span filters now.
- Oblouk 12y ago
  This is if all the comments came from the same IP and if the puppeteer was stupid enough to post all the comments at once. Additionally if the AI can beat the turing test then comparing the posts would not be practical. Passing the turing test signifies that it is not possible to distinguish the comments from human comments by looking at the comments alone.
  - r0s 12y ago
    You don't need to detect a machine, astroturfing now is mostly a manual process. Recognizing the same topics repeated is enough, even if it happens over time. The longer the time between posts, the less effective the campaign. The less cohesive the topic, the less effective the campaign. This already exists somewhat, machine or human is almost irrelevant.
    - lukifer 12y ago

InfiniteRand 12y ago

I'm not sure how alarmed I am by this. I certainly have seen this in action on Wikipedia at least, if not on the scale described, but I have little doubt it occurs and I can think of a couple events whose online conversations show evidence of this sort of manipulation. To some degree I admit, when I see the overwhelming online conversation swaying in one direction, it does influence. On the other hand, partially because I know online conversation manipulation is out there, I become less prone to read comments. I used to always read comments to Economist articles for example, now I rarely do. Ultimately, if there is online conversation manipulation but people barely pay attention to it, it will be overwhelmed by all the other random sources of influence in our lives. Whether, traffic, etc. all play some influence on our opinions, if the impact of the online conversation manipulation is less than that, I am not alarmed. There are two other aspects of this worth thinking about: 1. What is the problem we have with online conversation manipulation? Is it the sock-puppets alone? If you replace sock-puppets with volunteers, is that different enough? If you take volunteers who otherwise wouldn't participate in that online conversation, is that different enough? What are the aspects of this that are different from a legitimate campaign to change the opinions of others about a position you think is profoundly wrong. 2. One sad aspect of this is that it dilutes the impact of anonymous or semi-anonymous voices on the web. This was a short-cut to influence for ordinary web users. On the other hand, this is in a way natural, whenever the impact of a medium becomes popularly recognized, more people will try to utilize that medium, and its impact will be diluted. In the same way, just having a website used to be a short-cut to influence, but now has little effect.

stemc43 12y ago
US gov. actually admitted that it has a trolling network - it used to be called "viral peace". They claim it operates on "terrorist websites". And of course they give no definition of "terrorist websites" - But I think it's safe to assume reddit, twitter, facebook are terrorist websites.
- schoen 12y ago
  They also have some openly governmental accounts called "Think Again Turn Away", like https://twitter.com/ThinkAgain_DOS https://twitter.com/ThinkAgain_DOS https://www.facebook.com/ThinkAgainTurnAway https://www.facebook.com/ThinkAgainTurnAway They used to directly engage in debate with people more, while now they seem to have switched to a more posting-only style.

wyck 12y ago

This has been going on in one form of another throughout history. But if you want to sway opinion on the internet, it's not that easy. There are a lot of voices and noise in the mix, compared to how it was a mere 20 years ago. This is certainly important, but in the context of history, how people consume information is on a much better track, in my opinion things are actually getting better if you look at the big picture.

fiatmoney 12y ago

2013's "Most Reddit Addicted City" was Eglin Air Force Base. http://www.redditblog.com/2013/05/get-ready-for-global-reddit-meetup-day.html http://www.redditblog.com/2013/05/get-ready-for-global-reddi...

superuser2 12y ago
Air Force Bases are also full of technologically inclined 18 to 25 year old men, i.e the Reddit demographic.

Animats 12y ago

I've made this point before about search engine ranking. Most of the popularity signals used by search engines can be, and are, faked. I have two papers on this: "Social is bad for search, and search is bad for social" (2012), or how Google's use of social signals backfired, badly. http://www.sitetruth.com/doc/socialisbadforsearch09.pdf http://www.sitetruth.com/doc/socialisbadforsearch09.pdf How Google's use of social signals backfired, badly. "'Places' spam - the new front in the spam wars", or how Google's use of "local" information backfired. "http://www.sitetruth.com/doc/placesspam10.pdf" http://www.sitetruth.com/doc/placesspam10.pdf" (2010) Google's merge of data from Google Maps into the main search engine results created a whole new branch of local SEO spam. The fundamental problem is that the creation of fake online identities is cheap and easy. This can be partially fixed by taking a tough line on identity, but that's hard for services which are either big or have only a casual connection with their users. Facebook ran into the gay agenda enforcing a real names policy. The mobile guys can at least make people buy a phone to fake an identity. (A phone number is not enough; you can rent fake phone numbers. See "http://www.attlines.com http://www.attlines.com) An app that phones home with too much user information, though, is hard to fake cheaply. Yelp can tell if your phone has been to the place you're rating.

jordigh 12y ago
> the gay agenda I'm not sure if you know about this, but this is typically a term of derision and a strawman by anti-gay and homophobic groups. http://en.wikipedia.org/wiki/Homosexual_agenda http://en.wikipedia.org/wiki/Homosexual_agenda
- seanp2k2 12y ago
  +1; there are also lots of other marginalized groups of people who stand to lose something if aliases cannot be used. Edit: for example: http://www.theguardian.com/commentisfree/2014/sep/29/facebooks-real-names-policy-is-legal-but-its-also-problematic-for-free-speech http://www.theguardian.com/commentisfree/2014/sep/29/faceboo...
  - teddyh 12y ago

patcon 12y ago

Call me crazy, but it /has/ crossed my mind that #gamergate might have been a large-scale test of the ability to create widespread conflict and some degree of community crisis from thin air through sock-puppetry... EDIT: I honestly don't totally understand what gamergate is, but am in awe of the resurging energy of the whole thing, and hence the observation

deleted 12y ago
[deleted]

Udik 12y ago

Hmmm. The only verified case of sophisticated, organized sockpuppetry in the wild mentioned by the article seem to be related to the Israeli- Palestinian conflict and anti-semitism, and in every case they seem to be orchestrated to favour Israel. But all the comments here seem concentrated on the NSA, AIs, and even abortion. Now downvote me :)

krapp 12y ago
Then clearly either you, or the article, or both are just another reeking layer in the American government's propaganda onion. Edward Snowden clearly proved, scientifically, that the NSA/CIA/Military Industrial Complex controls everything and everyone I don't personally agree with, especially on the internet. Now go tell your controllers your mission has failed, Udik. If that even is your real name.