5 ms·

Internet trolls: Proactive policing

19 points by ghosh 12y ago

diminoten 12y ago
The work is here: http://arxiv.org/pdf/1504.00680v1.pdf http://arxiv.org/pdf/1504.00680v1.pdf According to the data, FBUs (future banned users, aka trolls) are significantly more readable in their writing than NBUs (never banned users). That's certainly surprising. Also of note is that FBUs tend to reply more than NBUs, which actually meshes with some features of HN -- the inability to reply to a nested comment from within the nest (one has to click on the comment itself to reply -- though this feature has always been somewhat ambiguous and not really deterrent of impulsive replies; I don't even notice it anymore). Also, I think HN deflates the score of posts with many comments, a strategy supported by this line in the paper: "Prior work also identified post frequency as a signal of a low quality discussion (Diakopoulos and Naaman 2011)." An absolutely fascinating paper. What are the consequences, from an IP standpoint, of taking this paper and implementing features based on its conclusions?
- existencebox 12y ago
  Wish I had something more constructive to add than this hand waving, but perhaps an interpretation (via metaphor) of the first point: If you fight with a weapon, don't you become skilled with it?
- michaelkeenan 12y ago
  It's a great paper, and a great application of machine learning! Though, the FBUs had worse readability than NBUs. I think you're misinterpreting the readability index - higher is worse, not better. From the paper: "Users who get banned in the future (FBUs) (a) write less similarly to other users in the same thread, (b) write posts that are harder to read (i.e., have a higher readability index), and (c) express less positive emotion."
  - diminoten 12y ago
    I saw that later on, I was specifically referencing chart 1(b).
- bduerst 12y ago
  Depends on the implementation. As a pre-banning feature, this mechanism from the paper may be a little heavy handed. Though the mechanism can be used to sift comment content, and highlight/raise the best content to the top based on user behavior. I'm surprised someone hasn't applied ML to reddit comments, in an attempt to predict comment quality. They certainly have the content for it.
mindslight 12y ago
Trolling is the inevitable reaction to groupthink. Predictably, groupthink is interested in ways of insulating itself from trolls.
- throwaway7767 12y ago
  > Trolling is the inevitable reaction to groupthink. Predictably, groupthink is interested in ways of insulating itself from trolls. The word 'trolling' seems to have taken on many definitions of late, you are probably using a different one than the article. I understood the article to mean troll in the sense of a person purposefully misdirecting discussion to make people angry (note that the troll in this sense usually does not believe what they are saying, they're "trolling" for reactions). It sounds like you are using the more recent use of "trolling" to mean "to disagree with someone".
  - jkyle 12y ago
    It sounds like you are using the more recent use of "trolling" to mean "to disagree with someone". I have always heard of trolling, within the context of internet forum discussions, used to refer to the former and not this later definition.
  - sp332 12y ago
    The article refers to "messages that contain abusive language; accounts set up specifically to target individuals; and threats of violence" as trolling. That seems different from just anti-groupthink.
    - sqrt17 12y ago
      The article is reporting about a scientific study (http://arxiv.org/pdf/1504.00680v1.pdf http://arxiv.org/pdf/1504.00680v1.pdf discussed in https://news.ycombinator.com/item?id=9399090 https://news.ycombinator.com/item?id=9399090) that specifically talks about "future banned users" without asking why these users would be banned; The rest of the economist article is journalistic speculation. The study authors do report that there's a propensity for polarization, i.e., that "we find a community bias against FBUs: posts written later in an FBU’s life are more likely to be deleted, regardless of whether they are actually worse".
mlvljr 12y ago
We'll feed on you!!1
bglazer 12y ago
The paper itself [1] and a Wired article covering it [2], were received very poorly on HN when they were originally posted before. [1] https://news.ycombinator.com/item?id=9399090 https://news.ycombinator.com/item?id=9399090 [2] https://news.ycombinator.com/item?id=9398399 https://news.ycombinator.com/item?id=9398399 I honestly didn't expect that. My impression was that people are generally afraid that this will be used as a way to auto-ban people just for disagreeing with popular sentiment. I think what people didn't understand was that this analysis is heavily based on community and admin actions, primarily banned users and deleted posts. While there is some analysis of the comments themselves, like readability and word similarity, this is not some SkyNet auto-ban bot. They are predicting how likely a user is to be banned, based primarily on the community's response to that user. My point is that if systems like this are used to flag or ban people for disagreeing with the group think, then the problem is with the community, not the algorithm. Edit: typos and clarity
RcouF1uZ4gsC 12y ago
I don't think this technique is going to be all that helpful. This is because you cannot consider specificity and sensitivity without taking into account the prevalence. Let us look at the data from the paper [1]. From the chart on page 3, we can see that the percentage of banned ranges from 1.7% to 3.3%. On the last page we have the phrase "Though average classifier precision is relatively high (0.80), one in five users identified as antisocial are nonetheless misclassified". Based on this, I think we have a specificity of 80%, and a sensitivity of 80%. Let us look at 1000 hypothetical users, using 3% as the prevalence. Flagged and Not Flagged represent the output of the algorithm, and Regular and Banned represent "Gold Standard" |Regular | Banned | -------------------------------- Total |970 | 30 ---------------------------------- Flagged |194 | 24 ---------------------------------- Not Flagged | 776 | 6 In this example, the algorithm will flag 218 out of a thousand people as being trolls. 194 of the 218 people flagged are actually not trolls. Doing any sort of automated banning based on this algorithm will I think be unacceptable. I may have some of my stats or assumptions wrong, and I would love to be educated why if that is the case. 1. http://arxiv.org/pdf/1504.00680v1.pdf http://arxiv.org/pdf/1504.00680v1.pdf

19 points by ghosh 12y ago

The work is here: http://arxiv.org/pdf/1504.00680v1.pdf http://arxiv.org/pdf/1504.00680v1.pdf According to the data, FBUs (future banned users, aka trolls) are significantly more readable in their writing than NBUs (never banned users). That's certainly surprising. Also of note is that FBUs tend to reply more than NBUs, which actually meshes with some features of HN -- the inability to reply to a nested comment from within the nest (one has to click on the comment itself to reply -- though this feature has always been somewhat ambiguous and not really deterrent of impulsive replies; I don't even notice it anymore). Also, I think HN deflates the score of posts with many comments, a strategy supported by this line in the paper: "Prior work also identified post frequency as a signal of a low quality discussion (Diakopoulos and Naaman 2011)." An absolutely fascinating paper. What are the consequences, from an IP standpoint, of taking this paper and implementing features based on its conclusions?

existencebox 12y ago
Wish I had something more constructive to add than this hand waving, but perhaps an interpretation (via metaphor) of the first point: If you fight with a weapon, don't you become skilled with it?
michaelkeenan 12y ago
It's a great paper, and a great application of machine learning! Though, the FBUs had worse readability than NBUs. I think you're misinterpreting the readability index - higher is worse, not better. From the paper: "Users who get banned in the future (FBUs) (a) write less similarly to other users in the same thread, (b) write posts that are harder to read (i.e., have a higher readability index), and (c) express less positive emotion."
- diminoten 12y ago
  I saw that later on, I was specifically referencing chart 1(b).
bduerst 12y ago
Depends on the implementation. As a pre-banning feature, this mechanism from the paper may be a little heavy handed. Though the mechanism can be used to sift comment content, and highlight/raise the best content to the top based on user behavior. I'm surprised someone hasn't applied ML to reddit comments, in an attempt to predict comment quality. They certainly have the content for it.

mindslight 12y ago

Trolling is the inevitable reaction to groupthink. Predictably, groupthink is interested in ways of insulating itself from trolls.

throwaway7767 12y ago
> Trolling is the inevitable reaction to groupthink. Predictably, groupthink is interested in ways of insulating itself from trolls. The word 'trolling' seems to have taken on many definitions of late, you are probably using a different one than the article. I understood the article to mean troll in the sense of a person purposefully misdirecting discussion to make people angry (note that the troll in this sense usually does not believe what they are saying, they're "trolling" for reactions). It sounds like you are using the more recent use of "trolling" to mean "to disagree with someone".
- jkyle 12y ago
  It sounds like you are using the more recent use of "trolling" to mean "to disagree with someone". I have always heard of trolling, within the context of internet forum discussions, used to refer to the former and not this later definition.
- sp332 12y ago
  The article refers to "messages that contain abusive language; accounts set up specifically to target individuals; and threats of violence" as trolling. That seems different from just anti-groupthink.
  - sqrt17 12y ago
    The article is reporting about a scientific study (http://arxiv.org/pdf/1504.00680v1.pdf http://arxiv.org/pdf/1504.00680v1.pdf discussed in https://news.ycombinator.com/item?id=9399090 https://news.ycombinator.com/item?id=9399090) that specifically talks about "future banned users" without asking why these users would be banned; The rest of the economist article is journalistic speculation. The study authors do report that there's a propensity for polarization, i.e., that "we find a community bias against FBUs: posts written later in an FBU’s life are more likely to be deleted, regardless of whether they are actually worse".

mlvljr 12y ago

We'll feed on you!!1

bglazer 12y ago

The paper itself [1] and a Wired article covering it [2], were received very poorly on HN when they were originally posted before. [1] https://news.ycombinator.com/item?id=9399090 https://news.ycombinator.com/item?id=9399090 [2] https://news.ycombinator.com/item?id=9398399 https://news.ycombinator.com/item?id=9398399 I honestly didn't expect that. My impression was that people are generally afraid that this will be used as a way to auto-ban people just for disagreeing with popular sentiment. I think what people didn't understand was that this analysis is heavily based on community and admin actions, primarily banned users and deleted posts. While there is some analysis of the comments themselves, like readability and word similarity, this is not some SkyNet auto-ban bot. They are predicting how likely a user is to be banned, based primarily on the community's response to that user. My point is that if systems like this are used to flag or ban people for disagreeing with the group think, then the problem is with the community, not the algorithm. Edit: typos and clarity

RcouF1uZ4gsC 12y ago

I don't think this technique is going to be all that helpful. This is because you cannot consider specificity and sensitivity without taking into account the prevalence. Let us look at the data from the paper [1]. From the chart on page 3, we can see that the percentage of banned ranges from 1.7% to 3.3%. On the last page we have the phrase "Though average classifier precision is relatively high (0.80), one in five users identified as antisocial are nonetheless misclassified". Based on this, I think we have a specificity of 80%, and a sensitivity of 80%. Let us look at 1000 hypothetical users, using 3% as the prevalence. Flagged and Not Flagged represent the output of the algorithm, and Regular and Banned represent "Gold Standard" |Regular | Banned | -------------------------------- Total |970 | 30 ---------------------------------- Flagged |194 | 24 ---------------------------------- Not Flagged | 776 | 6 In this example, the algorithm will flag 218 out of a thousand people as being trolls. 194 of the 218 people flagged are actually not trolls. Doing any sort of automated banning based on this algorithm will I think be unacceptable. I may have some of my stats or assumptions wrong, and I would love to be educated why if that is the case. 1. http://arxiv.org/pdf/1504.00680v1.pdf http://arxiv.org/pdf/1504.00680v1.pdf