6 ms·

'Troll hunting' algorithm could make web a better place

36 points by oska 12y ago

paulhauggis 12y ago
"including negative words and profanities." This is why I have a problem with things like this. "negative words" might be why I disagree with the politics of the article...and poor spelling and grammar? Are we now discriminating against a certain type of person now? Do people that can't spell very well deserve to have their opinions ignored? This seems like nothing more than a smoke-screen to silence opposing view points and opinions. The up/down vote buttons already serve this purpose quite well. The same is already happening on Twitter. Activist groups against discrimination pass out lists of people to perma-ban based on nothing more but opposing opinions...doing the same exact thing they are supposed to be against.
- olefoo 12y ago
  > Do people that can't spell very well deserve to have their opinions ignored? Yes. Indeed they do. Good spelling is a mark of conscientiousness and attention to detail. Bad spelling is indicative of loose morals and an addiction to video games and misogyny that should be extirpated from polite society. Note also that users of a public channel ( Twitter ) choosing to accept other peoples suggestions as to who they should ignore so that they are not subjected to the vilest forms of verbal/textual abuse that humanity has invented; is not censorship. Relevant XKCD https://xkcd.com/1357/ https://xkcd.com/1357/
  - bradjohnson 12y ago
    >Bad spelling is indicative of loose morals and an addiction to video games and misogyny that should be extirpated from polite society. Is that a joke? How can you possibly justify that conclusion?
    - TronPaul_ 12y ago
      I was waiting for the /s as well. Having spent some time on the internet I've spoken and chatted with people for whom English is a second language. Are you telling me that their deficient understanding of English indicates a lack of morality? Is the English language a religion now? Guess I didn't get the memo.
      - thaumasiotes 12y ago
        related: http://languagelog.ldc.upenn.edu/nll/?p=18667 http://languagelog.ldc.upenn.edu/nll/?p=18667 The author is the daughter of Czech emigrants, writing about her experience visiting the Czech Republic, with an focus on the difference between learning a new language as an adult and re-learning a language you once knew. From the link: > There can be a striking lack of accommodation or cooperation on the part of listeners [in rural Czech areas]. Once, when traveling with my brother, I watched him flounder in Czech at a small town gas station, trying to convey which pack of cigarettes he wanted to purchase—he had forgotten the brand name, and was trying to describe the appearance of the package. Ignoring his pointing gestures, the cashier sat stone-faced through his attempts. When she finally identified what he wanted, she tossed the cigarettes on the counter, saying contemptuously, “As you can see, the package is red, not pink.” My brother apologized, “I’m sorry, my Czech is very bad.” “I can see that,” she replied without cracking the slightest smile. Experience with second-language learners is, speaking broadly, vanishingly rare among humans. Most people never meet any language learners. The rural Czechs are going by a rule of thumb: someone speaking that badly is more likely to be severely mentally retarded than anything else. (And they're correct about that! If they started seeing a flood of foreign tourists with rudimentary Czech, they'd adjust accordingly.)
  - return0 12y ago
    > accept other peoples suggestions Are you insinuating they should accept suggestions from other nations and tribes? Or did you just omit the "'" , thus exposing yourself as a misspellist, a loosemoralist , a videogamist and misogynist? I hope the Trolly Inquisition finds you and burns your router!
  - flohofwoe 12y ago
    I can't tell whether this is serious or a joke, I hope it's a joke since it sounds like taken right from the McCarthy era. Ah, now I understand, you're trying to be a troll, very clever ;)
    - olefoo 12y ago
      The gentleman to whom I was replying made a reference to shared blocklists as censorship. Having had to support a couple of friends through the showers of verbal excrement that get thrown their way by a group that call itself "Gamergate". I am less than sympathetic to the viewpoint that someone can demand the right to be listened to as well as the right to speak. Allow me to be explicitly and excruciatingly clear on that point. You are free to say whatever you want, to print it, to publish it on the internet, to build signs on your property, etc. without fearing arrest or detention. But; you do not have the right to demand that others spend even a fraction of a second of their time on this earth paying attention to you. And by loose morals I mean that disgusting sense of entitlement to others time and bodily affection that seems to characterize the "Gamergate community".
  - hewhowhineth 12y ago
    You forgot to add pedophilia, racism, terrorism, and drugs to that list. Hundreds of millions of people all over the world speak English as their second, third or fourth language. That fact alone makes them more interesting than some grammar sharia asshole. Edit: grammar :)
- karmacondon 12y ago
  They didn't just choose those criteria out of thin air. They compiled data and ran a logistic regression that was able to predict whether or a not a given user would be banned from a web community based on the content of their posts. And it turned out that negative words, profanity, spelling and grammar were all predictive features in determining that result. Here's how science works: If you don't like the conclusions, you can question the methodology. You can conduct the experiment yourself to see if you can reproduce the results. You can run your own studies and see if you come up with different conclusions. But you don't get to say "Hmm, the data don't conform to my preconceived notions about how the world should work, so I reject their implications." Everyone doesn't agree with the idea that every thought from every person should be heard at all times. Some people are just jerks. "Trolling", by definition, is about making provocative statements to incite reactions, as opposed to contributing to conversations. There are places where that kind of thing is accepted (4chan) and places where it is not (hn). Tools that help communities to encourage their preferred forms of communication are good things, and will be increasingly important in the future.
  - xnull2guest 12y ago
    Right, but you can find correlations between being black and committing crime - compiling data and running logistic regressions be damned. These measures they came up with are heuristic proxy measures at the very best, and noise at the worst. The troll hunting algorithm has to face the false positive problem [1], which the paper does not address. My very legitimate content has been censored various places (notably Facebook) because it tripped 'anti-trolling and scam algorithms' but the things I was trying to post were Snowden and Manning, and TPP leaks. [1] http://understandinguncertainty.org/node/238 http://understandinguncertainty.org/node/238
  - 6d0debc071 12y ago
    > Some people are just jerks and "trolling", by definition, is about making provocative statements to incite reactions, as opposed to contributing to conversations. I believe the core of paulhauggis' point was closer to the idea that trolling and behaviour that leads to being banned are not the same thing. Scope in studies is as important as anything else.
SeanDav 12y ago
One of the things that I take from this article is that it appears to support the use of hellbanning as an anti troll measure. Not that hellbanning is a perfect solution - like everything else it depends on how it is implemented and managed.
- ryanlol 12y ago
  There is no good way to implement hellbans, the only people affected by them are the ones not expecting to get banned.
  - jfoster 12y ago
    Why do you think people expecting to get banned wouldn't be affected by them? Those that are aware of the existence of hellbanning probably won't be impeded very much, but I'm not convinced that hellbanning is widely known of. I'm sure lots of HN participants are aware of the existence of hellbans, but I think your average troll may not be. They may eventually figure it out, but if it slows them down, it has already done something good.
    - ryanlol 12y ago
      Hellbanning is most definitely widely known of where it is used. It maybe useful in a few individual cases, but most definitely not always. (And that's all ignoring the ethical issues)
      - danneu 12y ago
        I don't know about that, and it depends how it's implemented. HN broadcasts the fact that it hellbans users. There's even a toggle in your profile to see the posts of the damned. HN also seems to gratuitously hellban people - if someone isn't hellbanned for being an obvious troll, sometimes they seem to be hellbanned over lightweight snark or nothing at all. HN even has its own notoriously hellbanned "HN celebrity": the creator of Temple OS. It's no secret. Yet every day I notice hellbanned posts where the user has been hellbanned for some time now and they do not notice. On my forum, I only hellban users so uncontroversially egregious (bots, spammers, and people who only post to pilfer value from the community) that nobody notices or would care if they did. To "expose" the system, a user would pretty much have to create a topic like this: "Hey everyone, you don't know me because I've been trying to PM new users with links to my competing forum for half a year, but I just found out that nobody has been receiving my PMs because I've been hellbanned. Help me protest this system!"
Hytosys 12y ago
A broad brush[1] is acceptable if you're unable to use the Internet without being harassed by "contrarians". I am someone who is not often harassed online, but I am involved with people who frequently experience harassment. I've been using Twitter block lists[2] and have noticed a great absence of hateful responses. This is not a long-term solution, but it lowers the number of times a day that my heart sinks. I accept any incidental losses. The article suggests a finer brush using something like crowdsourcing. Perhaps harassment targets could pay to submit their profiles for monitoring, and the resultant block list could be made public for subscription by anyone on Block Together[2]. It would be nice to see networks take a stand against repeat harassers, but clearly people are tired of waiting around for that. [1] https://github.com/freebsdgirl/ggautoblocker https://github.com/freebsdgirl/ggautoblocker [2] https://blocktogether.org/ https://blocktogether.org/ EDIT: Without responses, I'm coming to some disappointing conclusions.
- return0 12y ago
  Where does 'written harassment' ends and where does groupthink begin? Should such harassment be an actual problem, there would already be such filtered alternatives to twitter and facebook. Right now i see it as an attempt to silence unfavorable opinions.
  - Hytosys 12y ago
    Could you expand on your first thought? I'm not sure I understand your intended connection between harassment and groupthink. In the hypothetical situation that harassment is prevalent, I'm curious as to what your alternatives to Twitter/Facebook would look like. What would the requirements of joining such a community be?
    - return0 12y ago
      If you start censoring people, soon you will find yourself only allowing the opinions of the majority. An alternative to twitter would be filtered, just as you suggest. It could even exclude "offending" opinions. There are plenty of easily offended (but very vocal) people who think all public channels should be like that. However we don't see any startup that is advocating censored speak catching up (I don't even know if there are any, but the social startup space is so overcrowded that there must be some).
reitanqild 12y ago
I read this totally expecting something very politically correct and was delighted to see someone have a neutral, technical take on this.
xnull2guest 12y ago
This has everything to do with Russian and other foreign propagandists and virtually nothing to do with real trolls and trolling. Unfortunately the wide net this will cast will undoubtedly catch legitimate free speech by citizens. How is NK hacking SONY 'attacking free speech' while censoring political comments, civilian or otherwise, trolling or otherwise, not? Is our nation under that much of a threat? I know America's power is waning, but if it is waning this badly we should all get very worried.
nothunttroll 12y ago
Supposing I was a troll, I would like an algorithm that correct my spelling and grammar, and at the same time gives a watered down form,non conflict form and zen way of speaking, to my colorful words. It should take care mainly of the first five or ten lines just to scape from the hunting algorithm. If some mega corporation is stealing your properties and rights, it should phrase that as somethign like: An act not really understood by this person is being performed by a big company, and you are feeling a little sad about it. Counterexamples: What happens when you apply that algorithm to R. Stallman, Linux Torvald, or people with strong opinions? Sometimes those supposed trolls are the person with the real information and correct views and not the blind sheep. By the way the former president of the FMI is now in a very difficult position in Spain, surely algorithms doesn't work so well to detect what they should detect. Disclaimer: Not a native speaker. Edit: grammar. Added counterexamples.
- xnull2guest 12y ago
  My prediction would be that websites would develop private versions of their troll hunting algorithm that would include input from notifications and information sharing programs (PRISM, TAXII). This would enable services to respond to government requests with minimum overhead to respond to changes in foreign 'diplomacy operations/HUMINT'.

36 points by oska 12y ago

"including negative words and profanities." This is why I have a problem with things like this. "negative words" might be why I disagree with the politics of the article...and poor spelling and grammar? Are we now discriminating against a certain type of person now? Do people that can't spell very well deserve to have their opinions ignored? This seems like nothing more than a smoke-screen to silence opposing view points and opinions. The up/down vote buttons already serve this purpose quite well. The same is already happening on Twitter. Activist groups against discrimination pass out lists of people to perma-ban based on nothing more but opposing opinions...doing the same exact thing they are supposed to be against.

olefoo 12y ago
> Do people that can't spell very well deserve to have their opinions ignored? Yes. Indeed they do. Good spelling is a mark of conscientiousness and attention to detail. Bad spelling is indicative of loose morals and an addiction to video games and misogyny that should be extirpated from polite society. Note also that users of a public channel ( Twitter ) choosing to accept other peoples suggestions as to who they should ignore so that they are not subjected to the vilest forms of verbal/textual abuse that humanity has invented; is not censorship. Relevant XKCD https://xkcd.com/1357/ https://xkcd.com/1357/
- bradjohnson 12y ago
  >Bad spelling is indicative of loose morals and an addiction to video games and misogyny that should be extirpated from polite society. Is that a joke? How can you possibly justify that conclusion?
  - TronPaul_ 12y ago
    I was waiting for the /s as well. Having spent some time on the internet I've spoken and chatted with people for whom English is a second language. Are you telling me that their deficient understanding of English indicates a lack of morality? Is the English language a religion now? Guess I didn't get the memo.
    - thaumasiotes 12y ago
      related: http://languagelog.ldc.upenn.edu/nll/?p=18667 http://languagelog.ldc.upenn.edu/nll/?p=18667 The author is the daughter of Czech emigrants, writing about her experience visiting the Czech Republic, with an focus on the difference between learning a new language as an adult and re-learning a language you once knew. From the link: > There can be a striking lack of accommodation or cooperation on the part of listeners [in rural Czech areas]. Once, when traveling with my brother, I watched him flounder in Czech at a small town gas station, trying to convey which pack of cigarettes he wanted to purchase—he had forgotten the brand name, and was trying to describe the appearance of the package. Ignoring his pointing gestures, the cashier sat stone-faced through his attempts. When she finally identified what he wanted, she tossed the cigarettes on the counter, saying contemptuously, “As you can see, the package is red, not pink.” My brother apologized, “I’m sorry, my Czech is very bad.” “I can see that,” she replied without cracking the slightest smile. Experience with second-language learners is, speaking broadly, vanishingly rare among humans. Most people never meet any language learners. The rural Czechs are going by a rule of thumb: someone speaking that badly is more likely to be severely mentally retarded than anything else. (And they're correct about that! If they started seeing a flood of foreign tourists with rudimentary Czech, they'd adjust accordingly.)
- return0 12y ago
  > accept other peoples suggestions Are you insinuating they should accept suggestions from other nations and tribes? Or did you just omit the "'" , thus exposing yourself as a misspellist, a loosemoralist , a videogamist and misogynist? I hope the Trolly Inquisition finds you and burns your router!
- flohofwoe 12y ago
  I can't tell whether this is serious or a joke, I hope it's a joke since it sounds like taken right from the McCarthy era. Ah, now I understand, you're trying to be a troll, very clever ;)
  - olefoo 12y ago
    The gentleman to whom I was replying made a reference to shared blocklists as censorship. Having had to support a couple of friends through the showers of verbal excrement that get thrown their way by a group that call itself "Gamergate". I am less than sympathetic to the viewpoint that someone can demand the right to be listened to as well as the right to speak. Allow me to be explicitly and excruciatingly clear on that point. You are free to say whatever you want, to print it, to publish it on the internet, to build signs on your property, etc. without fearing arrest or detention. But; you do not have the right to demand that others spend even a fraction of a second of their time on this earth paying attention to you. And by loose morals I mean that disgusting sense of entitlement to others time and bodily affection that seems to characterize the "Gamergate community".
- hewhowhineth 12y ago
  You forgot to add pedophilia, racism, terrorism, and drugs to that list. Hundreds of millions of people all over the world speak English as their second, third or fourth language. That fact alone makes them more interesting than some grammar sharia asshole. Edit: grammar :)
karmacondon 12y ago
They didn't just choose those criteria out of thin air. They compiled data and ran a logistic regression that was able to predict whether or a not a given user would be banned from a web community based on the content of their posts. And it turned out that negative words, profanity, spelling and grammar were all predictive features in determining that result. Here's how science works: If you don't like the conclusions, you can question the methodology. You can conduct the experiment yourself to see if you can reproduce the results. You can run your own studies and see if you come up with different conclusions. But you don't get to say "Hmm, the data don't conform to my preconceived notions about how the world should work, so I reject their implications." Everyone doesn't agree with the idea that every thought from every person should be heard at all times. Some people are just jerks. "Trolling", by definition, is about making provocative statements to incite reactions, as opposed to contributing to conversations. There are places where that kind of thing is accepted (4chan) and places where it is not (hn). Tools that help communities to encourage their preferred forms of communication are good things, and will be increasingly important in the future.
- xnull2guest 12y ago
  Right, but you can find correlations between being black and committing crime - compiling data and running logistic regressions be damned. These measures they came up with are heuristic proxy measures at the very best, and noise at the worst. The troll hunting algorithm has to face the false positive problem [1], which the paper does not address. My very legitimate content has been censored various places (notably Facebook) because it tripped 'anti-trolling and scam algorithms' but the things I was trying to post were Snowden and Manning, and TPP leaks. [1] http://understandinguncertainty.org/node/238 http://understandinguncertainty.org/node/238
- 6d0debc071 12y ago
  > Some people are just jerks and "trolling", by definition, is about making provocative statements to incite reactions, as opposed to contributing to conversations. I believe the core of paulhauggis' point was closer to the idea that trolling and behaviour that leads to being banned are not the same thing. Scope in studies is as important as anything else.

SeanDav 12y ago

One of the things that I take from this article is that it appears to support the use of hellbanning as an anti troll measure. Not that hellbanning is a perfect solution - like everything else it depends on how it is implemented and managed.

ryanlol 12y ago
There is no good way to implement hellbans, the only people affected by them are the ones not expecting to get banned.
- jfoster 12y ago
  Why do you think people expecting to get banned wouldn't be affected by them? Those that are aware of the existence of hellbanning probably won't be impeded very much, but I'm not convinced that hellbanning is widely known of. I'm sure lots of HN participants are aware of the existence of hellbans, but I think your average troll may not be. They may eventually figure it out, but if it slows them down, it has already done something good.
  - ryanlol 12y ago
    Hellbanning is most definitely widely known of where it is used. It maybe useful in a few individual cases, but most definitely not always. (And that's all ignoring the ethical issues)
    - danneu 12y ago
      I don't know about that, and it depends how it's implemented. HN broadcasts the fact that it hellbans users. There's even a toggle in your profile to see the posts of the damned. HN also seems to gratuitously hellban people - if someone isn't hellbanned for being an obvious troll, sometimes they seem to be hellbanned over lightweight snark or nothing at all. HN even has its own notoriously hellbanned "HN celebrity": the creator of Temple OS. It's no secret. Yet every day I notice hellbanned posts where the user has been hellbanned for some time now and they do not notice. On my forum, I only hellban users so uncontroversially egregious (bots, spammers, and people who only post to pilfer value from the community) that nobody notices or would care if they did. To "expose" the system, a user would pretty much have to create a topic like this: "Hey everyone, you don't know me because I've been trying to PM new users with links to my competing forum for half a year, but I just found out that nobody has been receiving my PMs because I've been hellbanned. Help me protest this system!"

Hytosys 12y ago

A broad brush[1] is acceptable if you're unable to use the Internet without being harassed by "contrarians". I am someone who is not often harassed online, but I am involved with people who frequently experience harassment. I've been using Twitter block lists[2] and have noticed a great absence of hateful responses. This is not a long-term solution, but it lowers the number of times a day that my heart sinks. I accept any incidental losses. The article suggests a finer brush using something like crowdsourcing. Perhaps harassment targets could pay to submit their profiles for monitoring, and the resultant block list could be made public for subscription by anyone on Block Together[2]. It would be nice to see networks take a stand against repeat harassers, but clearly people are tired of waiting around for that. [1] https://github.com/freebsdgirl/ggautoblocker https://github.com/freebsdgirl/ggautoblocker [2] https://blocktogether.org/ https://blocktogether.org/ EDIT: Without responses, I'm coming to some disappointing conclusions.

return0 12y ago
Where does 'written harassment' ends and where does groupthink begin? Should such harassment be an actual problem, there would already be such filtered alternatives to twitter and facebook. Right now i see it as an attempt to silence unfavorable opinions.
- Hytosys 12y ago
  Could you expand on your first thought? I'm not sure I understand your intended connection between harassment and groupthink. In the hypothetical situation that harassment is prevalent, I'm curious as to what your alternatives to Twitter/Facebook would look like. What would the requirements of joining such a community be?
  - return0 12y ago
    If you start censoring people, soon you will find yourself only allowing the opinions of the majority. An alternative to twitter would be filtered, just as you suggest. It could even exclude "offending" opinions. There are plenty of easily offended (but very vocal) people who think all public channels should be like that. However we don't see any startup that is advocating censored speak catching up (I don't even know if there are any, but the social startup space is so overcrowded that there must be some).

reitanqild 12y ago

I read this totally expecting something very politically correct and was delighted to see someone have a neutral, technical take on this.

xnull2guest 12y ago

This has everything to do with Russian and other foreign propagandists and virtually nothing to do with real trolls and trolling. Unfortunately the wide net this will cast will undoubtedly catch legitimate free speech by citizens. How is NK hacking SONY 'attacking free speech' while censoring political comments, civilian or otherwise, trolling or otherwise, not? Is our nation under that much of a threat? I know America's power is waning, but if it is waning this badly we should all get very worried.

nothunttroll 12y ago

Supposing I was a troll, I would like an algorithm that correct my spelling and grammar, and at the same time gives a watered down form,non conflict form and zen way of speaking, to my colorful words. It should take care mainly of the first five or ten lines just to scape from the hunting algorithm. If some mega corporation is stealing your properties and rights, it should phrase that as somethign like: An act not really understood by this person is being performed by a big company, and you are feeling a little sad about it. Counterexamples: What happens when you apply that algorithm to R. Stallman, Linux Torvald, or people with strong opinions? Sometimes those supposed trolls are the person with the real information and correct views and not the blind sheep. By the way the former president of the FMI is now in a very difficult position in Spain, surely algorithms doesn't work so well to detect what they should detect. Disclaimer: Not a native speaker. Edit: grammar. Added counterexamples.

xnull2guest 12y ago
My prediction would be that websites would develop private versions of their troll hunting algorithm that would include input from notifications and information sharing programs (PRISM, TAXII). This would enable services to respond to government requests with minimum overhead to respond to changes in foreign 'diplomacy operations/HUMINT'.