6 ms·

This paper is deeply flawed. It basically says "we rebuked one study, therefore bots aren't real." Anyone who has been on Reddit for the last year has seen go

This paper is deeply flawed. It basically says "we rebuked one study, therefore bots aren't real." Anyone who has been on Reddit for the last year has seen government run botnets posting spintext by the hundreds. Thinking there are no bots is wildly naive

Jiro 5y ago
It's the most common bot detection software and was used by over a thousand research papers.
freebreakfast 5y ago
The paper does not claim "bots aren't real." It claims: > The field of “social bot” research is fundamentally flawed. While “social bot” researchers have received an enormous amount of public attention, their methods are highly dubious. And, as we and others have demonstrated, they fail miserably and consistently when evaluated under real-world conditions. Studies claiming to investigate the prevalence, properties, or influence of “social bots” have, in reality, just investigated false positives and artifacts of the flawed detection methods employed.
mypastself 5y ago
The paper found that the vast majority of Twitter users classified as bots were actually human. They analyzed data sets previously used by three organizations responsible for the majority of available papers on the topic. The authors of the paper concluded that, in the future, any claims that withhold the list of account handles classified as bots should be ignored because the claims can’t be independently verified. So they didn’t rebuke just one study but datasets underlying most studies on the topic, they analyzed Twitter rather than Reddit, and they did not claim that bots don’t exist. If I am mistaken, someone who’s analyzed the paper in more depth than I did can correct me.
elcapitan 5y ago
I think this is specific about the notion of "social bots", i.e. bots that are mostly indistinguishable from humans or could even interact with them in a realistic way. Obviously everybody can write a spambot, or click a "post this on twitter" button (which is 90% of what people mistake for "bot posts", anyway).
mike_hearn 5y ago
Their research shows nearly all social bot papers are wrong, not just a single paper. I've summarized the paper and talk here, along with some more of my own inputs (I used to work on bot fighting): https://blog.plan99.net/fake-science-part-ii-bots-that-are-not-c66129e5e3f5 https://blog.plan99.net/fake-science-part-ii-bots-that-are-n... One of the points I make is that social bot research is a prolific academic field. There are nearly 10,000 papers indexed on Google Scholar on the topic and that's not all of them. And basically all of them are fantasy. Clearly, bots do exist, but the techniques in these papers are not able to detect them reliably and the authors don't seem to care.
- lmeyerov 5y ago
  "All" is an ambitious claim. They show some papers use a bad bot detector with even worse (and unrecommended) settings. Worse, they explicitly skip bot accounts that twitter confirmed, and thus evaluate on accounts twitter didn't think were bots. That kind of mistep means it fails to show botometer is a good/bad classifier (even if we all know that), so until they fix their methodology, their misreproduction is bad scientific grounds for disqualifying all papers using it. This is why we have peer review and you can pick what level of peer you trust your work at. I like the intent of the paper. But, I wouldn't have let it pass peer review as-is. I do agree social scientists deserve better tools because botometer is more of a crude footgun at this point. A lot has happened since botometer, and what may have made some sense 5-10 years ago doesn't make sense today. They could have written a scientific paper showing this and I'd 100% believe it, but instead they used garbage methodolgy and people are inferring whatever they want from that.
  - mike_hearn 5y ago
    Could you point out where they say they ignored all the accounts shut down by Twitter? In most cases they can't even do this because the sets of accounts identified as 'bots' by othe researchers aren't made public. If you look at their methodology for proving Botometer is flawed, it's based on taking sets of verified human Twitter accounts like US Congress, none of which were shut down by Twitter, and then showing high FP rates (like 50%). Some of the papers mentioned in the analysis don't use Botornot, but are still deeply flawed. I should point out here that I'm cited by the authors (Hearn2017) and the paper I looked at doesn't use Botometer at all, nor did I exclude accounts identified by Twitter for the simple reason that no specific accounts were named by the researchers at all. "I like the intent of the paper. But, I wouldn't have let it pass peer review as-is." Then you will be pleased to know that despite peer reviewers signing off on thousands of incorrect, evidence-free papers talking about social bots, this paper - which provides ample proof of its claims - remains a preprint because none of the journals interested in social bots are willing to publish it. What a surprise. You sound like someone who is or was a social scientist? I think you should take it from a guy who did get paid to fight bots at Google: none of this research matters to or is read by the people whom it's supposed to benefit. Industry ignores it or occasionally points out that it's flawed, as Twitter did here: https://blog.twitter.com/en_us/topics/company/2020/bot-or-not https://blog.twitter.com/en_us/topics/company/2020/bot-or-no... "What about tools like Botometer & Bot Sentinel? ... This is an extremely limited approach ... These tools do not account for these common Twitter use cases, how far we’ve come, and how things have evolved. As a result nuance can be lost. The outcome? Binary judgments of who’s a “bot or not”, which have real potential to poison our public discourse — particularly when they are pushed out through the media."
deleted 5y ago
[deleted]

Jiro 5y ago

It's the most common bot detection software and was used by over a thousand research papers.

freebreakfast 5y ago

The paper does not claim "bots aren't real." It claims: > The field of “social bot” research is fundamentally flawed. While “social bot” researchers have received an enormous amount of public attention, their methods are highly dubious. And, as we and others have demonstrated, they fail miserably and consistently when evaluated under real-world conditions. Studies claiming to investigate the prevalence, properties, or influence of “social bots” have, in reality, just investigated false positives and artifacts of the flawed detection methods employed.

mypastself 5y ago

The paper found that the vast majority of Twitter users classified as bots were actually human. They analyzed data sets previously used by three organizations responsible for the majority of available papers on the topic. The authors of the paper concluded that, in the future, any claims that withhold the list of account handles classified as bots should be ignored because the claims can’t be independently verified. So they didn’t rebuke just one study but datasets underlying most studies on the topic, they analyzed Twitter rather than Reddit, and they did not claim that bots don’t exist. If I am mistaken, someone who’s analyzed the paper in more depth than I did can correct me.

elcapitan 5y ago

I think this is specific about the notion of "social bots", i.e. bots that are mostly indistinguishable from humans or could even interact with them in a realistic way. Obviously everybody can write a spambot, or click a "post this on twitter" button (which is 90% of what people mistake for "bot posts", anyway).

mike_hearn 5y ago

Their research shows nearly all social bot papers are wrong, not just a single paper. I've summarized the paper and talk here, along with some more of my own inputs (I used to work on bot fighting): https://blog.plan99.net/fake-science-part-ii-bots-that-are-not-c66129e5e3f5 https://blog.plan99.net/fake-science-part-ii-bots-that-are-n... One of the points I make is that social bot research is a prolific academic field. There are nearly 10,000 papers indexed on Google Scholar on the topic and that's not all of them. And basically all of them are fantasy. Clearly, bots do exist, but the techniques in these papers are not able to detect them reliably and the authors don't seem to care.

lmeyerov 5y ago
"All" is an ambitious claim. They show some papers use a bad bot detector with even worse (and unrecommended) settings. Worse, they explicitly skip bot accounts that twitter confirmed, and thus evaluate on accounts twitter didn't think were bots. That kind of mistep means it fails to show botometer is a good/bad classifier (even if we all know that), so until they fix their methodology, their misreproduction is bad scientific grounds for disqualifying all papers using it. This is why we have peer review and you can pick what level of peer you trust your work at. I like the intent of the paper. But, I wouldn't have let it pass peer review as-is. I do agree social scientists deserve better tools because botometer is more of a crude footgun at this point. A lot has happened since botometer, and what may have made some sense 5-10 years ago doesn't make sense today. They could have written a scientific paper showing this and I'd 100% believe it, but instead they used garbage methodolgy and people are inferring whatever they want from that.
- mike_hearn 5y ago
  Could you point out where they say they ignored all the accounts shut down by Twitter? In most cases they can't even do this because the sets of accounts identified as 'bots' by othe researchers aren't made public. If you look at their methodology for proving Botometer is flawed, it's based on taking sets of verified human Twitter accounts like US Congress, none of which were shut down by Twitter, and then showing high FP rates (like 50%). Some of the papers mentioned in the analysis don't use Botornot, but are still deeply flawed. I should point out here that I'm cited by the authors (Hearn2017) and the paper I looked at doesn't use Botometer at all, nor did I exclude accounts identified by Twitter for the simple reason that no specific accounts were named by the researchers at all. "I like the intent of the paper. But, I wouldn't have let it pass peer review as-is." Then you will be pleased to know that despite peer reviewers signing off on thousands of incorrect, evidence-free papers talking about social bots, this paper - which provides ample proof of its claims - remains a preprint because none of the journals interested in social bots are willing to publish it. What a surprise. You sound like someone who is or was a social scientist? I think you should take it from a guy who did get paid to fight bots at Google: none of this research matters to or is read by the people whom it's supposed to benefit. Industry ignores it or occasionally points out that it's flawed, as Twitter did here: https://blog.twitter.com/en_us/topics/company/2020/bot-or-not https://blog.twitter.com/en_us/topics/company/2020/bot-or-no... "What about tools like Botometer & Bot Sentinel? ... This is an extremely limited approach ... These tools do not account for these common Twitter use cases, how far we’ve come, and how things have evolved. As a result nuance can be lost. The outcome? Binary judgments of who’s a “bot or not”, which have real potential to poison our public discourse — particularly when they are pushed out through the media."

deleted 5y ago

[deleted]