3 ms·

Section 3.2: the 10%+ of already-deleted accounts were removed. You can often actually infer why an account was removed, but they didn't. Furthermore, it is qu

Section 3.2: the 10%+ of already-deleted accounts were removed. You can often actually infer why an account was removed, but they didn't. Furthermore, it is questionable to me to know 10% were already confirmed removed and then state the reproducers found all of the remaining were great. If I went deeper, I bet even more issues. As is, this is already enough to remove any word like "all" and various weasel word phrasings. --- Ad-hominem shouldn't matter. But: Professionally, I work with top gov agencies, top 3 cloud providers, F500s, startups, misinfo teams, etc. to provide pluggable core tech for their core graph intelligence pipelines (Graphistry), such as for sigint tasks. Botnets, hackers, account take overs, account abuse, etc - threat hunting, threat research, AML teams, digital human trafficking. As a #data4good effort, we help non-profits & researchers on the same things, such as via our free tier & some of our own hands-on help. We're just a tiny startup, but helped with massive international takedowns, flagged one of the Jan 6 indictees back in December (that FB and friends didn't take down util after more pressure), regularly help folks publish breaking news, etc. Before, I was a well-cited academic (Test of Time award, Best of Years, etc), and work spanned early R&D for engineering tools your colleagues likely use today, and empirical & social research stuff for being able to judge work like this. Which I am, harshly. It looked like an easy win of a paper if they were dilligent, but ended up needlessly weasely. --- I agree with you on the tools weaknesses and big companies ignoring researchers, but for different reasons than researchers being wrong. The Twitter response is stock corporate CYA, so I don't take that at face value. Instead, consistent with what I've written, tools like botometer are largely irrelevant to modern practitioners here because reasons like: - Abuse teams at free social platforms are at battle with themselves and their surrounding organization & CEOs. The Facebook whistleblower accounts may particularly resonate well with you on why being factually right/wrong has little to do with what platforms do. - Botometer is dumb. Twitter went as far as acquiring the startup of one of the top GNN researchers: In contrast, botometer is from the dinosaur era. - Botometer is data-starved. Data is thin (no IPs, ...) and at low volumes (capped APIs). When you have APIs, app use logs, etc., same as someone self-hosting even the tiniest and most boring online store, you can do much better informed analyses. It's like a crypto app fraud & compliance team who only looks at blockchain data to decide good bad account/transaction, and ignores all the app log data & threat intel data.

mike_hearn 5y ago
"Section 3.2: the 10%+ of already-deleted accounts were removed." You've been implying that this is a methodological flaw of the paper we're talking about but Section 3.2 is an attempt to replicate someone else's study, one which claims Botometer isn't flawed. As they clearly point out the study wasn't replicable because the original authors didn't record/supply the data used to do the original classification, just numeric user IDs. Non-replicable research is exactly the kind of problem they're calling out here! Gallwitz and Kreil have pointed out by this paragraph that the way the accounts were pre-filtered will already avoid Botometer's biggest weaknesses. Yet, despite this fact, "A surprisingly high number of the 121 alleged bots in our sample are in reality individuals with academic or professional credentials, many of them directly related to the topic of vaccines" and "A number of the 121 alleged bots are, in reality, the official Twitter accounts of health-related organisations", and "some of the alleged vaccine bots are, in reality, Twitter accounts related to agricultural topics and pets". Section 3.2 points out that the study isn't replicable, which is by itself sufficient to discard it. Yet even when this problem is ignored and a replication is attempted anyway, a convincing takedown is easy to mount. It shows clearly that the authors never bothered to look at their "true positives" and is thus a great example of the problem in action. If you think it's easy to do a much better rebuttal to all these bad papers than these two German authors managed, great, let's see it? It sounds like you would be pointing out flaws in a widely used competitor, after all. I agree completely that Botometer is dumb and data starved, but where we seem to disagree is that you think it's possible to give social 'scientists' better tools. I don't think it is. Users consent to sharing their data with services when they tweet, post, etc but they don't consent to sharing it with random academics. The whole field of academic social bot research is DOA because of this problem, yet, academic institutions like universities, journals and granting bodies systematically cannot accept this. Anything they produce will always be a data starved PRNG.

mike_hearn 5y ago

"Section 3.2: the 10%+ of already-deleted accounts were removed." You've been implying that this is a methodological flaw of the paper we're talking about but Section 3.2 is an attempt to replicate someone else's study, one which claims Botometer isn't flawed. As they clearly point out the study wasn't replicable because the original authors didn't record/supply the data used to do the original classification, just numeric user IDs. Non-replicable research is exactly the kind of problem they're calling out here! Gallwitz and Kreil have pointed out by this paragraph that the way the accounts were pre-filtered will already avoid Botometer's biggest weaknesses. Yet, despite this fact, "A surprisingly high number of the 121 alleged bots in our sample are in reality individuals with academic or professional credentials, many of them directly related to the topic of vaccines" and "A number of the 121 alleged bots are, in reality, the official Twitter accounts of health-related organisations", and "some of the alleged vaccine bots are, in reality, Twitter accounts related to agricultural topics and pets". Section 3.2 points out that the study isn't replicable, which is by itself sufficient to discard it. Yet even when this problem is ignored and a replication is attempted anyway, a convincing takedown is easy to mount. It shows clearly that the authors never bothered to look at their "true positives" and is thus a great example of the problem in action. If you think it's easy to do a much better rebuttal to all these bad papers than these two German authors managed, great, let's see it? It sounds like you would be pointing out flaws in a widely used competitor, after all. I agree completely that Botometer is dumb and data starved, but where we seem to disagree is that you think it's possible to give social 'scientists' better tools. I don't think it is. Users consent to sharing their data with services when they tweet, post, etc but they don't consent to sharing it with random academics. The whole field of academic social bot research is DOA because of this problem, yet, academic institutions like universities, journals and granting bodies systematically cannot accept this. Anything they produce will always be a data starved PRNG.