This article is good to see but frustrating as similarly weak in execution due to cherry picking and selection bias. People will mistake the review's sloppiness

This article is good to see but frustrating as similarly weak in execution due to cherry picking and selection bias. People will mistake the review's sloppiness and issues with some papers with "social bots don't exist and detection is mostly false positives". It's not like Facebook is spending ~$1B every year here to ban 0 accounts. - Manipulated accounts are prevalent: People sell manipulation tools, and once you identify the tool fingerprints, you can often precisely identify the accounts using them. For example, one of our team members published on NBC etc on a 'cyborg' (mixed use) tool used by Qanon + Trump's social media manager which, initially, self-identified itself via a unique user agent tag. Those resulted in high volume same-text tweets & fake retweets as they'd blast out via members who gave personal account access, which is huge for scales like MAGA Twitter. We have plenty of examples like this, and reading the paper gives a weasel-worded impression that it's not happening. - The paper explicitly did not check accounts Twitter already removed for likewise suspicion of being bots. Note that Twitter is generally slower and and more conservative here than a researcher -- they have money at stake that really likes bots -- so when even Twitter agrees it's a TOS-abusing bot, that's generally a good sign. Ex: One of our collaborators was reporting on Saudi bots, and it took a year for Twitter to take them down, namely around the time Twitter acknowledged the Saudi gov had corrupted one of their employees and they announced a sweep. Skipping those accounts for validation is massive selection bias. Bans take awhile, and frustratingly generally too late as only well after the accounts are used for social media manipulation. But looking at old papers and ignoring the confirmed findings and then claiming no bots is bad science. - The original botometer paper was years ago, which matched a different era in Twitter use, Twitter moderation, and attacker patterns. The paper seems similar to evaluating a HFT algorithm from 10 years ago in the current stock market, and worse, with bad settings, and being surprised they lost money. You wouldnt invest your savings that way. Likewise, who would use a 5yr old virus scanner? It's a more boring paper when you phrase it this way. There are good takeaways from this paper for folks in the field, but they give a quite warped impression if you don't have the context for these Ex: Simple rules and features like in botometer can work, but settings matter (drift, ...) Ex: While we and others are moving to graph neural networks and other continuing advances in account/event analytics, we need to make sure these tools are accessible else social scientists will be stuck with old algorithms, crufty implementations, and confusing settings. (GNNs are super exciting in general, see our recent review on industry adoption of more modern graph intelligence w/ gradientflow .) Ex: Twitter's rules for accessing+ publishing data (... you largely can't) hurts infodemic research, forcing deep hacks in what should have been a simpler reproduction study like this Ex: Based on reception and writing of this paper, the term bots may be confusing the public on what social media manipulation looks like Edit: They could have used clean methodology and written a paper "The Botometer tool is increasingly unreliable; X% of papers relying on it are likely incorrect". I think they should, because it's probably higher than people suspect. But it's not what they did, so I wouldn't conclude much from it until they do. The (mis)reproduction study is not even close to the standards my (peer-reviewed) peers regularly work for.