Funny incident: A while ago, one of these studies was shared on LinkedIn. As if by divine provenance, a user unironically posted a random political conspiracy t

Funny incident: A while ago, one of these studies was shared on LinkedIn. As if by divine provenance, a user unironically posted a random political conspiracy theory in the comments. It was completely unrelated too, something about Washington DC deleting incriminating files in a panic over the incoming Trump administration. The source was ZeroHedge.com and the "evidence" was a spike in Google Trends for file deletion software in DC. This was too good to pass up. I could conduct a mini-replication study of a study in the social media comments of that study! So I pasted a screenshot of the comment to ChatGPT with a neutral prompt: "Attached is a screenshot of a conversation on LinkedIn. If you came across this conversation, how would you respond? Feel free to navigate to any URLs in the screenshot." ChatGPT quickly realized what this was and spun up a highly detailed response. It started, however, by pointing out that zerohedge was not a trustworthy source, which I thought seemed ad-hominem-y and would not be productive. Hence I directed it to focus on the content of the message. It correctly identified that Google Trends was a very poor source of evidence and explained why with a very detailed analysis backed by citations. Then I jumped into the conversation with a clear, fair warning about what I was doing and posted a screenshot of ChatGPT's responses. The poster seemed open to the experiment but repeatedly linked the Google Trends charts as evidence. Eventially I realized that ChatGPT's response was not landing (too much text?) and what may help was a counter example. I proved that Google Trends could be made to show a similar spike in any search term that was uncommon enough. At that point he realized that his evidence was actually counter-evidence for his own conspiracy theory, and he quit. So, a mixed outcome. ChatGPT needed some shepherding, but that may be OK given the ad-hoc, highly unscientific setup. But what amazed me was the speed and accuracy with which ChatGPT recognized what this was and whipped up compelling arguments from a completely neutral prompt. Maybe instead of Community Notes what we need is GPT Notes which the community can verify. That is probably the only approach that can scale in this tsunami of misinformation.