robots.txt seems to be an irresistible attractor for some, most recently in a crusade against all kinds of GenAI. I get not wanting to have our data serve as t

robots.txt seems to be an irresistible attractor for some, most recently in a crusade against all kinds of GenAI. I get not wanting to have our data serve as training data, but I've also seen moderately large newspapers throwing literally all LLM bots in there, i.e. not only those that scrape training data, but also those that process users' search requests or even direct article fetches. The obvious, but possibly not expected, result was that this newspaper became effectively invisible to user searches in ChatGPT. Not sure if I'm an outlier here, but I personally click through to many of the sources ChatGPT provides, so they must be losing tons of traffic that way.