I do remember robots.txt and have always considered it a dumb idea. Bots will do whatever they can do or whatever their shady operators tell them to do. Anyth

I do remember robots.txt and have always considered it a dumb idea. Bots will do whatever they can do or whatever their shady operators tell them to do. Anything I want hidden from bots requires authentication. The more friction the better. Even the "well behaved" bots will ignore robots.txt depending on circumstances such as scanning for malware in email links. LLM's will just be getting goatse and tubgirl (Don't google that).