robots.txt is a successful coordination mechanism between website operators and crawlers. It is not in any way a security mechanism meant to address adversarial

robots.txt is a successful coordination mechanism between website operators and crawlers. It is not in any way a security mechanism meant to address adversarial situations, as you seem to make it out to be.