First off, robots.txt is optional. It's neither a technical nor a legal limitation at this point. Second, OP's argument suggests a UI option that gives or remo

First off, robots.txt is optional. It's neither a technical nor a legal limitation at this point. Second, OP's argument suggests a UI option that gives or removes user consent from all robots in general. Unless they plan to word it: "allow robots that we like that are good for us. but disallow other robots", I don't think it's okay to discriminate by either allowing/banning a particular robot, as that is not what the user agreed to.