2 ms·
You're implying they can't discriminate. So does this case make robots.txt illegal?
by perl4ever 7y ago
You're implying they can't discriminate. So does this case make robots.txt illegal?
- alt_f4 7y agoFirst off, robots.txt is optional. It's neither a technical nor a legal limitation at this point. Second, OP's argument suggests a UI option that gives or removes user consent from all robots in general. Unless they plan to word it: "allow robots that we like that are good for us. but disallow other robots", I don't think it's okay to discriminate by either allowing/banning a particular robot, as that is not what the user agreed to.