3 ms·
In practice, the crawlers that read robots.txt don't do adversarial interoperability. If a website doesn't want to be in Google's index, they can opt out, and n
by skybrian 7y ago
In practice, the crawlers that read robots.txt don't do adversarial interoperability. If a website doesn't want to be in Google's index, they can opt out, and no lawyers are needed to do that.