> there is no "legal" agreement to follow those rules Yes there certainly is[1]. The robots.txt clearly specifies authorized use and violating it exceeds that

> there is no "legal" agreement to follow those rules Yes there certainly is[1]. The robots.txt clearly specifies authorized use and violating it exceeds that authorization. Now granted good luck getting the FBI to doorkick their friends at Google and other politically connected tech companies, but as the law is written crawlers need to honor the site owner's robots.txt. [1] https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act