One thing is a robots.txt policy, meant mostly for search crawlers. Another thing is the copyright of the content, terms of use policies, etc. Abiding by a ro

One thing is a robots.txt policy, meant mostly for search crawlers. Another thing is the copyright of the content, terms of use policies, etc. Abiding by a robots.txt policy doesn't make you immune to copyright, terms of service, law in various jurisdictions, etc. If you think that you are probably a kleptomaniac. Just create a robots.txt with "User-Agent: one billion asterisks" so that the crawlers die when parsing it.