4 ms·
What's the penalty (if any) for ignoring robots.txt and crawling anyway? Any citations of it being legally enforced via TOS or ASP?
by alphadog 16y ago
What's the penalty (if any) for ignoring robots.txt and crawling anyway? Any citations of it being legally enforced via TOS or ASP?
- quag 16y agoThere is no penalty, but they can always block the crawler or serve rubbish.
- moultano 16y agorobots.txt in the past has been the legal defense against claims of copyright infringement.
- uxp 16y agoCurious if you have any citations for this remark. I have always been under the impression that robots.txt is entirely optional, both to implement and to respect.
- moultano 16y agoIANAL and looking for other sources, but it's discussed along with many other factors here: http://www.benedict.com/Digital/Internet/Field/Field.aspx http://www.benedict.com/Digital/Internet/Field/Field.aspx Essentially, the existence of robots.txt and meta noindex has made courts more comfortable ruling that the index of search engines constitutes fair use.
- uxp 16y agoBut this still doesn't specifically answer the question, "What is the penalty for ignoring 'User-agent:* Disallow: /' in robots.txt, or meta tag no-index", only that without a robots.txt or no-index tag, crawlers can't be found liable for accessing information that is freely available to them. Interesting link, however. I was unaware of that case.